How-To Tutorials

article-image-how-to-build-weather-app-using-kotlin-for-javascript

04 May 2018

19 min read

How to build a weather app using Kotlin for JavaScript

04 May 2018

In this tutorial, we will be covering JavaScript from a browser perspective. We will create a single page web app that will show the weather forecast for seven days from the current date. The user will provide a ZIP code as input for which the weather will be displayed. We will display all the basic information about the weather on a given day. We believe in learning by doing practicals. Let's see the power of Kotlin from a browser perspective. [box type="shadow" align="" class="" width=""]This article is an excerpt from the book, Kotlin Blueprints, written by Ashish Belagali, Hardik Trivedi, and Akshay Chordiya. This book is a practical guide to building industry-grade web, mobile, and desktop applications in Kotlin using frameworks such as Spring Boot and Node.js[/box] Conceptually, we will cover the following points while making a web app: Setting up a project to use Kotlin along with JavaScript Showing simple text using Kotlin code Interacting with Document Object Model (DOM) using Kotlin DSL and usage of kotlinx.html Creating your first Kotlin and JavaScript project Tighten your shoelaces! As a first step, we will do the setup and create a simple app that prints on a console and changes the background color of a page. Choosing an IDE From Microsoft Visual Studio, NetBeans to Eclipse and Code::Blocks, we have a series of great and powerful IDEs. Each of them has their own pros and cons. JetBrains is one of the giants that is famous for its cutting-edge software and IntelliJ IDEA Ultimate is considered among one of the most intelligent IDEs for Java. It supports Kotlin and JavaScript by default. There is no other hassle in setting up the environments. Just install it from https://www.jetbrains.com/idea and you are all set to create your first JavaScript project using Kotlin. Creating a project If you are all done with setting up an IDE, launch IntelliJ IDEA and select Create New Project. You will then have the following screen opened. Select Kotlin | Kotlin (JavaScript) options as shown in the following screenshot: Make sure you select Kotlin (JavaScript) as highlighted in the preceding screenshot. The next step is to provide your Project name and choose a destination for your project directory: Creating an HTML page No browser project is complete without an HTML page. Create an index.html page in the root directory of your project. And write the following lines in a <body> tag: <body> <script type="text/javascript" src="out/production/KotlinWeb/lib/kotlin.js"></script> <script type="text/javascript" src="out/production/KotlinWeb/KotlinWeb.js"></script> </body> Creating a Main.kt file After creating our index.html page. Let's create our first Kotlin file. Name it as Main.kt or provide any desired name. Create a file in the src folder and write the following function inside: fun main(args: Array<String>) { document.bgColor="FF0000" val message = "Kotlin Blueprints" println("Your first JS code using Kotlin") } Build the project, by selecting the Build | Build Project menu option. On expanding the project explorer on the left of your workspace you will have the following type of directory structure: Make sure you double-check that the <script> tags are added in the <body>. They should match the name with the files created inside out/production/KotlinBluePrintsJSDemo/. Running the project If you have followed all the steps simply execute your index.html file in any browser and you should see the following output on your console and a red colored page rendered on your DOM: Congratulations! You have executed your first Kotlin code on the browser. Since we have code written in Kotlin, source code needs to recompile every time we update the code. Simply reloading an HTML page will not work. So build your project from the Build | Build Project menu option. Developing a weather forecast web app It was fun writing Kotlin code for a browser and seeing it working, wasn't it? Now we should target bigger goals. Let's develop another app step by step. We will build a weather forecast app, where the user will enter a ZIP code and can see the weather details (seven-day forecast) for the provided region. We will use the OpenWeatherMap API to get the weather details. Please find more details at https://openweathermap.org/api. Before we move to the next step we should create a new project named KotlinBluePrintsJSDemo. Some quick steps to follow: Create a Kotlin+JavaScript project named KotlinBluePrintsJSDemo. Create an index.html page under the root directory. Create a Main.kt file inside the src directory. Add script tags to add two JavaScript files, kotlin.js and KotlinBluePrintsJSDemo.js. Build a project. We want to create an app that will look like this at the end. Entirely in Kotlin: Creating a UI with dummy data The very first thing we do is to create a dummy view and get a clear idea of how our HTML page will look. We will also use a bit of CSS to give basic styles to our <div> tags. Simple HTML approach Now we shall look at the index.html file that we created by writing the following code. It's boring plain HTML tags: <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Kotlin BluePrints JS Demo</title> </head> <body> <link rel="stylesheet" type="text/css" href="css/main.css"> <div id="container"> <label>Enter zip code : <input id="zipCode" type="number"> </label> <button id="submitZipCode" type="button">Get Weather</button> <div class="weatherContainer"> <div class="weatherBlock"> <div>13 Oct, 2017</div> <img src="images/weather_img.png" height="40px" width="40px"> <div> <span>35</span> <span>20</span> </div> </div> <div class="weatherBlock"> <div>13 Oct, 2017</div> <img src="images/weather_img.png" height="40px" width="40px"> <div> <span>35</span> <span>20</span> </div> </div>  </div> </div> <script src="out/production/KotlinBluePrintsJSDemo/lib/kotlin.js"> </script> <script src="out/production/KotlinBluePrintsJSDemo /KotlinBluePrintsJSDemo.js"></script> </body> </html> Observe two tags, <script> and <link>. We haven't added CSS yet. Let's create a CSS folder under the root directory and create a main.css file inside. The main.css will contain the following code for now: .weatherContainer { width: 90%; background: #EEEEEE; margin: 10px auto; position: relative; text-align:center; } .weatherBlock { background: #FFFFFF; height: 100px; width: 100px; display:inline-block; margin: 10px; } In a source code, we have also created an images directory and put some weather images in it to make the UI more beautiful. Creating UI using Kotlin The index.html page contains all the HTML code. We need to now move that HTML code to Kotlin. Kotlin has the capability to manipulate the DOM element and it can also deal with the tree elements and their hierarchy. Simply put two <script> tags and a parent <div> tag in an HTML page and everything will go to a Kotlin page: <div id="container"> </div> Now, in Main.kt we will write the HTML code that we previously wrote inside index.html. Main.kt and it will look as follows: fun main(args: Array<String>) { createUserInput() } fun createUserInput() { val root = document.getElementById("container") root?.innerHTML = "<label>Enter zip code : <input id="zipCode" type="number"></label>" + "<button id="submitZipCode" type="button">Get Weather</button>" + "<div class="weatherContainer">" + "<div class="weatherBlock">" + "<div>13 Oct, 2017</div>" + "<img src="images/weather_img.png" height="40px" width="40px">"+ "<div>" + "<span>35</span>" + "<span>20</span>" + "</div>" + "</div>" + "<div class="weatherBlock">" + "<div>13 Oct, 2017</div>" + "<img src="images/weather_img.png" height="40px" width="40px">"+ "<div>" + "<span>35</span>" + "<span>20</span>" + "</div>" + "</div>" // Similarly add remaining divs } Take a note of the document object and its function getElementById. This is coming from the kotlin.browser.document package. Also org.w3c.dom as companion classes for all HTML elements. With object root, we get access to an innerHTML property and we can assign any valid HTML strings to it and it will get rendered. It is noteworthy that the nullability of root objects is handled with Null Safety operator ? of Kotlin. What is DSL? Now, the previous approach doesn't create much difference. Kotlin would want to do better! Let us introduce you to the beautiful concept of DSL. DSL stands for Domain Specific Language. As the name indicates, it gives you the feeling that you are writing code in a language, using terminology particular to a given domain, without being geeky, but then this terminology is cleverly embedded as a syntax in a powerful language. If you are from the Groovy community you must be aware of builders. Groovy builders allow defining data in a semi declarative way. It's a kind of mini-language of its own. Builders are considered good for generating XML and laying out UI components. Kotlin DSL uses Lambdas a lot. DSL in Kotlin are type-safe builders. It means we can detect compilation errors in IntelliJ's beautiful IDE. The type-check builders are much better than the dynamically-typed builders of Groovy. Using Kotlinx.html The DSL to build HTML trees is a pluggable dependency. We, therefore, need to set it up and configure it for our project. For now, we will keep things simple and add the dependency in them in the form of a .jar file. We will keep this .jar file in the lib folder, which will reside at the root level. The library is created by the JetBrains team only and it's open source. You can find it at https://github.com/Kotlin/kotlinx.html. You can simply visit the URL https://dl.bintray.com/kotlin/kotlinx.html/org/jetbrains/kotlinx/kotlinx-html-js/0.6.4/ and download the .jar file from there. For this demo app, we have used v 0.6.4. The .jar repository page can look as follows: To set up the kotlinx.html dependency in your app please follow these steps: In our app, we are using v 0.6.4. Make sure you download the JAR file named kotlinx-html-js-0.6.4.jar. Please verify that you have kept the .jar file inside the lib directory. Also, do not forget to add the .jar file as a library. Right-click on the .jar file and select Add As Library…. Select classes as a category while adding them as a library. Or you can simply choose to add the dependency via Gradle, in that, you need to add the following things to your build.gradle file: repositories { jcenter() } dependencies { //Fill this in with the version of kotlinx in use in your project def kotlinx_html_version = "your_version_here" // include for client-side compile "org.jetbrains.kotlinx:kotlinx-html- js:${kotlinx_html_version}" } Refactoring the HTML code using DSL The DSL code to make a button with the title "Get Weather" looks as follows: button { +"Get Weather" type = ButtonType.button onClickFunction = { // Your code to handle button click goes here. } } Simple and clean code. Similarly, let's create a function that will display an entire div, which has a label, text input, and button: fun getInputDiv(): HTMLDivElement { val inputDiv = document.create.div { label { +"Enter zip code : " input { id = "zipCode" type = InputType.number value = 411021.toString() } } button { +"Get Weather" type = ButtonType.button onClickFunction = { // Your code to handle button click goes here } } } return inputDiv } Observe how we have provided ID, input types, and a default ZIP code value. A default ZIP code value is optional. Let's spend some time understanding the previous code. label, input, button, type, id, and onClickFunction are nothing but functions. They are basically Lambda functions. Some of the functions that use Lambda parameters and call variations can be as follows: someFunction({}) someFunction("KotlinBluePrints",1,{}) someFunction("KotlinBluePrints",1){} someFunction{} Let's run the code. You may get an error on the console saying: Error Uncaught Error: Error loading module 'KotlinBluePrintsJSDemo'. Its dependency 'kotlinx-html-js' was not found. Please, check whether 'kotlinx-html-js' is loaded prior to 'KotlinBluePrintsJSDemo'. This is because kotlinx-html-js is missing, which is required to process the DSL generated code. You can see the kotlinx-html-js file generated under the out/production/KotlinBluePrintsJSDemo/lib path. Calling a weather API Now it's time to get the weather data and display it on the page. We will use XMLHttpRequest to achieve this. Register yourself at http://openweathermap.org/appid and get your application ID. Your application ID will be appended to the actual URL to make the authenticated call to the weather API. Once you get the app ID let's keep that information in the Constants.kt file: const val IMAGE_URL = "http://openweathermap.org/img/w/%s.png" const val BASE_URL = "https://api.openweathermap.org/data/2.5/forecast/daily? mode=json&units=metric&cnt=7" const val APP_ID = "Your open weather map application id" const val FULL_URL = "$BASE_URL&appid=$APP_ID&q=" The Constants.kt file is not as simple as it looks. Check how we have stored different values. We have used const val, which is equivalent to const and static used combined. Also defining FULL_URL uses the concept of string interpolation. String interpolation is used to concatenate static strings along with string objects. You can also call functions in string interpolation as follows: h4 { +"Weather info for ${forecastResult.city.name}, (${forecastResult.city.country})" } Now, in onClickFunction we write the following code to perform the API call and on the successful response we call a showData function, which takes a forecastResult object: onClickFunction = { val zipCode = document.getElementById("zipCode") as HTMLInputElement val xmlHttpRequest = XMLHttpRequest() xmlHttpRequest.open("GET", FULL_URL + zipCode.value, false) xmlHttpRequest.send() println(xmlHttpRequest.responseText) val forecastResult = JSON.parse<ForecastResult> (xmlHttpRequest.responseText) showData(forecastResult) } Reading data from input elements See how we read data from input elements: document.getElementById("zipCode") as HTMLInputElement The as HTMLInputElement construct is basically casting a result into the HTMLInputElement class. Using as directly is not advisable because it can give you ClassCastException; a proper way to use it is as? HTMLInputElement. This returns null if the class cast fails. And Kotlin will force you to use a Null Safety operator from that very moment. Data classes We are maintaining ForecastResult, which is our model. For this purpose, we have data classes in Kotlin. One of the coolest features in Kotlin is data classes. All the pain that we used to endure to create and maintain POJO classes in Java is gone. No need to have those dedicated packages to hold your model class. Any Kotlin file can hold your data class. By default it provides you methods such as toString(), equals(), copy(), and hashCode() method implementation. In Android, we mostly use these types of classes to hold our JSON responses in the form of model classes. You can check out the data classes we created in ServerResponses.kt: data class ForecastResult(val city: City, val list: Array<Forecast>) data class City(val id: Long, val name: String, val coord: Coordinates, val country: String, val population: Int) data class Coordinates(val lon: Float, val lat: Float) data class Forecast(val dt: Long, val temp: Temperature, val pressure: Float, val humidity: Int, val weather: Array<Weather>, val speed: Float, val deg: Int, val clouds: Int) data class Temperature(val day: Float, val min: Float, val max: Float, val night: Float, val eve: Float, val morn: Float) data class Weather(val id: Long, val main: String, val description: String, val icon: String) Some of the points to consider while using data classes are: The primary constructor needs to have at least one parameter All primary constructor parameters need to be marked as val or var Data classes cannot be abstract, open, sealed, or inner (Before version 1.1) data classes may only implement interfaces Showing data to the user Now comes the interesting part. We gate a ForecastResult object, which holds all the records. The list object holds records for seven days. Let's create a showData function that takes a ForecastResult object and display title text in <h4>. The code will look like the following snippet. Also, it has yet again one more example of string interpolation: fun showData(forecastResult: ForecastResult) { val root = document.getElementById("container") root?.appendChild(document.create.div(classes = "currentTemp") { h4 { +"Weather info for ${forecastResult.city.name (${forecastResult.city.country})" } }) } This is simple now, quickly create a showForecast function that will be called from showData and will display the weather forecast for seven days. The showForecast is used with a function from Kotlin. thewith() is one of those functions that is liked by the developer community a lot; it makes use of Kotlin sweeter. The with() function accepts the receiver and the code written inside the function automatically applies to the receiver object. It's an inline function. Check out the following document: /** * Calls the specified function [block] with the given [receiver] as its receiver and returns its result. */ public inline fun <T, R> with(receiver: T, block: T.() -> R): R = receiver.block() In the code, observe how each iteration is using a with block. We have removed some of the lines from the original code, so that we can have the clean code snippet here: forecastResult.list.forEachIndexed { index, forecast -> with(forecast) { weatherContainer.appendChild(document.create.div(classes = "weatherBlock") { div { p(classes = "currentTemp") { +"${Math.round(temp.day)} °C" } } img(classes = "weatherImage") { src = "images/weather_img.png" } div { span(classes = "secondaryText") { +weather[0].main } } div { with(temp) { span(classes = "primaryText") { +"${Math.round(max)} °C" } span(classes = "secondaryText") { +" /${Math.round(min)} °C" } } } onClickFunction = { showDetailedForecast(forecastResult.city, forecast) } }) } } DSL and Kotlin code are now beautifully gelled. Also, notice the onClickFunction that we wrote on div. Sweet, isn't it? Showing weather details A very small part of the app is left now. Let's show some more details to the user. Along with this, we will also learn a few more features of Kotlin. We have created a showDetailedForecast function that takes the City and Forecast objects as parameters. The following code snippets provide two things to learn: fun showDetailedForecast(city: City, forecast: Forecast) { val root = document.getElementById("container") val weatherDetailDiv = document.create.div(classes = "detailsContainer") val basicDetailDiv = document.create.div { p(classes = "secondaryText") { +"${city.name}, ${city.country} (${city.coord.lat},${city.coord.lon})" } p(classes = "secondaryText") { +forecast.dt.getFullDate() } p(classes = "secondaryText") { +"${forecast.weather[0].main}, ${forecast.weather[0].description}" } } val otherDetailsDiv = document.create.div { div { id = "leftDiv" span(classes = "currentTemp") { +"${Math.round(forecast.temp.day)} °C" } img { src = "images/weather_img.png" width = 90.toString() height = 90.toString() } } div { id = "rightDiv" p(classes = "secondaryText") { +"Pressure: ${forecast.pressure} mb" } p(classes = "secondaryText") { +"Humidity: ${forecast.humidity} %" } p(classes = "secondaryText") { +"Wind: ${forecast.speed} mph" } p(classes = "secondaryText") { +"Cloudiness: ${forecast.clouds} %" } } div(classes = "clearBoth") } weatherDetailDiv.appendChild(basicDetailDiv) weatherDetailDiv.appendChild(otherDetailsDiv) root?.appendChild(weatherDetailDiv) } Named parameters In Kotlin, we can call/bind a parameter with their name for any function. We can call the preceding function by interchanging the parameter sequence as well. Something like the following: showDetailedForecast(forecast = forecast, city = forecastResult.city) Observe that we swapped the place of the variable. And no wonder, all CSS classes that we have applied so far have a named parameter. Check all previous <div>, <h>, and <p> tags. Consider the following examples: val weatherDetailDiv = document.create.div(classes = "detailsContainer") button(classes = "getWeatherButton") span(classes = "primaryText") { +"${Math.round(max)} °C" } Extension functions Extension functions are a beautiful feature of Kotlin. Extension functions allow us to add the functions in the native class sets. All extension functions are statically resolved. Check out DateExtension.kt, it has three extension functions written for Long objects. They return different date formats. The code inside it may look a bit strange, which we will discuss in the following section: fun Long.getShortDate(): String { val getFormattedDate: dynamic = js("window.getShortDate") return getFormattedDate(this) } fun Long.getFullDate(): String { val getFormattedDate: dynamic = js("window.getFullDate") return getFormattedDate(this) } fun Long.getFullWeekDay(): String { val getFormattedDate: dynamic = js("window.getFullWeekDay") return getFormattedDate(this) } We don't need to write utility methods in Kotlin. We should prefer extension functions over Utils. Do not try to have any heavy methods as extension functions, instance functions are always good. Writing extension functions to format dates and to have some validation functions is OK. But it's not good to write an API calling function for any string class. Remember they are statically resolved. A project loaded with static is not good for memory. Giving final touches We wrote many lines of code so far. We also refactored them periodically. Once again it's a time to refactor and look for the possible improvements. Let's take a look back and see if there is any possibility of refactoring the code further. Adding CSS Let's add some custom font and style some of the missed HTML elements. We have used Robot font, you can use any font of your desire. It's a simple one-liner code to mention the font in the app. Add the following line to your index.html page just after the <body> tag: <link href="https://fonts.googleapis.com/css? family=Roboto+Condensed" rel="stylesheet"> And in main.css apply the font to an entire HTML page: html * { font-family: 'Roboto Condensed', sans-serif; } Reload the page. Looks beautiful now, doesn't it? To summarize, we learned various elements of Kotlin such as setting up Kotlin for JavaScript projects, interacting with DOM elements, DSL, and so on. The purpose of this article was to show that Kotlin's support for JavaScript is no more an experiment. It's already production ready. You can see what can be done using the benefits of statically typed programming languages and powerful JavaScript ecosystems. To know more about how to use Kotlin code for writing a Node.js application, you may refer to this book Kotlin Blueprints. Build your first Android app with Kotlin How to convert Java code into Kotlin 5 application development tools that will matter in 2018

0
0
67612

How-To Tutorials

article-image-vertex-ai-workbench-your-complete-guide-to-scaling-machine-learning-with-google-cloud

Jasmeet Bhatia, Kartik Chaudhary

04 Nov 2024

15 min read

Vertex AI Workbench: Your Complete Guide to Scaling Machine Learning with Google Cloud

Jasmeet Bhatia, Kartik Chaudhary

04 Nov 2024

15 min read

1
1
67379

article-image-python-scripting-essentials

Packt

17 May 2016

15 min read

Python Scripting Essentials

Packt

17 May 2016

15 min read

In this article by Rejah Rehim, author of the book Mastering Python Penetration Testing, we will cover: Setting up the scripting environment in different operating systems Installing third-party Python libraries Working with virtual environments Python language basics (For more resources related to this topic, see here.) Python is still the leading language in the world of penetration testing (pentesting) and information security. Python-based tools include all kinds oftools used for inputting massive amounts of random data to find errors and security loop holes, proxies, and even the exploit frameworks. If you are interested in tinkering with pentesting tasks, Python is the best language to learn because of its large number of reverse engineering and exploitation libraries. Over the years, Python has received numerous updates and upgrades. For example, Python 2 was released in 2000 and Python 3 in 2008. Unfortunately, Python 3 is not backward compatible; hence most of the programs written in Python 2 will not work in Python 3. Even though Python 3 was released in 2008, most of the libraries and programs still use Python 2. To do better penetration testing, the tester should be able to read, write, and rewrite python scripts. As a scripting language, security experts have preferred Python as a language to develop security toolkits. Its human-readable code, modular design, and large number of libraries provide a start for security experts and researchers to create sophisticated toolswith it. Python comes with a vast library (standard library) that accommodates almost everything from simple I/O to platform-specific APIcalls. Many of the default and user-contributed libraries and modules can help us in penetration testing with building tools to achieve interesting tasks. Setting up the scripting environment Your scripting environment is basically the computer you use for your daily workcombined with all the tools in it that you use to write and run Python programs. The best system to learn on is the one you are using right now. This section will help you to configure the Python scripting environment on your computer so that you can create and run your own programs. If you are using Mac OS X or Linux installation in your computer, you may have a Python Interpreter pre-installed in it. To find out if you have one, open terminal and type python. You will probably see something like this: $ python Python 2.7.6 (default, Mar 22 2014, 22:59:56) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> From the preceding output, we can see that Python 2.7.6 is installed in this system. By issuing python in your terminal, you started Python interpreter in the interactive mode. Here, you can play around with Python commands; what you type will run and you'll see the outputs immediately. You can use your favorite text editor to write your Python programs. If you do not have one, then try installing Geany or Sublime Text and it should be perfect for you. These are simple editors and offer a straightforward way to write as well as run your Python programs. In Geany, the output is shown in a separate terminal window, whereas Sublime Text uses an embedded terminal window. Sublime Text is not free, but it has a flexible trial policy that allows you to use the editor without any stricture. It is one of the few cross-platform text editors that is quite apt for beginners and has a full range of functions targeting professionals. Setting up in Linux Linux system is built in a way that makes it smooth for users to get started with Python programming. Most Linux distributions already have Python installed. For example, the latest versions of Ubuntu and Fedora come with Python 2.7. Also, the latest versions of Redhat Enterprise (RHEL) and CentOS come with Python 2.6. Just for the records, you might want to check it. If it is not installed, the easiest way to install Python is to use the default package manger of your distribution, such as apt-get, yum, and so on. Install Python by issuing the following commands in the terminal. For Debian / Ubuntu Linux / Kali Linux users: sudo apt-get install python2 For Red Hat / RHEL / CentOS Linux user sudo yum install python To install Geany, leverage your distribution'spackage manger. For Debian / Ubuntu Linux / Kali Linux users: sudo apt-get install geany geany-common For Red Hat / RHEL / CentOS Linux users: sudo yum install geany Setting up in Mac Even though Macintosh is a good platform to learn Python, many people using Macs actually run some Linux distribution or the other on their computer or run Python within a virtual Linux machine. The latest version of Mac OS X, Yosemite, comes with Python 2.7 preinstalled. Once you verify that it is working, install Sublime Text. For Python to run on your Mac, you have to install GCC, which can be obtained by downloading XCode, the smaller command-line tool. Also, we need to install Homebrew, a package manager. To install Homebrew, open Terminal and run the following: $ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" After installing Homebrew, you have to insert the Homebrew directory into your PATH environment variable. You can do this by including the following line in your ~/.profile file: export PATH=/usr/local/bin:/usr/local/sbin:$PATH Now that we are ready to install Python 2.7, run the following command in your terminal that will do the rest: $ brew install python To install Sublime Text, go to Sublime Text's downloads page in http://www.sublimetext.com/3 and click on the OS X link. This will get you the Sublime Text installer for your Mac. Setting up in Windows Windows does not have Python preinstalled on it. To check whether it isinstalled, open a command prompt and type the word python, and press Enter. In most cases, you will get a message that says Windows does not recognize python as a command. We have to download an installer that will set Python for Windows. Then, we have to install and configure Geany to run Python programs. Go to Python's download page in https://www.python.org/downloads/windows/and download the Python 2.7 installer, which is compatible with your system. If you are not aware of your operating systems architecture, then download 32-bit installers, which will work on both the architectures, but 64-bit will only work on 64-bit systems. To install Geany, go to Geany'sdownload page viahttp://www.geany.org/Download/Releases and download the full installer variant, which has a description Full Installer including GTK 2.16. By default, Geany doesn't know where Python resides on your system. So, we need to configure it manually. For this, write a Hello world program in Geany, save it anywhere in your system as hello.py, and run it. There are three methods you can run a python program in Geany: Select Build | Execute. Press F5. Click the icon with three gears on it: When you have a running hello.py program in Geany, go to Build | Set Build Commands. Then, enter the python commands option withC:Python27python -m py_compile "%f"and execute command withC:Python27python "%f". Now, you can run your Python programs while coding in Geany. It is recommended to run a Kali Linux distribution as a virtual machine and use this as your scripting environment. Kali Linux comes with a number of tools preinstalled and is based on Debian Linux, so you'll also be able to install a wide variety of additional tools and libraries. Also, some of the libraries will not work properly on Windows systems. Installing third-party libraries We will be using many Python libraries and this section will help you install and use third-party libraries. Setuptools and pip One of the most useful pieces of third-party Python software is Setuptools. With Setuptools, you could download and install any compliant Python libraries with a single command. The best way to install Setuptools on any system is to download the ez_setup.py file from https://bootstrap.pypa.io/ez_setup.pyand run this file with your Python installation. In Linux, run this in terminal with the correct path to theez_setup.py script: sudo python path/to/ez_setup.py For Windows 8 or the older versions of Windows with PowerShell 3 installed, start Powershell with Administrative privileges and run this command in it: > (Invoke-WebRequest https://bootstrap.pypa.io/ez_setup.py).Content | python - For Windows systems without a PowerShell 3 installed, download the ez_setup.py file from the link provided previously using your web browser and run that file with your Python installation. pipis a package management system used to install and manage software packages written in Python.After the successful installation of Setuptools, you can install pip by simply opening a command prompt and running the following: $ easy_install pip Alternatively, you could also install pip using your default distribution package managers: On Debian, Ubuntu and Kali Linux: sudo apt-get install python-pip On Fedora: sudo yum install python-pip Now, you could run pip from the command line. Try installing a package with pip: $ pip install packagename Working with virtual environments Virtual environment helps separate dependencies required for different projects; by working inside the virtual environment, it also helps to keep our global site-packages directory clean. Using virtualenv and virtualwrapper virtualenv is a python module which helps to create isolated Python environments for our each scripting experiments, which creates a folder with all necessary executable files and modules for a basic python project. You can install virtual virtualenv with the following command: sudo pip install virtualenv To create a new virtual environment,create a folder and enter into the folder from commandline: $ cd your_new_folder $ virtualenv name-of-virtual-environment This will initiate a folder with the provided name in your current working directory with all the Python executable files and pip library, which will then help install other packages in your virtual environment. You can select a Python interpreter of your choice by providing more parameters, such as the following command: $ virtualenv -p /usr/bin/python2.7 name-of-virtual-environment This will create a virtual environment with Python 2.7 .We have to activate it before we start using this virtual environment: $ source name-of-virtual-environment/bin/activate Now, on the left-hand side of the command prompt, the name of the active virtual environment will appear. Any package that you install inside this prompt using pip will belong to the active virtual environment, which will be isolated from all the other virtual environments and global installation. You can deactivate and exit from the current virtual environment using this command: $ deactivate virtualenvwrapper provides a better way to use virtualenv. It also organize all the virtual environments in one place. To install, we can use pip, but let's make sure we have installed virtualenv before installing virtualwrapper. Linux and OS X users can install it with the following method: $ pip install virtualenvwrapper Also,add thesethree lines inyour shell startup file like .bashrc or .profile. export WORKON_HOME=$HOME/.virtualenvs export PROJECT_HOME=$HOME/Devel source /usr/local/bin/virtualenvwrapper.sh This will set theDevel folder in your home directory as the location of your virtual environment projects. For Windows users, we can use another package virtualenvwrapper-win . This can also be installed with pip. pip install virtualenvwrapper-win Create a virtual environment with virtualwrapper: $ mkvirtualenv your-project-name This creates a folder with the provided name inside ~/Envs. To activate this environment, we can use workon command: $ workon your-project-name These two commands can be combined with the single one,as follows: $ mkproject your-project-name We can deactivate the virtual environment with the same deactivate command in virtualenv. To delete a virtual environment, we can use the following command: $ rmvirtualenv your-project-name Python language essentials In this section, we will go through the idea of variables, strings, data types, networking, and exception handling. As an experienced programmer, this section will be just a summarization of what you already know about Python. Variables and types Python is brilliant in case of variables—variable point to data stored in a memory location. This memory location may contain different values, such as integer, real number, Booleans, strings, lists, and dictionaries. Python interprets and declares variables when you set some value to this variable. For example, if we set: a = 1 andb = 2 Then, we will print the sum of these two variables with: print (a+b) The result will be 3 as Python will figure out both a and b are numbers. However, if we had assigned: a = "1" and b = "2" Then,the output will be 12, since both a and b will be considered as strings. Here, we do not have to declare variables or their type before using them, as each variable is an object. The type() method can be used to getthe variable type. Strings As any other programming language, strings are one of the important things in Python. They are immutable. So, they cannot be changed once they are defined. There are many Python methods, which can modify string. They do nothing to the original one, but create a copy and return after modifications. Strings can be delimited with single quotes, double quotes, or in case of multiple lines, we can use triple quotes syntax. We can use the character to escape additional quotes, which come inside a string. Commonly used string methods are: string.count('x'):This returns the number of occurrences of 'x' in the string string.find('x'):This returns the position of character 'x'in the string string.lower():This converts the string into lowercase string.upper():This converts the string into uppercase string.replace('a', 'b'):This replaces alla with b in the string Also, we can get the number of characters including white spaces in a string with the len() method. #!/usr/bin/python a = "Python" b = "Pythonn" c = "Python" print len(a) print len(b) print len(c) You can read more about the string function via https://docs.python.org/2/library/string.html. Lists Lists allow to store more than one variable inside it and provide a better method for sorting arrays of objects in Python. They also have methods that will help to manipulate the values inside them. list = [1,2,3,4,5,6,7,8] print (list[1]) This will print 2, as the Python index starts from 0. Print out the whole list: list = [1,2,3,4,5,6,7,8] for x in list: print (x) This will loop through all the elements and print them. Useful list methods are: .append(value):This appends an element at the end of list .count('x'):This gets the the number of 'x' in list .index('x'):This returns the index of 'x' in list .insert('y','x'):This inserts 'x' at location 'y' .pop():This returns last element and also remove it from list .remove('x'):This removes first 'x' from list .reverse():This reverses the elements in the list .sort():This sorts the list alphabetically in ascending order, or numerical in ascending order Dictionaries A Python dictionary is a storage method for key:value pairs. In Python, dictionaries are enclosed in curly braces, {}. For example, dictionary = {'item1': 10, 'item2': 20} print(dictionary['item2']) This will output 20. We cannot create multiple values with the same key. This will overwrite the previous value of the duplicate keys. Operations on dictionaries are unique. Slicing is not supported in dictionaries We can combine two distinct dictionaries to one by using the update method. Also, the update method will merge existing elements if they conflict: a = {'apples': 1, 'mango': 2, 'orange': 3} b = {'orange': 4, 'lemons': 2, 'grapes ': 4} a.update(b) Print a This will return: {'mango': 2, 'apples': 1, 'lemons': 2, 'grapes ': 4, 'orange': 4} To delete elements from a dictionary, we can use the del method: del a['mango'] print a This will return: {'apples': 1, 'lemons': 2, 'grapes ': 4, 'orange': 4} Networking Sockets are the basic blocks behind all the network communications by a computer. All network communications go through a socket. So, sockets are the virtual endpoints of any communication channel that takes place between two applications, which may reside on the same or different computers. The socket module in Python provides us a better way to create network connections with Python. So, to make use of this module, we will have to import this in our script: import socket socket.setdefaulttimeout(3) newSocket = socket.socket() newSocket.connect(("localhost",22)) response = newSocket.recv(1024) print response This script will get the response header from the server. Handling Exceptions Even though we wrote syntactically correct scripts, there will be some errors while executing them. So, we will have to handle the errors properly. The simplest way to handle exception in Python is try-except: Try to divide a number with zero in your Python interpreter: >>> 10/0 Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: integer division or modulo by zero So, we can rewrite this script with thetry-except blocks: try: answer = 10/0 except ZeroDivisionError, e: answer = e print answer This will return the error integer division or modulo by zero. Summary Now, we have an idea about basic installations and configurations that we have to do before coding. Also, we have gone through the basics of Python, which may help us speed up scripting. Resources for Article: Further resources on this subject: Exception Handling in MySQL for Python [article] An Introduction to Python Lists and Dictionaries [article] Python LDAP applications - extra LDAP operations and the LDAP URL library [article]

0
0
67256

How-To Tutorials

article-image-concurrency-and-parallelism-in-golang-tutorial

Natasha Mathur

06 Jul 2018

11 min read

How Concurrency and Parallelism works in Golang [Tutorial]

Natasha Mathur

06 Jul 2018

11 min read

Computer and software programs are useful because they do a lot of laborious work very fast and can also do multiple things at once. We want our programs to be able to do multiple things simultaneously, and the success of a programming language can depend on how easy it is to write and understand multitasking programs. Concurrency and parallelism are two terms that are bound to come across often when looking into multitasking and are often used interchangeably. However, they mean two distinctly different things. In this article, we will look at how concurrency and parallelism work in Go using simple examples for better understanding. Let's get started! This article is an excerpt from a book 'Distributed Computing with Go' written by V.N. Nikhil Anurag. The standard definitions given on the Go blog are as follows: Concurrency: Concurrency is about dealing with lots of things at once. This means that we manage to get multiple things done at once in a given period of time. However, we will only be doing a single thing at a time. This tends to happen in programs where one task is waiting and the program decides to run another task in the idle time. In the following diagram, this is denoted by running the yellow task in idle periods of the blue task. Parallelism: Parallelism is about doing lots of things at once. This means that even if we have two tasks, they are continuously working without any breaks in between them. In the diagram, this is shown by the fact that the green task is running independently and is not influenced by the red task in any manner: It is important to understand the difference between these two terms. Let's look at a few concrete examples to further elaborate upon the difference between the two. Concurrency Let's look at the concept of concurrency using a simple example of a few daily routine tasks and the way we can perform them. Imagine you start your day and need to get six things done: Make hotel reservation Book flight tickets Order a dress Pay credit card bills Write an email Listen to an audiobook The order in which they are completed doesn't matter, and for some of the tasks, such as writing an email or listening to an audiobook, you need not complete them in a single sitting. Here is one possible way to complete the tasks: Order a dress. Write one-third of the email. Make hotel reservation. Listen to 10 minutes of audiobook. Pay credit card bills. Write another one-third of the email. Book flight tickets. Listen to another 20 minutes of audiobook. Complete writing the email. Continue listening to audiobook until you fall asleep. In programming terms, we have executed the above tasks concurrently. We had a complete day and we chose particular tasks from our list of tasks and started to work on them. For certain tasks, we even decided to break them up into pieces and work on the pieces between other tasks. We will eventually write a program which does all of the preceding steps concurrently, but let's take it one step at a time. Let's start by building a program that executes the tasks sequentially, and then modify it progressively until it is purely concurrent code and uses goroutines. The progression of the program will be in three steps: Serial task execution. Serial task execution with goroutines. Concurrent task execution. Code overview The code will consist of a set of functions that print out their assigned tasks as completed. In the cases of writing an email or listening to an audiobook, we further divide the tasks into more functions. This can be seen as follows: writeMail, continueWritingMail1, continueWritingMail2 listenToAudioBook, continueListeningToAudioBook Serial task execution Let's first implement a program that will execute all the tasks in a linear manner. Based on the code overview we discussed previously, the following code should be straightforward: package main import ( "fmt" ) // Simple individual tasks func makeHotelReservation() { fmt.Println("Done making hotel reservation.") } func bookFlightTickets() { fmt.Println("Done booking flight tickets.") } func orderADress() { fmt.Println("Done ordering a dress.") } func payCreditCardBills() { fmt.Println("Done paying Credit Card bills.") } // Tasks that will be executed in parts // Writing Mail func writeAMail() { fmt.Println("Wrote 1/3rd of the mail.") continueWritingMail1() } func continueWritingMail1() { fmt.Println("Wrote 2/3rds of the mail.") continueWritingMail2() } func continueWritingMail2() { fmt.Println("Done writing the mail.") } // Listening to Audio Book func listenToAudioBook() { fmt.Println("Listened to 10 minutes of audio book.") continueListeningToAudioBook() } func continueListeningToAudioBook() { fmt.Println("Done listening to audio book.") } // All the tasks we want to complete in the day. // Note that we do not include the sub tasks here. var listOfTasks = []func(){ makeHotelReservation, bookFlightTickets, orderADress, payCreditCardBills, writeAMail, listenToAudioBook, } func main() { for _, task := range listOfTasks { task() } } We take each of the main tasks and start executing them in simple sequential order. Executing the preceding code should produce unsurprising output, as shown here: Done making hotel reservation. Done booking flight tickets. Done ordering a dress. Done paying Credit Card bills. Wrote 1/3rd of the mail. Wrote 2/3rds of the mail. Done writing the mail. Listened to 10 minutes of audio book. Done listening to audio book. Serial task execution with goroutines We took a list of tasks and wrote a program to execute them in a linear and sequential manner. However, we want to execute the tasks concurrently! Let's start by first introducing goroutines for the split tasks and see how it goes. We will only show the code snippet where the code actually changed here: /******************************************************************** We start by making Writing Mail & Listening Audio Book concurrent. *********************************************************************/ // Tasks that will be executed in parts // Writing Mail func writeAMail() { fmt.Println("Wrote 1/3rd of the mail.") go continueWritingMail1() // Notice the addition of 'go' keyword. } func continueWritingMail1() { fmt.Println("Wrote 2/3rds of the mail.") go continueWritingMail2() // Notice the addition of 'go' keyword. } func continueWritingMail2() { fmt.Println("Done writing the mail.") } // Listening to Audio Book func listenToAudioBook() { fmt.Println("Listened to 10 minutes of audio book.") go continueListeningToAudioBook() // Notice the addition of 'go' keyword. } func continueListeningToAudioBook() { fmt.Println("Done listening to audio book.") } The following is a possible output: Done making hotel reservation. Done booking flight tickets. Done ordering a dress. Done paying Credit Card bills. Wrote 1/3rd of the mail. Listened to 10 minutes of audio book. Whoops! That's not what we were expecting. The output from the continueWritingMail1, continueWritingMail2, and continueListeningToAudioBook functions is missing; the reason being that we are using goroutines. Since goroutines are not waited upon, the code in the main function continues executing and once the control flow reaches the end of the main function, the program ends. What we would really like to do is to wait in the main function until all the goroutines have finished executing. There are two ways we can do this—using channels or using WaitGroup. We'll use WaitGroup now. In order to use WaitGroup, we have to keep the following in mind: Use WaitGroup.Add(int) to keep count of how many goroutines we will be running as part of our logic. Use WaitGroup.Done() to signal that a goroutine is done with its task. Use WaitGroup.Wait() to wait until all goroutines are done. Pass WaitGroup instance to the goroutines so they can call the Done() method. Based on these points, we should be able to modify the source code to use WaitGroup. The following is the updated code: package main import ( "fmt" "sync" ) // Simple individual tasks func makeHotelReservation(wg *sync.WaitGroup) { fmt.Println("Done making hotel reservation.") wg.Done() } func bookFlightTickets(wg *sync.WaitGroup) { fmt.Println("Done booking flight tickets.") wg.Done() } func orderADress(wg *sync.WaitGroup) { fmt.Println("Done ordering a dress.") wg.Done() } func payCreditCardBills(wg *sync.WaitGroup) { fmt.Println("Done paying Credit Card bills.") wg.Done() } // Tasks that will be executed in parts // Writing Mail func writeAMail(wg *sync.WaitGroup) { fmt.Println("Wrote 1/3rd of the mail.") go continueWritingMail1(wg) } func continueWritingMail1(wg *sync.WaitGroup) { fmt.Println("Wrote 2/3rds of the mail.") go continueWritingMail2(wg) } func continueWritingMail2(wg *sync.WaitGroup) { fmt.Println("Done writing the mail.") wg.Done() } // Listening to Audio Book func listenToAudioBook(wg *sync.WaitGroup) { fmt.Println("Listened to 10 minutes of audio book.") go continueListeningToAudioBook(wg) } func continueListeningToAudioBook(wg *sync.WaitGroup) { fmt.Println("Done listening to audio book.") wg.Done() } // All the tasks we want to complete in the day. // Note that we do not include the sub tasks here. var listOfTasks = []func(*sync.WaitGroup){ makeHotelReservation, bookFlightTickets, orderADress, payCreditCardBills, writeAMail, listenToAudioBook, } func main() { var waitGroup sync.WaitGroup // Set number of effective goroutines we want to wait upon waitGroup.Add(len(listOfTasks)) for _, task := range listOfTasks{ // Pass reference to WaitGroup instance // Each of the tasks should call on WaitGroup.Done() task(&waitGroup) } // Wait until all goroutines have completed execution. waitGroup.Wait() } Here is one possible output order; notice how continueWritingMail1 and continueWritingMail2 were executed at the end after listenToAudioBook and continueListeningToAudioBook: Done making hotel reservation. Done booking flight tickets. Done ordering a dress. Done paying Credit Card bills. Wrote 1/3rd of the mail. Listened to 10 minutes of audio book. Done listening to audio book. Wrote 2/3rds of the mail. Done writing the mail. Concurrent task execution In the final output of the previous part, we can see that all the tasks in listOfTasks are being executed in serial order, and the last step for maximum concurrency would be to let the order be determined by Go runtime instead of the order in listOfTasks. This might sound like a laborious task, but in reality this is quite simple to achieve. All we need to do is add the go keyword in front of task(&waitGroup): func main() { var waitGroup sync.WaitGroup // Set number of effective goroutines we want to wait upon waitGroup.Add(len(listOfTasks)) for _, task := range listOfTasks { // Pass reference to WaitGroup instance // Each of the tasks should call on WaitGroup.Done() go task(&waitGroup) // Achieving maximum concurrency } // Wait until all goroutines have completed execution. waitGroup.Wait() Following is a possible output: Listened to 10 minutes of audio book. Done listening to audio book. Done booking flight tickets. Done ordering a dress. Done paying Credit Card bills. Wrote 1/3rd of the mail. Wrote 2/3rds of the mail. Done writing the mail. Done making hotel reservation. If we look at this possible output, the tasks were executed in the following order: Listen to audiobook. Book flight tickets. Order a dress. Pay credit card bills. Write an email. Make hotel reservations. Now that we have a good idea on what concurrency is and how to write concurrent code using goroutines and WaitGroup, let's dive into parallelism. Parallelism Imagine that you have to write a few emails. They are going to be long and laborious, and the best way to keep yourself entertained is to listen to music while writing them, that is, listening to music "in parallel" to writing the emails. If we wanted to write a program that simulates this scenario, the following is one possible implementation: package main import ( "fmt" "sync" "time" ) func printTime(msg string) { fmt.Println(msg, time.Now().Format("15:04:05")) } // Task that will be done over time func writeMail1(wg *sync.WaitGroup) { printTime("Done writing mail #1.") wg.Done() } func writeMail2(wg *sync.WaitGroup) { printTime("Done writing mail #2.") wg.Done() } func writeMail3(wg *sync.WaitGroup) { printTime("Done writing mail #3.") wg.Done() } // Task done in parallel func listenForever() { for { printTime("Listening...") } } func main() { var waitGroup sync.WaitGroup waitGroup.Add(3) go listenForever() // Give some time for listenForever to start time.Sleep(time.Nanosecond * 10) // Let's start writing the mails go writeMail1(&waitGroup) go writeMail2(&waitGroup) go writeMail3(&waitGroup) waitGroup.Wait() } The output of the program might be as follows: Done writing mail #3. 19:32:57 Listening... 19:32:57 Listening... 19:32:57 Done writing mail #1. 19:32:57 Listening... 19:32:57 Listening... 19:32:57 Done writing mail #2. 19:32:57 The numbers represent the time in terms of Hour:Minutes:Seconds and, as can be seen, they are being executed in parallel. You might have noticed that the code for parallelism looks almost identical to the code for the final concurrency example. However, in the function listenForever, we are printing Listening... in an infinite loop. If the preceding example was written without goroutines, the output would keep printing Listening... and never reach the writeMail function calls. Goroutines are concurrent and, to an extent, parallel; however, we should think of them as being concurrent. The order of execution of goroutines is not predictable and we should not rely on them to be executed in any particular order. We should also take care to handle errors and panics in our goroutines because even though they are being executed in parallel, a panic in one goroutine will crash the complete program. Finally, goroutines can block on system calls, however, this will not block the execution of the program nor slow down the performance of the overall program. We looked at how goroutine can be used to run concurrent programs and also learned how parallelism works in Go. If you found this post useful, do check out the book 'Distributed Computing with Go' to learn more about Goroutines, channels and messages, and other concepts in Go. Golang Decorators: Logging & Time Profiling Essential Tools for Go Programming Why is Go the go-to language for cloud-native development? – An interview with Mina Andrawos

0
0
67251

article-image-testing-restful-web-services-with-postman

Vijin Boricha

10 Apr 2018

3 min read

Testing RESTful Web Services with Postman

Vijin Boricha

10 Apr 2018

3 min read

In today's tutorial, we are going to leverage Postman framework to successfully test RESTful Web Services. We will also discuss a simple JUnit test case, which is calling the getAllUsers method in userService. We can check the following code: @RunWith(SpringRunner.class) @SpringBootTest public class UserTests { @Autowired UserService userSevice; @Test public void testAllUsers(){ List<User> users = userSevice.getAllUsers(); assertEquals(3, users.size()); } } In the preceding code, we have called getAllUsers and verified the total count. Let's test the single-user method in another test case: // other methods @Test public void testSingleUser(){ User user = userSevice.getUser(100); assertTrue(user.getUsername().contains("David")); } In the preceding code snippets, we just tested our service layer and verified the business logic. However, we can directly test the controller by using mocking methods. Postman First, we shall start with a simple API for getting all the users: http://localhost:8080/user The earlier method will get all the users. The Postman screenshot for getting all the users is as follows: In the preceding screenshot, we can see that we get all the users that we added before. We have used the GET method to call this API. Adding a user – Postman Let's try to use the POST method in user to add a new user: http://localhost:8080/user Add the user, as shown in the following screenshot: In the preceding result, we can see the JSON output: { "result" : "added" } Generating a JWT – Postman Let's try generating the token (JWT) by calling the generate token API in Postman using the following code: http://localhost:8080/security/generate/token We can clearly see that we use subject in the Body to generate the token. Once we call the API, we will get the token. We can check the token in the following screenshot: Getting the subject from the token By using the existing token that we created before, we will get the subject by calling the get subject API: http://localhost:8080/security/get/subject The result will be as shown in the following screenshot: In the preceding API call, we sent the token in the API to get the subject. We can see the subject in the resulting JSON. You read an excerpt from Building RESTful Web Services with Spring 5 - Second Edition written by Raja CSP Raman. From this book, you will learn to build resilient software in Java with the help of the Spring 5.0 framework. Check out the other tutorials from this book: How to develop RESTful web services in Spring Applying Spring Security using JSON Web Token (JWT) More Spring 5 tutorials: Introduction to Spring Framework Preparing the Spring Web Development Environment

0
0
66295

How-To Tutorials

Clint Bodungen

01 Feb 2024

6 min read

ChatGPT Prompting

Clint Bodungen

01 Feb 2024

6 min read

Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!This article is an excerpt from the book, ChatGPT for Cybersecurity Cookbook, by Clint Bodungen. Master ChatGPT and the OpenAI API, and harness the power of cutting-edge generative AI and large language models to revolutionize the way you perform penetration testing, threat detection, and risk assessment.IntroductionIn this article, we will explore the basics of ChatGPT prompting using the ChatGPT interface, which is different from the OpenAI Playground we used in the previous recipe. The advantage of using the ChatGPT interface is that it does not consume account credits and is better suited for generating formatted output, such as writing code or creating tables. Getting ready To use the ChatGPT interface, you will need to have an active OpenAI account. If you haven't already, please set up your ChatGPT account. How to do it… In this recipe, we'll guide you through using the ChatGPT interface to generate a Python script that retrieves a user's public IP address. By following these steps, you'll learn how to interact with ChatGPT in a conversation-like manner and receive context-aware responses, including code snippets. Now, let's proceed with the steps in this recipe: 1. In your browser, go to https://chat.openai.com and click “Log in” 2. Log in using your OpenAI credentials. 3. Once you are logged in, you will be taken to the ChatGPT interface. The interface is similar to a chat application, with a text box at the bottom where you can enter your prompts. Figure – The ChatGPT interface 4. ChatGPT uses a conversation-based approach, so you can simply type your prompt as a message and press "Enter" or click the button to receive a response from the model. For example, you can ask ChatGPT to generate a piece of Python code to find the public IP address of a user: Figure – Entering a prompt ChatGPT will generate a response containing the requested Python code, along with a thorough explanation. Figure – ChatGPT response with code 5. Continue the conversation by asking follow-up questions or providing additional information, and ChatGPT will respond accordingly. Figure – ChatGPT contextual follow-up response 6. Run the ChatGPT generated code by clicking on “Copy code”, paste it into your code editor of choice (I personally use Visual Studio Code), save it as a “.py” Python script, and run from a terminal. PS D:\GPT\ChatGPT for Cybersecurity Cookbook> python .\my_ip.py Your public IP address is: Your local network IP address is: 192.168.1.105 Figure – Running the ChatGPT generated script How it works… By using the ChatGPT interface to enter prompts, you can generate context-aware responses and content that continues over the course of an entire conversation like a chatbot. The conversation-based approach allows for more natural interactions and the ability to ask follow-up questions or provide additional context. The responses can even include complex formatting such as code snippets or tables (more on tables later). There’s more… As you become more familiar with ChatGPT, you can experiment with different prompt styles, instructions, and contexts to obtain the desired output for your cybersecurity tasks. You can also compare the results generated through the ChatGPT interface and the OpenAI Playground to determine which approach best fits your needs. Tip:You can further refine the generated output by providing very clear and specific instructions or using roles. It also helps to divide complex prompts into several smaller prompts, giving ChatGPT one instruction per prompt, building on the previous prompts as you go. In the upcoming recipes, we will delve into more advanced prompting techniques that utilize these techniques to help you get the most accurate and detailed responses from ChatGPT. As you interact with ChatGPT, your conversation history is automatically saved in the left panel of the ChatGPT interface. This feature allows you to easily access and review your previous prompts and responses. By leveraging the conversation history feature, you can keep track of your interactions with ChatGPT and quickly reference previous responses for your cybersecurity tasks or other projects. Figure – Conversation history in the ChatGPT interface To view a saved conversation, simply click on the desired conversation in the left panel. You can also create new conversations by clicking on the "+ New chat" button located at the top of the conversation list. This enables you to separate and organize your prompts and responses based on specific tasks or topics. Caution Keep in mind that when you start a new conversation, the model loses the context of the previous conversation. If you want to reference any information from a previous conversation, you will need to include that context in your new prompt. ConclusionIn conclusion, this article has unveiled the power of ChatGPT and its conversation-driven approach, making complex tasks like retrieving your public IP address a breeze. With step-by-step guidance, you've learned to harness ChatGPT's capabilities and enjoy context-aware responses, all while keeping your account credits intact. As you dive deeper into the world of ChatGPT, you'll discover its versatility in various applications and the potential to optimize your cybersecurity endeavors. By mastering ChatGPT's conversational prowess, you're on the path to seamless, productive interactions and a future filled with AI-driven possibilities.Author BioClint Bodungen is a cybersecurity professional with 25+ years of experience and the author of Hacking Exposed: Industrial Control Systems. He began his career in the United States Air Force and has since many of the world's largest energy companies and organizations, working for notable cybersecurity companies such as Symantec, Kaspersky Lab, and Booz Allen Hamilton. He has published multiple articles, technical papers, and training courses on cybersecurity and aims to revolutionize cybersecurity education using computer gaming (“gamification”) and AI technology. His flagship product, ThreatGEN® Red vs. Blue, is the world’s first online multiplayer cybersecurity simulation game, designed to teach real-world cybersecurity.

0
0
66198

article-image-how-greedy-algorithms-work

Richard Gall

10 Apr 2018

2 min read

How greedy algorithms work

Richard Gall

10 Apr 2018

2 min read

What is a greedy algorithm? Greedy algorithms are useful for optimization problems. They make the optimal choice at a localized and immediate level, with the aim being that you’ll find the optimal solution you want. It’s important to note that they don’t always find you the best solution for the data science problem you’re trying to solve - so apply them wisely. In the below video tutorial above, from Fundamental Algorithms in Scala, you'll learn when and how to apply a simple greedy algorithm, and see examples of both an iterative and recursive algorithm in action. with examples of both an iterative algorithm and a recursive algorithm. The advantages and disadvantages of greedy algorithms Greedy algorithms have a number of advantages and disadvantages. While on the one hand, it's relatively easy to come up with them, it is actually pretty challenging to identify the issues around the 'correctness' of your algorithm. That means that ultimately the optimization problem you're trying to solve by using greedy algorithms isn't really a technical issue as such. Instead, it's more of an issue with the scope and definition of your data analysis project. It's a human problem, not a mechanical one. Different ways to apply greedy algorithms There are a number of areas where greedy algorithms can most successfully be applied. In fact, it's worth exploring some of these problems if you want to get to know them in more detail. They should give you a clearer indication of how they work, what makes them useful, and potential drawbacks. Some of the best examples are: Huffman coding Dijkstra's algorithm Continuous knapsack problem To learn more about other algorithms check out these articles: 4 popular algorithms for Distance-based outlier detection 10 machine learning algorithms every engineer needs to know 4 Clustering Algorithms every Data Scientist should know Backpropagation Algorithm To learn to implement specific algorithms, use these tutorials: Creating a reference generator for a job portal using Breadth First Search (BFS) algorithm Implementing gradient descent algorithm to solve optimization problems Implementing face detection using the Haar Cascades and AdaBoost algorithm Getting started with big data analysis using Google’s PageRank algorithm Implementing the k-nearest neighbors' algorithm in Python Machine Learning Algorithms: Implementing Naive Bayes with Spark MLlib

0
0
66175

article-image-practical-ai-in-excel-create-a-linear-regression-model

M.T White

28 Jun 2023

12 min read

Practical AI in Excel: Create a Linear Regression Model

M.T White

28 Jun 2023

12 min read

0
0
66166

article-image-vue-js-3-0-is-ditching-javascript-for-typescript-what-else-is-new

Bhagyashree R

01 Oct 2018

5 min read

Vue.js 3.0 is ditching JavaScript for TypeScript. What else is new?

Bhagyashree R

01 Oct 2018

5 min read

Last week, Evan You, the creator of Vue.js gave a summary of what to expect in the coming major release of Vue.js 3.0. To provide a better support for TypeScript, the codebase is being written in TypeScript leaving behind vanilla JS. This new codebase currently targets evergreen browsers such as Google Chrome, and assumes baseline native ES2015 support. Let’s see what else we will see in this major iteration: High-level API changes Template syntax will not see much changes, except some tweaks in the scoped slots syntax. Vue.js 3.0 will come with native support for class-based components. This will provide users with an API that is pleasant to use in native ES2015 without the need of any transpilation or stage-x features. The Vue.js 3.x codebase will be written in TypeScript, providing improved support for TypeScript. Support for the 2.x object-based component format will be provided by internally transforming the object to a corresponding class. Functional components can now be plain functions, however, the async components will need to be explicitly created via a helper function. The virtual DOM format used in render functions will see major changes. Upgrading will be easier if you don’t heavily rely on handwritten (non-JSX) render functions in your app. Mixins will still be supported. Cleaner and more maintainable source code architecture To make contributing to Vue.js easier, Vue.js 3.0 is being re-written from the ground up for a cleaner and more maintainable architecture. To do this, the developers are breaking some internal functionalities into individual packages to isolate the scope of complexity. For example, the observer module will be converted to its own package, with its own public API and tests. As mentioned earlier, the codebase is being re-written in TypeScript. This makes proficiency in TypeScript a primary prerequisite for contributing to the new codebase. However, the type information and IDE support will enable a new contributor to easily make meaningful contributions. Proxy-based observation mechanism Vue.js 3.0 will come with a Proxy-based observer implementation that provides reactivity tracking with full language coverage. This aims to eliminate a number of limitations of the current implementation of Vue.js 2, which is based on Object.defineProperty: Detection of property addition or deletion Detection of Array index mutation or .length mutation Support for Map, Set, WeakMap and WeakSet Additionally, this new observer will have the following features: Exposed API for creating observables: This provides a lightweight and simple cross-component state management solution for small to medium scale scenarios. Lazy observation by default: In Vue.js 3.x, only the data used to render the initially visible part of an app will need to be observed. This will eliminate the overhead on app startup if your dataset is huge. Immutable observables: Immutable versions of a value can be created to prevent mutations even on nested properties, except when the system temporarily unlocks it internally. Better debugging capabilities: Two new hooks, renderTracked and renderTriggered are added. These will help you precisely trace when and why a component re-render is tracked or triggered. Other runtime improvements Smaller runtime The new codebase is designed to be tree-shaking friendly. The built-in components and directive runtime helpers will be imported on-demand and are tree-shakable. As a result, the constant baseline size for the new runtime is <10kb gzipped. Improved performance On initial benchmarks, the developers are observing up to 100% performance improvement across the board. Vue.js 3.0 will reduce the time spent in JavaScript when your app boots up. Built-in support for Fragments and Portals Vue 3.0 will come with built-in support for Fragments and Portals. Fragments are the components returning multiple root nodes. Portals are introduced to render a sub-tree in another part of the DOM, instead of inside the component. Improved slots mechanism All compiler-generated slots are now functions and invoked during the child component’s render call. This will ensure dependencies in slots are collected as dependencies for the child instead of the parent. This means that: When a slot content changes, only the child re-renders When the parent re-renders the child does not have to if its slot content did not change This improvement will provide even more precise change detection at the component tree level. Custom Renderer API Using this API you will be able to create custom renderers. With this API, it will be easier for the render-to-native projects like Weex and NativeScript Vue to stay up-to-date with upstream changes. This API will also make the creation of custom renderers for various other purposes trivially easier. Along with these, they have announced few compiler improvements and IE11 support. They haven’t revealed any date yet but we can expect Vue.js 3.0 to release in 2019. To know more, check out their official announcement on Medium. Vue CLI 3.0 is here as the standard build toolchain behind Vue applications Introducing Vue Native for building native mobile apps with Vue.js Testing Single Page Applications (SPAs) using Vue.js developer tools

0
0
66006

How-To Tutorials

article-image-how-to-perform-sentiment-analysis-using-python-tutorial

Sugandha Lahoti

15 Sep 2018

4 min read

How to perform sentiment analysis using Python [Tutorial]

Sugandha Lahoti

15 Sep 2018

4 min read

Sentiment analysis is one of the most popular applications of NLP. Sentiment analysis refers to the process of determining whether a given piece of text is positive or negative. In some variations, we consider "neutral" as a third option. This technique is commonly used to discover how people feel about a particular topic. This is used to analyze the sentiments of users in various forms, such as marketing campaigns, social media, e-commerce customers, and so on. In this article, we will perform sentiment analysis using Python. This extract is taken from Python Machine Learning Cookbook by Prateek Joshi. This book contains 100 recipes that teach you how to perform various machine learning tasks in the real world. How to Perform Sentiment Analysis in Python Step 1: Create a new Python file, and import the following packages: import nltk.classify.util from nltk.classify import NaiveBayesClassifier from nltk.corpus import movie_reviews Step 2: Define a function to extract features: def extract_features(word_list): return dict([(word, True) for word in word_list]) Step 3: We need training data for this, so we will use movie reviews in NLTK: if __name__=='__main__': # Load positive and negative reviews positive_fileids = movie_reviews.fileids('pos') negative_fileids = movie_reviews.fileids('neg') Step 4: Let's separate these into positive and negative reviews: features_positive = [(extract_features(movie_reviews.words(fileids=[f])), 'Positive') for f in positive_fileids] features_negative = [(extract_features(movie_reviews.words(fileids=[f])), 'Negative') for f in negative_fileids] Step 5: Divide the data into training and testing datasets: # Split the data into train and test (80/20) threshold_factor = 0.8 threshold_positive = int(threshold_factor * len(features_positive)) threshold_negative = int(threshold_factor * len(features_negative)) Step 6: Extract the features: features_train = features_positive[:threshold_positive] + features_negative[:threshold_negative] features_test = features_positive[threshold_positive:] + features_negative[threshold_negative:] print "\nNumber of training datapoints:", len(features_train) print "Number of test datapoints:", len(features_test) Step 7: We will use a Naive Bayes classifier. Define the object and train it: # Train a Naive Bayes classifier classifier = NaiveBayesClassifier.train(features_train) print "\nAccuracy of the classifier:", nltk.classify.util.accuracy(classifier, features_test) Step 8: The classifier object contains the most informative words that it obtained during analysis. These words basically have a strong say in what's classified as a positive or a negative review. Let's print them out: print "\nTop 10 most informative words:" for item in classifier.most_informative_features()[:10]: print item[0] Step 9: Create a couple of random input sentences: # Sample input reviews input_reviews = [ "It is an amazing movie", "This is a dull movie. I would never recommend it to anyone.", "The cinematography is pretty great in this movie", "The direction was terrible and the story was all over the place" ] Step 10: Run the classifier on those input sentences and obtain the predictions: print "\nPredictions:" for review in input_reviews: print "\nReview:", review probdist = classifier.prob_classify(extract_features(review.split())) pred_sentiment = probdist.max() Step 11: Print the output: print "Predicted sentiment:", pred_sentiment print "Probability:", round(probdist.prob(pred_sentiment), 2) If you run this code, you will see three main things printed on the Terminal. The first is the accuracy, as shown in the following image: The next is a list of most informative words: The last is the list of predictions, which are based on the input sentences: How does the Code work? We use NLTK's Naive Bayes classifier for our task here. In the feature extractor function, we basically extract all the unique words. However, the NLTK classifier needs the data to be arranged in the form of a dictionary. Hence, we arranged it in such a way that the NLTK classifier object can ingest it. Once we divide the data into training and testing datasets, we train the classifier to categorize the sentences into positive and negative. If you look at the top informative words, you can see that we have words such as "outstanding" to indicate positive reviews and words such as "insulting" to indicate negative reviews. This is interesting information because it tells us what words are being used to indicate strong reactions. Thus we learn how to perform Sentiment Analysis in Python. For more interesting machine learning recipes read our book, Python Machine Learning Cookbook. Understanding Sentiment Analysis and other key NLP concepts. Twitter Sentiment Analysis. Sentiment Analysis of the 2017 US elections on Twitter.

0
0
65714

How-To Tutorials

article-image-top-100-essential-data-science-tools-repos-streamline-your-workflow-today

Merlyn Shelley

27 Jun 2024

14 min read

Top 100+ Essential Data Science Tools & Repos: Streamline Your Workflow Today!

Merlyn Shelley

27 Jun 2024

14 min read

IntroductionAs data professionals, navigating the vast sea of Big Data often leaves us searching for the right tools to harness its potential. Whether we're defining intricate problems, identifying emerging trends, or crafting innovative solutions, the challenge is undeniable. Too often, this quest has us wandering aimlessly through the web, seeking elusive answers. Here at the DataPro Newsletter team, we understand this all too well. That's why, in celebration of our 100th edition, we're thrilled to present a special gift to our valued readers—a thorough reference module brimming with resources. This carefully curated collection features over 100 of the most popular tools and GitHub repositories. Each one is not only widely used and trusted but is also consistently updated with the latest breakthroughs to enhance your data processing capabilities. Think of this module as your treasure chest, designed to streamline your workflow and inspire innovative solutions. Bookmark this page for quick access whenever you encounter challenges in any area of data science and machine learning, from DataOps to Recommender Systems to Quantitative Finance—we've got it all covered! So, dive into this one-stop reference module, explore its depths, and let the spirit of data kinship propel you forward. Here's to more empowering tools and transformative insights from your DataPro team—cheers! DataOps/MLOps kestra-io/kestra: Kestra is an open-source orchestrator for scheduled and event-driven workflows, leveraging Infrastructure as Code for reliable management. open-metadata/OpenMetadata: OpenMetadata is a unified platform for data discovery, observability, and governance, featuring a central repository, column lineage, and team collaboration. dolthub/dolt: Dolt is a SQL database with Git-like version control features, accessible via MySQL or a command line interface. iterative/dvc: DVC is a tool for reproducible machine learning, enabling data and model versioning, lightweight pipelines, experiment tracking, and easy sharing. quiltdata/quilt: Quilt allows creating versioned datasets with Python and an S3 bucket. It supports data-driven teams, aiding rapid experimentation and collaboration. Real-time Data Processing allinurl/goaccess: GoAccess is a real-time web log analyzer for *nix systems and browsers, offering fast HTTP statistics. More details: goaccess.io. feathersjs/feathers: Feathers is a TypeScript/JavaScript framework for building APIs and real-time apps, compatible with various backends and frontends. apache/age: Apache AGE extends PostgreSQL with graph database capabilities, supporting both relational SQL and openCypher graph queries seamlessly. zephyrproject-rtos/zephyr: Real-time OS for diverse hardware, from IoT sensors to smart watches, emphasizing scalability, security, and resource efficiency. hazelcast/hazelcast: Hazelcast integrates stream processing and fast data storage for real-time insights, enabling immediate action on data-in-motion within unified platform. Data Quality Management WeBankFinTech/Qualitis: Qualitis manages data quality through verification, notification, and management across various data sources, solving data processing-related quality issues. raystack/optimus: Optimus is a robust workflow orchestrator for data transformation, modeling, pipelines, and quality management, emphasizing ease of use and reliability. Toloka/crowd-kit: Crowd-Kit is a Python library for crowdsourced annotation, featuring aggregation methods, metrics, and datasets to simplify working with crowd data. ydataai/ydata-profiling: ydata-profiling offers a streamlined, fast EDA solution akin to pandas' df.describe(), providing detailed DataFrame analysis exportable in formats like HTML and JSON. cleanlab/cleanlab: cleanlab automates data and label cleaning by detecting issues in ML datasets, enhancing model training with real-world data. Predictive Analytics spring-cloud/spring-cloud-dataflow: Spring Cloud Data Flow enables microservices-driven data processing pipelines on Cloud Foundry and Kubernetes, supporting diverse use cases like streaming and batch processing. ScottfreeLLC/AlphaPy: AlphaPy, a Python ML framework, caters to speculators and data scientists with scikit-learn, pandas, and additional tools for feature engineering and visualization. retentioneering/retentioneering-tools: Retentioneering simplifies analyzing clickstreams and user paths, offering deeper insights than funnel analysis, benefiting data and marketing analysts. genular/pandora: PANDORA offers advanced analytics for biomedical research, employing machine learning tools like clustering, PCA, UMAP, and interpretable models for discovery. nabeel-oz/qlik-py-tools: Qlik's SSE integrates modern data science into Qlik Sense, enabling business users to leverage advanced analytics through Python-based functions. Deep Learning Lightning-AI/pytorch-lightning: Lightning 2.0 simplifies PyTorch workflows with a stable API, enabling scalable training and deployment of AI models efficiently. ultralytics/yolov5: YOLOv5 by Ultralytics is a leading vision AI model, built on extensive open-source research and development for advanced performance. hpcaitech/ColossalAI: Colossal-AI simplifies distributed deep learning with user-friendly tools, enabling easy parallel training and inference similar to local model development. naptha/tesseract.js: Tesseract.js simplifies OCR with a webassembly-based Tesseract engine, supporting both browser and Node.js environments with easy integration and setup. microsoft/DeepSpeed: DeepSpeed enables efficient training of models like ChatGPT with significant speed improvements and cost reductions across all scales. Reinforcement Learning ray-project/ray: Ray is a unified framework that scales AI and Python applications with a distributed runtime and specialized AI libraries. d2l-ai/d2l-en: An open-source book using Jupyter notebooks to make deep learning accessible, blending concepts, context, and interactive code examples. Unity-Technologies/ml-agents: Unity ML-Agents enables games and simulations for training intelligent agents with deep reinforcement learning and imitation learning, fostering innovation in AI. google/trax: Trax is a Google Brain-endorsed deep learning library known for clear code and speed, demonstrated in a Colab notebook. wandb/wandb: The repository includes a CLI and Python API for visualizing and tracking machine learning experiments effectively. VowpalWabbit/vowpal_wabbit: Vowpal Wabbit advances machine learning with online, hashing, allreduce, and active learning techniques, pushing the frontier of ML capabilities. Time Series Analysis taosdata/TDengine: TDengine is a high-performance, open-source time-series database designed for IoT, connected cars, industrial IoT, and DevOps environments. timescale/timescaledb: An open-source SQL database for time-series data, optimized for rapid data ingestion and complex querying, available as a PostgreSQL extension. influxdata/telegraf: Telegraf is an agent for gathering and processing metrics, logs, and data, featuring 300+ plugins and community-driven development for flexibility. questdb/questdb: QuestDB is an open-source time-series database known for high throughput ingestion, fast SQL queries, and operational simplicity, ideal for various high-cardinality datasets. ccfos/nightingale: Nightingale is an all-in-one, open-source, cloud-native monitoring system combining data collection, visualization, and alerting capabilities seamlessly. Data Engineering PrefectHQ/prefect: Prefect simplifies Python data pipeline orchestration, transforming scripts into dynamic workflows that react to changes and ensure resilience. airbytehq/airbyte: Airbyte, an open-source data integration platform, offers 300+ connectors for seamless ELT pipelines between diverse data sources and destinations. argoproj/argo-workflows: Argo Workflows orchestrates parallel jobs on Kubernetes via container-native workflows, supporting DAGs and accelerating compute-intensive tasks like ML and data processing. dagster-io/dagster: Dagster is a cloud-native data pipeline orchestrator with integrated lineage, observability, declarative programming, and robust testability across the lifecycle. Avaiga/taipy: Taipy simplifies web app development for data scientists & ML engineers using Python, focusing on AI algorithms with no extra languages. Business Intelligence ankane/blazer: SQL-based tool for data exploration, chart creation, dashboard sharing. Supports various data sources, variables, checks, audits, and security integrations. evidence-dev/evidence: Open-source BI tool uses Markdown with SQL queries for data sourcing, rendering charts, and generating templated, dynamic web pages. lightdash/lightdash: Empower teams with self-service data insights using dbt: define metrics, visualize data, and share dashboards seamlessly across your organization. TuiQiao/CBoard: User-friendly open BI platform for self-service reporting and dashboards, simplifying data insights and sharing across teams effortlessly. quarylabs/quary: BI platform for engineers to connect databases, write SQL for table transformations, create charts, dashboards, and reports with collaboration and deployment capabilities. Data Visualization netdata/netdata: Real-time metrics collection and visualization for servers, cloud, Kubernetes, and edge/IoT devices, scaling effortlessly across diverse environments. directus/directus: Open-source API and dashboard for managing SQL database content with REST & GraphQL interfaces, supporting various databases, and customizable for on-premises or cloud deployment. airbnb/visx: Reusable low-level visualization components combining d3's power with React's DOM updating capabilities for dynamic data visualization. uber/react-vis: React component library for diverse data visualizations: line, bar, scatter, heatmaps, pie charts, sunbursts, radar charts, and more. bokeh/bokeh: Interactive visualization library for web browsers, offering versatile graphics creation and high-performance interactivity for large datasets and dashboards. apache/echarts: Free JavaScript library for intuitive, interactive, and customizable charts, ideal for enhancing commercial products with powerful visualizations. Recommender Systems NicolasHug/Surprise: Python scikit for building recommender systems with explicit rating data, emphasizing experiment control, dataset handling, and diverse prediction algorithms. gorse-io/gorse: Open-source recommendation system in Go, designed for universal integration into online services, automating model training based on user interaction data. recommenders-team/recommenders: Recommenders, a Linux Foundation project, offers Jupyter notebooks for building classic and cutting-edge recommendation systems, covering data prep, modeling, evaluation, optimization, and production deployment on Azure. alibaba/Alink: Alink, developed by Alibaba's PAI team, integrates Flink for ML algorithms. PyAlink supports various Flink versions, maintaining compatibility up to Flink 1.13. RUCAIBox/RecBole: RecBole, built on Python and PyTorch, facilitates research with 91 recommendation algorithms across general, sequential, context-aware, and knowledge-based categories. Quantitative Finance AI4Finance-Foundation/FinGPT: FinGPT is a cost-effective, adaptable financial large language model for quick updates and fine-tuning, enhancing accessibility compared to BloombergGPT. google/tf-quant-finance: This library leverages TensorFlow's hardware acceleration and automatic differentiation for high-performance mathematical methods, mid-level functions, and pricing models support. goldmansachs/gs-quant: GS Quant, a Python toolkit by Goldman Sachs, aids in developing quantitative trading strategies and risk management solutions with robust market experience. domokane/FinancePy: A Python finance library specializing in pricing and managing financial derivatives across fixed-income, equity, FX, and credit markets. romanmichaelpaolucci/Q-Fin: QFin is evolving with enhanced object-oriented principles, deprecating old modules like PDEs/SDEs, introducing 'stochastics' for model calibration and option pricing. avhz/RustQuant: This Rust library for quantitative finance covers diverse modules from autodiff and data handling to instruments pricing and stochastic processes. Responsible AI microsoft/responsible-ai-toolbox: Responsible AI Toolbox offers interfaces and libraries for model and data exploration, enabling developers to monitor and improve AI responsibly. Giskard-AI/giskard: Giskard, an open-source Python library, detects performance, bias, and security issues in AI applications, spanning LLMs to traditional ML models. fairlearn/fairlearn: Fairlearn, a Python package, helps developers assess and mitigate fairness issues in AI systems with algorithms and assessment metrics provided. Azure/PyRIT: PyRIT is an open-access Python tool for generative AI, aiding security professionals and ML engineers in identifying system risks. ModelOriented/DALEX: DALEX enhances model transparency to prevent failure through its explainability tools, supporting understanding and trust in complex AI systems. JohnSnowLabs/langtest: LangTest simplifies testing of AI models with over 60 tests in one line, covering robustness, bias, fairness, and accuracy across various NLP frameworks. Explainable AI (XAI) SeldonIO/alibi: Alibi is a Python library focused on machine learning model inspection, offering diverse explanation methods for classification and regression models. Trusted-AI/AIX360: AI Explainability 360 offers an open-source Python toolkit for detailed model interpretability across various data types, supporting diverse explanation methods. dssg/aequitas: Aequitas is an open-source toolkit for bias auditing and Fair ML, aiding data scientists and researchers in assessing and correcting model biases. albermax/innvestigate: iNNvestigate is a Python library providing a unified interface for various methods to analyze neural networks' predictions and understand their internal workings. mindsdb/lightwood: Lightwood is an AutoML framework simplifying machine learning pipelines with JSON-AI syntax, allowing customization and automation across diverse data types. Anomaly Detection SeldonIO/alibi-detect: Alibi Detect is a Python library for detecting outliers, adversarial attacks, and drift in tabular, text, image, and time series data. datamllab/tods: TODS automates outlier detection in multivariate time-series data with modules for data processing, feature analysis, and diverse detection algorithms. pygod-team/pygod: PyGOD is a Python library using PyTorch Geometric for graph outlier detection, offering 10+ algorithms and easy integration with PyOD. Jingkang50/OpenOOD: This repository replicates methods from the Generalized Out-of-Distribution Detection Framework for fair comparison across anomaly, novelty, and out-of-distribution detection methods. yzhao062/pyod: PyOD is a Python library for detecting anomalies in multivariate data, offering diverse algorithms for various project scales and datasets. chaos-genius/chaos_genius: Chaos Genius is an open-source ML-powered analytics engine for outlier detection and root cause analysis at scale. Supply Chain Analytics guacsec/guac: GUAC creates a high fidelity graph database for software security, facilitating organizational outcomes like audit, policy, and risk management. owasp-dep-scan/blint: BLint is a Binary Linter using lief to verify executable security and capabilities, now supporting SBOM generation for compatible binaries. samirsaci/picking-route: This repository focuses on improving warehouse productivity through Python-based tools and methodologies, particularly addressing order batching and optimizing picking routes using the Single Picker Routing Problem (SPRP). ragamarkely/scanalytics: Scanalytics automates Supply Chain Analytics & Design tasks in Python, streamlining analyses and reducing manual spreadsheet work for assignments. aitechtools/SunFlow: SunFlow optimizes supply chain design with comprehensive modeling of materials, components, suppliers, manufacturers, and customers, integrating costs, capacities, and constraints. CIOL-SUST/SupplyGraph: This repository introduces a benchmark dataset for applying Graph Neural Networks (GNNs) to supply chain networks, enabling research in optimization and prediction. Network Optimization ray-project/ray: Ray is a scalable framework with a distributed runtime and AI libraries designed to accelerate AI and Python applications. svg/svgo: SVGO optimizes SVG files by removing redundant metadata, comments, and hidden elements to improve file efficiency and rendering performance. zeux/meshoptimizer: meshoptimizer is a C/C++ library optimizing GPU rendering by reducing mesh complexity and storage overhead, compatible with Rust via meshopt crate. cvxpy/cvxpy: CVXPY is a Python-based modeling language designed for convex optimization problems, providing a natural expression format aligned with mathematical conventions. guofei9987/scikit-opt: The repository provides Python implementations of various swarm intelligence algorithms such as Genetic Algorithm, Particle Swarm Optimization, and others for optimization tasks. Speech Processing espnet/espnet: ESPnet is a detailed speech processing toolkit using PyTorch, covering recognition, synthesis, translation, enhancement, diarization, and understanding tasks. mozilla/DeepSpeech: DeepSpeech is an open-source Speech-To-Text engine based on Baidu's research, implemented using TensorFlow for accessibility and performance. microsoft/SpeechT5: The repository proposes SpeechT5, adapting T5's text-to-text approach for self-supervised speech and text representation learning using shared encoders and modality-specific nets. sloria/TextBlob: Python library simplifying NLP tasks like POS tagging, sentiment analysis, and classification with a straightforward API for textual data. pytorch/audio: Torchaudio integrates PyTorch with audio processing, emphasizing GPU acceleration, trainable features via autograd, and maintaining a consistent tensor-based style. Graph Data Science neo4j/graph-data-science: The Neo4j Graph Data Science (GDS) library offers graph algorithms, transformations, and ML pipelines, accessible via Cypher within Neo4j. cncf/landscape-graph: This repository explores open source project dynamics, evolution, and collaboration using a Graph Data Model for insightful community analysis. BlueBrain/nexus: Blue Brain Nexus organizes and enhances data with a Knowledge Graph ecosystem, featuring various products, libraries, and tools for comprehensive use. lynxkite/lynxkite: LynxKite is a robust graph data science platform with a user-friendly interface and powerful Python API for large datasets. dgraph-io/dgraph: Dgraph is a scalable GraphQL database optimized for performance, offering ACID transactions and distributed architecture for real-time queries. arangodb/arangodb: ArangoDB is a versatile multi-model database supporting documents, graphs, and key-values, empowering high-performance applications with SQL-like queries and JavaScript extensions. ETL/ELT (Extract, Transform, Load / Extract, Load, Transform) redpanda-data/connect: Redpanda Connect is a robust stream processor for seamless data integration, featuring a powerful mapping language and easy deployment options. turbot/steampipe: Steampipe simplifies data access from APIs with CLI, Postgres FDWs, SQLite extensions, export tools, and cloud-based Turbot Pipes. risingwavelabs/risingwave: RisingWave is a cost-efficient streaming database compatible with Postgres, designed for real-time event streaming data processing and analysis. apache/dolphinscheduler: Apache DolphinScheduler is a modern data orchestration platform with low-code workflow creation, robust task management, and cloud-native capabilities. rudderlabs/rudder-server: RudderStack is a privacy-focused, Segment-alternative platform in Golang and React. It simplifies data collection and integrates with warehouses and tools for enriched customer data pipelines. We hope this extensive collection of tools and techniques proves to be a valuable asset in your daily data practice. May it help you achieve smoother workflows and better outcomes!

2
0
65583

article-image-introduction-functional-programming

Packt

01 Dec 2016

19 min read

Introduction to the Functional Programming

Packt

01 Dec 2016

19 min read

0
0
65539

article-image-react-native-really-native-framework

Packt

21 Feb 2018

11 min read

Is React Native is really Native framework?

Packt

21 Feb 2018

11 min read

This article by Vladimir Novick, author of the book React Native - Building Mobile Apps with JavaScript, introduces the concept of how the the React Native is really a Native framework, it's working, information flow, architecture, and benefits. (For more resources related to this topic, see here.) Introduction So how React Native is different? Well it doesn’t fall under hybrid category because the approach is different. When hybrid apps are trying to make platform specific features reusable between platforms, React Native have platform independent features, but also have lots of specific platform implementations. Meaning that on iOS and on Android code will look different, but somewhere between 70-90 percent of code will be reused. Also React Native does not depend on HTML or CSS. You write in JavaScript, but this JavaScript is compiled to platform specific Native code using React Native bridge. It happens all the time, but it’s optimize to a way, that application will run smoothly in 60fps. So to summarize React Native is not really a Native framework, but It’s much closer to Native code, than hybrid apps. And now let’s dive a bit deeper and understand how JavaScript gets converted into a Native code. How React Native bridge from JavaScript to Native world works? Let’s dive a bit deeper and understand how React Native works under the hood, which will help us understand how JavaScript is compiled to a Native code and how the whole process works. It’s crucial to know how the whole process works, so if you will have performance issues one day, you will understand where it originates. Information flow So we’ve talked about React concepts that power up React Native and one of them is that UI is a function of data. You change the state and React knows what to update. Let’s visualize now how information flows through common React app. Check out the diagram: We have React component, which passes data to three child components Under the hood what is happening is, Virtual DOM tree is created representing these component hierarchy When state of the parent component is updated, React knows how to pass information to the children Since children are basically representation of UI, React figures out how to batch Browser DOM updates and executes them So now let’s remove Browser DOM and think that instead of batching Browser DOM updates, React Native does the same with calls to Native modules. So what about passing information down to Native modules? It can be done in two ways: Shared mutable data Serializable messages exchanged between JavaScript and Native modules React Native is going with the second approach. Instead of mutating data on shareable objects it passes asynchronous serialized batched messages to React Native Bridge. Bridge is the layer that is responsible for glueing together Native and JavaScript environments. Architecture Let’s take a look at the following diagram, which explains how React Native Architecture is structured and walk through the diagram: In diagram, pictured three layers: Native, Bridge and JavaScript. Native layer is pictured at the last in picture, because the layer that is closer to device itself. Bridge is the layer that connects between JavaScript and Native modules and basically is a transport layer that transport asynchronous serialized batched response messages from JavaScript to Native modules. When event is executed on Native layer. It can be touch, timer, network request. Basically any event involving device Native modules, It’s data is collected and is sent to the Bridge as a serialized message. Bridge pass this message to JavaScript layer. JavaScript layer is an event loop. Once Bridge passes Serialized payload to JavaScript, Event is processed and your application logic comes into play. If you update state, triggering your UI to re-render for example, React Native will batch Update UI and send them to the Bridge. Bridge will pass this Serialized batched response to Native layer, which will process all commands, that it can distinguish from serialized batched response and will Update UI accordingly. Threading model Up till now we’ve seen that there are lots of stuff going on under the hood of React Native. It’s important to know that everything is done on three main threads: UI (application main thread) Native modules JavaScript Runtime UI thread is the main Native thread where Native level rendering occurs. It is here, where your platform of choice, iOS or Android, does measures, layouting, drawing. If your application accesses any Native APIs, it’s done on a separate Native modules thread. For example, if you want to access the camera, Geo location, photos, and any other Native API. Panning and gestures in general are also done on this thread. JavaScript Runtime thread is the thread where all your JavaScript code will run. It’s slower than UI thread since it’s based on JavaScript event loop, so if you do complex calculations in your application, that leads to lots of UI changes, these can lead to bad performance. The rule of thumb is that if your UI will change slower than 16.67ms, then UI will appear sluggish. What are benefits of React Native? React Native brings with it lots of advantages for mobile development. We covered some of them briefly before, but let’s go over now in more detail. These advantages are what made React Native so popular and trending nowadays. And most of all it give web developers to start developing Native apps with relatively short learning curve compared to overhead learning Objective-C and Java. Developer experience One of the amazing changes React Native brings to mobile development world is enhancing developer experience. If we check developer experience from the point of view of web developer, it’s awesome. For mobile developer it’s something that every mobile developer have dreamt of. Let’s go over some of the features React Native brings for us out of the box. Chrome DevTools debugging Every web developer is familiar with Chrome Developer tools. These tools give us amazing experience debugging web applications. In mobile development debugging mobile applications can be hard. Also it’s really dependent on your target platform. None of mobile application debugging techniques does not even come near web development experience. In React Native, we already know, that JavaScript event loop is running on a separate thread and it can be connected to Chrome DevTools. By clicking Ctrl/Cmd + D in application simulator, we can attach our JavaScript code to Chrome DevTools and bring web debugging to a mobile world. Let’s take a look at the following screenshot: Here you see a React Native debug tools. By clicking on Debug JS Remotely, a separate Google Chrome window is opened where you can debug your applications by setting breakpoints, profiling CPU and memory usage and much more. Elements tab in Chrome Developer tools won’t be relevant though. For that we have a different option. Let’s take a look at what we will get with Chrome Developer tools Remote debugger. Currently Chrome developer tools are focused on Sources tab. You can notice that JavaScript is written in ECMAScript 2015 syntax. For those of you who are not familiar with React JSX, you see weird XML like syntax. Don’t worry, this syntax will be also covered in the book in the context of React Native. If you put debugger inside your JavaScript code, or a breakpoint in your Chrome development tools, the app will pause on this breakpoint or debugger and you will be able to debug your application while it’s running. Live reload As you can see in React Native debugging menu, the third row says Live Reload. If you enable this option, whenever you change your code and save, the application will be automatically reloaded. This ability to Live reload is something mobile developers only dreamt of. No need to recompile application after each minor code change. Just save and the application will reload itself in simulator. This greatly speed up application development and make it much more fun and easy than conventional mobile development. The workflow for every platform is different while in React Native the experience is the same. Does not matter for which platform you develop. Hot reload Sometimes you develop part of the application which requires several user interactions to get to. Think of, for example logging in, opening menu and choosing some option. When we change our code and save, while live reload is enabled, our application is reloaded and we need to once again do these steps. But it does not have to be like that. React Native gives us amazing experience of hot reloading. If you enable this option in React Native development tools and if you change your React Native component, only the component will be reloaded while you stay on the same screen you were before. This speeds up the development process even more. Component hierarchy inspections I’ve said before, that we cannot use elements panel in Chrome development tools, but how you inspect your component structure in React Native apps? React Native gives us built in option in development tools called Show Inspector. When clicking it, you will get the following window: After inspector is opened, you can select any component on the screen and inspect it. You will get the full hierarchy of your components as well as their styling: In this example I’ve selected Welcome to React Native! text. In the opened pane I can see it’s dimensions, padding margin as well as component hierarchy. As you can see it’s IntroApp/Text/RCTText. RCTText is not a React Native JavaScript component, but a Native text component, connected to React Native bridge. In that way you also can see that component is connected to a Native text component. There are even more dev tools available in React Native, that I will cover later on, but we all can agree, that development experience is outstanding. Web inspired layout techniques Styling for Native mobile apps can be really painful sometimes. Also it’s really different between iOS and Android. React Native brings another solution. As you may’ve seen before the whole concept of React Native is bringing web development experience to mobile app development. That’s also the case for creating layouts. Modern way of creating layout for the web is by using flexbox. React Native decided to adopt this modern technique for web and bring it also to the mobile world with small differences. In addition to layouting, all styling in React Native is very similar to using inline styles in HTML. Let’s take a look at example: const styles = StyleSheet.create({ container: { flex: 1, justifyContent: 'center', alignItems: 'center', backgroundColor: '#F5FCFF', }); As you can see in this example, there are several properties of flexbox used as well as background color. This really reminds CSS properties, however instead of using background-color, justify-content and align-items, CSS properties are named in a camel case manner. In order to apply these styles to text component for example. It’s enough to pass them as following: <Text styles={styles.container}>Welcome to React Native </Text> Styling will be discussed in the book, however as you can see from example before , styling techniques are similar to web. They are not dependant on any platform and the same for both iOS and Android Code reusability across applications In terms of code reuse, if an application is properly architectured (something we will also learn in this book), around 80% to 90% of code can be reused between iOS and Android. This means that in terms of development speed React Native beats mobile development. Sometimes even code used for the web can be reused in React Native environment with small changes. This really brings React Native to top of the list of the best frameworks to develop Native mobile apps. Summary In this article, we learned about the concept of how the React Native is really a Native framework, working, information flow, architecture, and it's benefits briefly. Resources for Article: Further resources on this subject: Building Mobile Apps [article] Web Development with React and Bootstrap [article] Introduction to JavaScript [article]

0
1
64855

article-image-installing-numpy-scipy-matplotlib-ipython

Packt Editorial Staff

12 Oct 2014

7 min read

Installing NumPy, SciPy, matplotlib, and IPython

Packt Editorial Staff

12 Oct 2014

7 min read

This article written by Ivan Idris, author of the book, Python Data Analysis, will guide you to install NumPy, SciPy, matplotlib, and IPython. We can find a mind map describing software that can be used for data analysis at https://www.xmind.net/m/WvfC/. Obviously, we can't install all of this software in this article. We will install NumPy, SciPy, matplotlib, and IPython on different operating systems. [box type="info" align="" class="" width=""]Packt has the following books that are focused on NumPy: NumPy Beginner's Guide Second Edition, Ivan Idris NumPy Cookbook, Ivan Idris Learning NumPy Array, Ivan Idris [/box] SciPy is a scientific Python library, which supplements and slightly overlaps NumPy. NumPy and SciPy, historically shared their codebase but were later separated. matplotlib is a plotting library based on NumPy. IPython provides an architecture for interactive computing. The most notable part of this project is the IPython shell. Software used The software used in this article is based on Python, so it is required to have Python installed. On some operating systems, Python is already installed. You, however, need to check whether the Python version is compatible with the software version you want to install. There are many implementations of Python, including commercial implementations and distributions. [box type="note" align="" class="" width=""]You can download Python from https://www.python.org/download/. On this website, we can find installers for Windows and Mac OS X, as well as source archives for Linux, Unix, and Mac OS X.[/box] The software we will install has binary installers for Windows, various Linux distributions, and Mac OS X. There are also source distributions, if you prefer that. You need to have Python 2.4.x or above installed on your system. Python 2.7.x is currently the best Python version to have because most Scientific Python libraries support it. Python 2.7 will be supported and maintained until 2020. After that, we will have to switch to Python 3. Installing software and setup on Windows Installing on Windows is, fortunately, a straightforward task that we will cover in detail. You only need to download an installer, and a wizard will guide you through the installation steps. We will give steps to install NumPy here. The steps to install the other libraries are similar. The actions we will take are as follows: Download installers for Windows from the SourceForge website (refer to the following table). The latest release versions may change, so just choose the one that fits your setup best. Library URL Latest Version NumPy http://sourceforge.net/projects/numpy/files/ 1.8.1 SciPy http://sourceforge.net/projects/scipy/files/ 0.14.0 matplotlib http://sourceforge.net/projects/matplotlib/files/ 1.3.1 IPython http://archive.ipython.org/release/ 2.0.0 Choose the appropriate version. In this example, we chose numpy-1.8.1-win32-superpack-python2.7.exe. Open the EXE installer by double-clicking on it. Now, we can see a description of NumPy and its features. Click on the Next button.If you have Python installed, it should automatically be detected. If it is not detected, maybe your path settings are wrong. Click on the Next button if Python is found; otherwise, click on the Cancel button and install Python (NumPy cannot be installed without Python). Click on the Next button. This is the point of no return. Well, kind of, but it is best to make sure that you are installing to the proper directory and so on and so forth. Now the real installation starts. This may take a while. [box type="note" align="" class="" width=""]The situation around installers is rapidly evolving. Other alternatives exist in various stage of maturity (see https://www.scipy.org/install.html). It might be necessary to put the msvcp71.dll file in your C:Windowssystem32 directory. You can get it from http://www.dll-files.com/dllindex/dll-files.shtml?msvcp71.[/box] Installing software and setup on Linux Installing the recommended software on Linux depends on the distribution you have. We will discuss how you would install NumPy from the command line, although, you could probably use graphical installers; it depends on your distribution (distro). The commands to install matplotlib, SciPy, and IPython are the same – only the package names are different. Installing matplotlib, SciPy, and IPython is recommended, but optional. Most Linux distributions have NumPy packages. We will go through the necessary steps for some of the popular Linux distros: Run the following instructions from the command line for installing NumPy on Red Hat: $ yum install python-numpy To install NumPy on Mandriva, run the following command-line instruction: $ urpmi python-numpy To install NumPy on Gentoo run the following command-line instruction: $ sudo emerge numpy To install NumPy on Debian or Ubuntu, we need to type the following: $ sudo apt-get install python-numpy The following table gives an overview of the Linux distributions and corresponding package names for NumPy, SciPy, matplotlib, and IPython. Linux distribution NumPy SciPy matplotlib IPython Arch Linux python-numpy python-scipy python-matplotlib Ipython Debian python-numpy python-scipy python-matplotlib Ipython Fedora numpy python-scipy python-matplotlib Ipython Gentoo dev-python/numpy scipy matplotlib ipython OpenSUSE python-numpy, python-numpy-devel python-scipy python-matplotlib ipython Slackware numpy scipy matplotlib ipython Installing software and setup on Mac OS X You can install NumPy, matplotlib, and SciPy on the Mac with a graphical installer or from the command line with a port manager such as MacPorts, depending on your preference. Prerequisite is to install XCode as it is not part of OS X releases. We will install NumPy with a GUI installer using the following steps: We can get a NumPy installer from the SourceForge website http://sourceforge.net/projects/numpy/files/. Similar files exist for matplotlib and SciPy. Just change numpy in the previous URL to scipy or matplotlib. IPython didn't have a GUI installer at the time of writing. Download the appropriate DMG file usually the latest one is the best.Another alternative is the SciPy Superpack (https://github.com/fonnesbeck/ScipySuperpack). Whichever option you choose it is important to make sure that updates which impact the system Python library don't negatively influence already installed software by not building against the Python library provided by Apple. Open the DMG file (in this example, numpy-1.8.1-py2.7-python.org-macosx10.6.dmg). Double-click on the icon of the opened box, the one having a subscript that ends with .mpkg. We will be presented with the welcome screen of the installer. Click on the Continue button to go to the Read Me screen, where we will be presented with a short description of NumPy. Click on the Continue button to the License the screen. Read the license, click on the Continue button and then on the Accept button, when prompted to accept the license. Continue through the next screens and click on the Finish button at the end. Alternatively, we can install NumPy, SciPy, matplotlib, and IPython through the MacPorts route, with Fink or Homebrew. The following installation steps shown, installs all these packages. [box type="info" align="" class="" width=""]For installing with MacPorts, type the following command: sudo port install py-numpy py-scipy py-matplotlib py- ipython [/box] Installing with setuptools If you have pip you can install NumPy, SciPy, matplotlib and IPython with the following commands. pip install numpy pip install scipy pip install matplotlib pip install ipython It may be necessary to prepend sudo to these commands, if your current user doesn't have sufficient rights on your system. Summary In this article, we installed NumPy, SciPy, matplotlib and IPython on Windows, Mac OS X and Linux. Resources for Article: Further resources on this subject: Plotting Charts with Images and Maps Importing Dynamic Data Python 3: Designing a Tasklist Application

0
0
64635

article-image-setting-gradle-properties-to-build-a-project

Savia Lobo

30 Jul 2018

10 min read

Setting Gradle properties to build a project [Tutorial]

Savia Lobo

30 Jul 2018

10 min read

0
0
64427

How to build a weather app using Kotlin for JavaScript

Vertex AI Workbench: Your Complete Guide to Scaling Machine Learning with Google Cloud

Python Scripting Essentials

How Concurrency and Parallelism works in Golang [Tutorial]

Testing RESTful Web Services with Postman

ChatGPT Prompting

How greedy algorithms work

Practical AI in Excel: Create a Linear Regression Model

Vue.js 3.0 is ditching JavaScript for TypeScript. What else is new?

How to perform sentiment analysis using Python [Tutorial]

Trending Topics

Top 100+ Essential Data Science Tools & Repos: Streamline Your Workflow Today!

Introduction to the Functional Programming

Is React Native is really Native framework?

Installing NumPy, SciPy, matplotlib, and IPython

Setting Gradle properties to build a project [Tutorial]

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access