Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-creating-coin-material
Packt
10 Mar 2016
7 min read
Save for later

Creating a Coin Material

Packt
10 Mar 2016
7 min read
In this article by Alan Thorn, the author of Unity 5.x By Example, the coin object, as a concept, represents a basic or fundamental unit in our game logic because the player character should be actively searching the level looking for coins to collect before a timer runs out. This means that the coin is more than mere appearance; its purpose in the game is not simply eye candy, but is functional. It makes an immense difference to the game outcome whether the coin is collected by the player or not. Therefore, the coin object, as it stands, is lacking in two important respects. Firstly, it looks dull and grey—it doesn't really stand out and grab the player's attention. Secondly, the coin cannot actually be collected yet. Certainly, the player can walk into the coin, but nothing appropriate happens in response. Figure 2.1: The coin object so far The completed CollectionGame project, as discussed in this article and the next, can be found in the book companion files in the Chapter02/CollectionGame folder. (For more resources related to this topic, see here.) In this section, we'll focus on improving the coin appearance using a material. A material defines an algorithm (or instruction set) specifying how the coin should be rendered. A material doesn't just say what the coin should look like in terms of color; it defines how shiny or smooth a surface is, as opposed to rough and diffuse. This is important to recognize and is why a texture and material refer to different things. A texture is simply an image file loaded in memory, which can be wrapped around a 3D object via its UV mapping. In contrast, a material defines how one or more textures can be combined together and applied to an object to shape its appearance. To create a new material asset in Unity, right-click on an empty area in the Project panel, and from the context menu, choose Create | Material. See Figure 2.2. You can also choose Assets | Create | Material from the application menu. Figure 2.2: Creating a material A material is sometimes called a Shader. If needed, you can create custom materials using a Shader Language or you can use a Unity add-on, such as Shader Forge. After creating a new material, assign it an appropriate name from the Project panel. As I'm aiming for a gold look, I'll name the material mat_GoldCoin. Prefixing the asset name with mat helps me know, just from the asset name, that it's a material asset. Simply type a new name in the text edit field to name the material. You can also click on the material name twice to edit the name at any time later. See Figure 2.3: Figure 2.3: Naming a material asset Next, select the material asset in the Project panel, if it's not already selected, and its properties display immediately in the object Inspector. There are lots of properties listed! In addition, a material preview displays at the bottom of the object Inspector, showing you how the material would look, based on its current settings, if it were applied to a 3D object, such as a sphere. As you change material settings from the Inspector, the preview panel updates automatically to reflect your changes, offering instant feedback on how the material would look. See the following screenshot: Figure 2.4: Material properties are changed from the Object Inspector Let's now create a gold material for the coin. When creating any material, the first setting to choose is the Shader type because this setting affects all other parameters available to you. The Shader type determines which algorithm will be used to shade your object. There are many different choices, but most material types can be approximated using either Standard or Standard (Specular setup). For the gold coin, we can leave the Shader as Standard. See the following screenshot: Figure 2.5: Setting the material Shader type Right now, the preview panel displays the material as a dull grey, which is far from what we need. To define a gold color, we must specify the Albedo. To do this, click on the Albedo color slot to display a Color picker, and from the Color picker dialog, select a gold color. The material preview updates in response to reflect the changes. Refer to the following screenshot: Figure 2.6: Selecting a gold color for the Albedo channel The coin material is looking better than it did, but it's still supposed to represent a metallic surface, which tends to be shiny and reflective. To add this quality to our material, click and drag the Metallic slider in the object Inspector to the right-hand side, setting its value to 1. This indicates that the material represents a fully metal surface as opposed to a diffuse surface such as cloth or hair. Again, the preview panel will update to reflect the change. See Figure 2.7: Figure 2.7: Creating a metallic material We now have a gold material created, and it's looking good in the preview panel. If needed, you can change the kind of object used for a preview. By default, Unity assigns the created material to a sphere, but other primitive objects are allowed, including cubes, cylinders, and torus. This helps you preview materials under different conditions. You can change objects by clicking on the geometry button directly above the preview panel to cycle through them. See Figure 2.8: Figure 2.8: Previewing a material on an object When your material is ready, you can assign it directly to meshes in your scene just by dragging and dropping. Let's assign the coin material to the coin. Click and drag the material from the Project panel to the coin object in the scene. On dropping the material, the coin will change appearance. See Figure 2.9: Figure 2.9: Assigning the material to the coin You can confirm that material assignment occurred successfully and can even identify which material was assigned by selecting the coin object in the scene and viewing its Mesh Renderer component from the object Inspector. The Mesh Renderer component is responsible for making sure that a mesh object is actually visible in the scene when the camera is looking. The Mesh Renderer component contains a Materials field. This lists all materials currently assigned to the object. By clicking on the material name from the Materials field, Unity automatically selects the material in the Project panel, making it quick and simple to locate materials. See Figure 2.10, The Mesh Renderer component lists all materials assigned to an object: Mesh objects may have multiple materials with different materials assigned to different faces. For best in-game performance, use as few unique materials on an object as necessary. Make the extra effort to share materials across multiple objects, if possible. Doing so can significantly enhance the performance of your game. For more information on optimizing rendering performance, see the online documentation at http://docs.unity3d.com/Manual/OptimizingGraphicsPerformance.html. Figure 2.10: The Mesh Renderer component lists all materials assigned to an object That's it! You now have a complete and functional gold material for the collectible coin. It's looking good. However, we're still not finished with the coin. The coin looks right, but it doesn't behave right. Specifically, it doesn't disappear when touched, and we don't yet keep track of how many coins the player has collected overall. To address this, then, we'll need to script. Summary Excellent work! In this article, you've completed the coin collection game as well as your first game in Unity. Resources for Article: Further resources on this subject: Animation features in Unity 5 [article] Saying Hello to Unity and Android [article] Learning NGUI for Unity [article]
Read more
  • 0
  • 0
  • 18617

article-image-using-chatgpt-for-data-enrichment
Jyoti Pathak
06 Nov 2023
10 min read
Save for later

Using ChatGPT For Data Enrichment

Jyoti Pathak
06 Nov 2023
10 min read
Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!IntroductionBusinesses thrive on information in today's data-driven era. However, raw data often needs enrichment to reveal its full potential. Here enters ChatGPT, a powerful tool not only for communication but also for enhancing data enrichment processes.Let us delve into the prospects of using ChatGPT for data enrichment.Does ChatGPT Do Data Mining?ChatGPT's prowess extends to data mining, unraveling valuable insights from vast datasets. Its natural language processing abilities allow it to decipher complex data structures, making it a versatile ally for researchers and analysts. By processing textual data, ChatGPT identifies patterns, enabling efficient data mining techniques.Process of data mining by ChatGPTChatGPT's ability to assist in data mining stems from its advanced natural language processing (NLP) capabilities. Here's an elaboration on the process of how ChatGPT can be utilized for data mining:1. Understanding Natural Language Queries:ChatGPT excels at understanding complex natural language queries. When provided with a textual prompt, it comprehends the context and intent behind the query. This understanding forms the basis for its data mining capabilities.2. Processing and Analyzing Textual Data:ChatGPT can process large volumes of textual data, including articles, reports, customer reviews, social media posts, etc. It can identify patterns, extract relevant information, and summarize lengthy texts, making it valuable for extracting insights from textual data sources.3. Contextual Analysis:ChatGPT performs contextual analysis to understand the relationships between words and phrases in a text. This contextual understanding enables ChatGPT to identify entities (such as names, places, and products) and their connections within the data, enhancing the precision of data mining results.4. Topic Modeling:ChatGPT can identify prevalent topics within textual data. Recognizing recurring themes and keywords helps categorize and organize large datasets into meaningful topics. This process is essential for businesses seeking to understand trends and customer preferences from textual data sources.5. Sentiment Analysis:ChatGPT can assess the sentiment expressed in textual data, distinguishing between positive, negative, and neutral sentiments. Sentiment analysis is crucial for businesses to gauge customer satisfaction, brand perception, market sentiment from online posts and reviews, and customer feedback.6. Data Summarization:ChatGPT can summarize extensive datasets, condensing large volumes of information into concise and informative summaries. This summarization capability is valuable for data mining, enabling analysts to quickly grasp essential insights without delving into voluminous data sources.7. Custom Queries and Data Extraction:Users can formulate custom queries and prompts tailored to specific data mining tasks. By asking ChatGPT precise questions about the data, users can extract targeted information, enabling them to focus on the particular aspects of the data relevant to their analysis.8. Interactive Exploration:ChatGPT allows for interactive exploration of data. Users can iteratively refine their queries based on the responses received, enabling a dynamic and exploratory approach to data mining. This interactivity facilitates a deeper understanding of the data and helps uncover hidden patterns and insights.By leveraging these capabilities, ChatGPT assists in data mining by transforming unstructured textual data into structured, actionable insights. Its adaptability to various queries and ability to process and analyze large datasets make it a valuable tool for businesses and researchers engaged in data mining.ChatGPT's ability to analyze JSON dataChatGPT can seamlessly analyze JSON data, a fundamental format for structuring data. Leveraging Python, integrating ChatGPT with JSON data becomes straightforward. Below is an illustrative Python code snippet demonstrating this integration:import openai import json # Your JSON data json_data = {    "key": "value",    "array": [1, 2, 3],    "nested": {        "inner_key": "inner_value"    } } # Convert JSON data to a string json_string = json.dumps(json_data) # Interact with ChatGPT response = openai.Completion.create(    engine="text-davinci-003",    prompt=json_string,    max_tokens=150  # Adjust the token limit as needed ) print(response.choices[0].text.strip())Also, ChatGPT can be used to analyze JSON data in various ways. Here are a few more examples of how ChatGPT can assist with JSON data analysis:1. JSON Data SummarizationChatGPT can be prompted to summarize complex JSON data structures, highlighting key attributes and relationships. For instance:Prompt: "Can you summarize the following JSON data for me?" {    "user": {        "name": "Sbani Paul",        "age": 30,        "email": "sbanipaul@example.com"    },    "order": {        "id": "123456",        "products": ["Widget A", "Widget B", "Widget C"],        "total": 150.75    } }ChatGPT Response: "The JSON data consists of user information and an order summary. The user, Sbani Paul, is 30 years old and can be reached at sbanipaul@example.com. The order with ID 123456 includes products Widget A, Widget B, and Widget C, totaling $150.75."2. JSON Schema ValidationChatGPT can help validate JSON data against a specified schema, ensuring data integrity and adherence to predefined data structures. For example:Prompt: "Is this JSON data valid according to the specified schema?" {    "name": "Aliza",    "age": 25,    "email": "Aliza@example.com" }ChatGPT Response: "Yes, the provided JSON data is valid according to the specified schema. It includes 'name' (string), 'age' (number), and 'email' (string) fields."3. JSON Data Transformation ChatGPT can assist in transforming JSON data into different formats, for instance, converting JSON to CSV or XML formats for specific use cases or applications.Prompt: "How can I transform this JSON data into CSV format?" [    {        "name": "Aliza",        "age": 30,        "city": "New York"    },    {        "name": "Bob",        "age": 35,        "city": "Los Angeles"    } ] ChatGPT Response: "To transform the given JSON data into CSV format, you can represent each object as a row and use commas to separate values. Here's the CSV representation:name,age,cityAliza,30,New YorkBob,35,Los Angeles4. Nested JSON Data AnalysisChatGPT can navigate through nested JSON structures, providing insights into specific elements within the data hierarchy.Prompt: "Can you provide details about the 'address' field in this nested JSON data?" {    "user": {        "name": "Eva",        "age": 28,        "address": {            "street": "123 Main St",            "city": "Anytown",            "postal_code": "54321"        }    } }ChatGPT Response: "Certainly, the 'address' field contains specific details. The user, Eva, resides at 123 Main St in Anytown with the postal code 54321."ChatGPT's ability to comprehend and respond to prompts about JSON data makes it a valuable tool for developers and data analysts working with structured data formats. Whether it's validation, transformation, or detailed analysis, ChatGPT can assist in various aspects of JSON data processing.What Is the Data Enrichment Method?Data enrichment transforms raw data into a goldmine of insights. This process involves augmenting existing data with supplementary information. Techniques include:Web scraping for real-time dataAPI integrations for seamless access to external databases.Leveraging machine learning algorithms to predict missing data.Data enrichment amplifies the value of datasets, enhancing analytical depth. The methods are diverse and dynamic, tailored to enhance the value of raw data. Let us go through an elaboration on the fundamental techniques of data enrichment:1. Web ScrapingWeb scraping involves extracting data from websites. It enables businesses to gather real-time information, news updates, pricing details, and more. By scraping relevant websites, organizations enrich their datasets with the latest and most accurate data available on the web. Web scraping tools can be programmed to extract specific data points from various web pages, ensuring the enrichment of databases with up-to-date information.2. API IntegrationsApplication Programming Interfaces (APIs) act as bridges between different software systems. Many platforms provide APIs that allow seamless data exchange. By integrating APIs into data enrichment processes, businesses can access external databases, social media platforms, weather services, financial data, and other sources. This integration ensures that datasets are augmented with comprehensive and diverse information, enhancing their depth and relevance.3. ChatGPT InteractionChatGPT's natural language processing abilities make it a valuable tool for data enrichment. Businesses can interact with ChatGPT to extract context-specific information by providing specific prompts. For example, ChatGPT can be prompted to summarize lengthy textual documents, analyze market trends, or provide detailed explanations about particular topics. These interactions enrich datasets by incorporating expert insights and detailed analyses, enhancing the overall understanding of the data.4. Machine Learning AlgorithmsMachine learning algorithms are pivotal in data enrichment, especially when dealing with large datasets. These algorithms can predict missing data points by analyzing patterns within the existing dataset. A variety of strategies, such as regression analysis, decision trees, and neural networks, are employed to fill gaps in the data intelligently. By accurately predicting missing values, machine learning algorithms ensure that datasets are complete and reliable, making them suitable for in-depth analysis and decision-making.5. Data Normalization and TransformationData normalization involves organizing and structuring data in a consistent format. It ensures that data from disparate sources can be effectively integrated and compared. Conversely, transformation consists of converting data into a standardized format, making it uniform and compatible. These processes are crucial for data integration and enrichment, enabling businesses to use consistent, high-quality data.6. Data AugmentationData augmentation involves expanding the dataset by creating variations of existing data points. In machine learning, data augmentation techniques are often used to enhance the diversity of training datasets, leading to more robust models. By applying similar principles, businesses can create augmented datasets for analysis, providing a broader perspective and enhancing the accuracy of predictions and insights.By employing these diverse methods, businesses can ensure their datasets are comprehensive and highly valuable. Data enrichment transforms raw data into a strategic asset, empowering organizations to make data-driven decisions to gain a competitive edge in their respective industries.ConclusionIncorporating ChatGPT into data enrichment workflows revolutionizes how businesses harness information. By seamlessly integrating with various data formats and employing diverse enrichment techniques, ChatGPT ensures that data isn't just raw facts but a source of actionable intelligence. Stay ahead in the data game – leverage ChatGPT to unlock the full potential of your datasets.Author BioJyoti Pathak is a distinguished data analytics leader with a 15-year track record of driving digital innovation and substantial business growth. Her expertise lies in modernizing data systems, launching data platforms, and enhancing digital commerce through analytics. Celebrated with the "Data and Analytics Professional of the Year" award and named a Snowflake Data Superhero, she excels in creating data-driven organizational cultures.Her leadership extends to developing strong, diverse teams and strategically managing vendor relationships to boost profitability and expansion. Jyoti's work is characterized by a commitment to inclusivity and the strategic use of data to inform business decisions and drive progress.
Read more
  • 0
  • 0
  • 18600

article-image-thomas-munro-from-enterprisedb-on-parallelism-in-postgresql
Bhagyashree R
17 Dec 2019
7 min read
Save for later

Thomas Munro from EnterpriseDB on parallelism in PostgreSQL

Bhagyashree R
17 Dec 2019
7 min read
PostgreSQL is a powerful, open-source object-relational database system. Since its introduction, it has been well-received by developers for its reliability, feature robustness, data-integrity, better licensing, and much more. However, one of its limitations has been the lack of support for parallelism, which changed in the subsequent releases. At PostgresOpen 2018, Thomas Munro, a programmer at EnterpriseDB and PostgreSQL contributor talked about how parallelism has evolved in PostgreSQL over the years. In this article, we will see some of the key parallelism-specific features that Munro discussed in his talk. [box type="shadow" align="" class="" width=""] Further Learning This article gives you a glimpse of query parallelism in PostgreSQL. If you want to explore it further along with other concepts like data replication, and database performance, check out our book Mastering PostgreSQL 11 - Second Edition by Hans-Jürgen Schönig. This second edition of Mastering PostgreSQL 11 helps you build dynamic database solutions for enterprise applications using PostgreSQL, which enables database analysts to design both the physical and technical aspects of the system architecture with ease. [/box] Evolution of parallelism in PostgreSQL PostgreSQL uses a process-based architecture instead of a thread-based one. On startup, it launches a “postmaster” process and after that creates a new process for every database session. Previously, it did not support parallelism in a single connection and each query used to run serially. The absence of “intra-query parallelism” in PostgreSQL was a huge limitation for answering the queries faster. Parallelism here means allowing a single process to have multiple threads to query the system and utilize the increasing CPU core counts. The foundation for parallelism in PostgreSQL was laid out in the 9.4 and 9.5 releases. These came with infrastructure updates like dynamic shared memory segments, shared memory queues, and background workers. PostgreSQL 9.6 was actually the first release that came with user-visible features for parallel query execution. It supported executor nodes: gather, parallel sequential scan, partial aggregate, and finalize aggregate. However, this was not enabled by default. Then in 2017, PostgreSQL 10 was released, which had parallelism enabled by default. It had a few more executor nodes including gather merge, parallel index scan, and parallel bitmap heap scan. Last year, PostgreSQL 11 came out with a couple of more executor nodes including parallel append and parallel hash join. It also introduced partition-wise joins and parallel CREATE INDEX. Key parallelism-specific features in PostgreSQL Parallel sequential scans Parallel sequential scans was the very first feature for parallel query execution. Introduced in PostgreSQL 9.6, this scan distributes blocks of a table among different processes. This assignment is done one after the other to ensure that the access to the table remains sequential. The processes that run in parallel and scan the tuples of a table are called parallel workers. There is one special worker called leader, which is responsible for coordinating and collecting the output of the scan from each of the worker. The leader may or may not participate in scanning the database depending on its load in dividing and combining processes. Parallel index scan Parallel index scan is based on the same concept as parallel sequential scan, but it involves more communication and waiting. Currently, the parallel index scans are supported only for B-Tree indexes. In a parallel index scan, index pages are scanned in parallel. Each process will scan a single index block and return all tuples referenced by that block. Meanwhile, other processes will also scan different index blocks and return the tuples. The results of a parallel B-Tree scan are then returned in sorted order. Parallel bitmap heap scan Again, this also has the same concept as the parallel sequential scan. Explaining the difference, Munro said, “You’ve got a big bitmap and you are skipping ahead to the pages that contain interesting tuples.” In parallel bitmap heap scan, one process is chosen as the leader, who performs a scan of one or more indexes and creates bitmap indicating which table blocks need to be visited. These table blocks are then divided among the worker processes as in a parallel sequential scan. Here the heap scan is done in parallel, but the underlying index scan is not. Parallel joins PostgreSQL supports all three join strategies in parallel query plans: nested loop join, hash join, or merge join. However, there is no parallelism supported in the inner loop. The entire loop is scanned as a whole, and the parallelism comes into play when each worker executes the inner loop as a whole. The results of each join are sent to gather node to produce the final results. Nested loop join: The nested loop is the most basic way for PostgreSQL to perform a join. Though it is considered to be slow, it can be efficient if the inner side is an index scan. This is because the outer tuples and hence the loops that loop up values in the index will be divided among worker processes. Merge join: The inner side is executed in full. It can be inefficient when sort needs to be performed because the work and resulting data are duplicated in every cooperating process. Hash join: In this join as well, the inner side is executed in full by every worker process to build identical copies of the hash table. It is inefficient in cases when the hash table is large or the plan is expensive. However, in parallel hash join, the inner side is a parallel hash that divides the work of building a shared hash table over the cooperating processes. This is the only join in which we can have parallelism on both sides. Partition-wise join Partition-wise join is a new feature introduced in PostgreSQL 11. In partition-wise join, the planner knows that both sides of the join have matching partition schemes. Here a join between two similarly partitioned tables are broken down into joins between their matching partitions if there is an equi-join condition between the partition key of joining tables. Munro explains, “It becomes parallelizable with the advent of parallel append, which can then run different branches of that query plan in different processes. But if you do that then granularity of parallelism is partitioned, which is in some ways good and in some ways bad compared to block-based granularity.” He further adds, “It means when the last worker runs out of work to do everyone else has to wait for that before the query is finished. Whereas, if you use block-based parallelism you don’t have the problem but there are some advantages as a result of that as well.” Parallel aggregation in PostgreSQL Calculating aggregates can be very expensive and when evaluated in a single process it could take a considerable amount of time. This problem was solved in PostgreSQL 9.6 with the introduction of parallel aggregation. This is essentially a divide and conquer strategy where multiple workers calculate a part of aggregate before the final value based on these calculations is calculated by the leader. This article walked you through some of the parallelism-specific features in PostgreSQL presented by Munro in his PostgresOpen 2018 talk.  If you want to get to grips with other advanced PostgreSQL features and SQL functions, do have a look at our Mastering PostgreSQL 11 - Second Edition book by Hans-Jürgen Schönig. By the end of this book, you will be able to use your database to its utmost capacity by implementing advanced administrative tasks with ease. PostgreSQL committer Stephen Frost shares his vision for PostgreSQL version 12 and beyond Introducing PostgREST, a REST API for any PostgreSQL database written in Haskell Percona announces Percona Distribution for PostgreSQL to support open source databases 
Read more
  • 0
  • 0
  • 18587

article-image-design-spring-aop
Packt
14 Oct 2009
12 min read
Save for later

Design with Spring AOP

Packt
14 Oct 2009
12 min read
Designing and implementing an enterprise Java application means not only dealing with the application core business and architecture, but also with some typical enterprise requirements. We have to define how the application manages concurrency so that the application is robust and does not suffer too badly from an increase in the amount of requests. We have to define the caching strategies for the application because we don't want that CPU or data-intensive operations to be executed over and over. We have to define roles and profiles, applying security policies and restricting access to application parts, because different kind of users will probably have different rights and permissions. All these issues require writing additional code that clutters our application business code and reduces its modularity and maintainability. But we have a choice. We can design our enterprise Java application keeping AOP in mind. This will help us to concentrate on our actual business code, taking away all the infrastructure issues that can be otherwise expressed as crosscutting concerns. This article will introduce such issues, and will show how to design and implement them with Spring 2.5 AOP support. Concurrency with AOP For many developers, concurrency remains a mystery. Concurrency is the system's skill to act with several requests simultaneously, so that threads don't corrupt the state of the objects when they gain access at the same time. A number of good books have been written on this subject, such as Concurrent Programming in Java and Java Concurrency in Practice. They deserve much attention, since concurrency is an aspect that's hard to understand, and not immediately visible to developers. Problems in the area of concurrency are hard to reproduce. However, it's important to keep concurrency in mind to assure that the application is robust regardless of the number of users it will serve. If we don't take into account concurrency and document when and how the problems of concurrency are considered, we will build an application taking some risks by supposing that the CPU will never simultaneously schedule processes on parts of our application that are not thread-safe. To ensure the building of robust and scalable systems, we use proper patterns: There are JDK packages just for concurrency. They are in the java.util.concurrent package, a result of JSR-166. One of these patterns is the read-write lock pattern, of which there is the interface java.util.concurrent.locks.ReadWriteLock and implementations, one of which is ReentrantReadWriteLock. The goal of ReadWriteLock is to allow the reading of an object from virtually endless number of threads, while only one thread at a time can modify it. In this way, the state of the object can never be corrupted because threads to reading the object's state will always read up-to-date data, and the thread modifying the state of the object in question will be able to act without the possibility of the object's state being corrupted. Another necessary skill is that the result of a thread's action can be visible to the other threads. The behavior is the same as we could have achieved using synchronized, but when using a read-write lock we are explicitly synchronizing the actions, whereas with synchronized synchronization is implicit. Now let's see an example of ReadWriteLock on BankAccountThreadSafe object. Before the read operation, that needs to be safe, we set the read lock. After the read operation, we release the read lock. Before the write operation that needs to be safe, we set the write lock. After a state modification, we release the write lock. package org.springaop.chapter.five.concurrent; import java.util.Date; import java.util.concurrent.locks.Lock; import java.util.concurrent.locks.ReadWriteLock; import java.util.concurrent.locks.ReentrantReadWriteLock; public final class BankAccountThreadSafe { public BankAccountThreadSafe(Integer id) { this.id = id; balance = new Float(0); startDate = new Date(); } public BankAccountThreadSafe(Integer id, Float balance) { this.id = id; this.balance = balance; startDate = new Date(); } public BankAccountThreadSafe(Integer id, Float balance, Date start) { this.id = id; this.balance = balance; this.startDate = start; } public boolean debitOperation(Float debit) { wLock.lock(); try { float balance = getBalance(); if (balance < debit) { return false; } else { setBalance(balance - debit); return true; } } finally { wLock.unlock(); } } public void creditOperation(Float credit) { wLock.lock(); try { setBalance(getBalance() + credit); } finally { wLock.unlock(); } } private void setBalance(Float balance) { wLock.lock(); try { balance = balance; } finally { wLock.unlock(); } } public Float getBalance() { rLock.lock(); try { return balance; } finally { rLock.unlock(); } } public Integer getId() { return id; } public Date getStartDate() { return (Date) startDate.clone(); } ... private Float balance; private final Integer id; private final Date startDate; private final ReadWriteLock lock = new ReentrantReadWriteLock(); private final Lock rLock = lock.readLock(); private final Lock wLock = lock.writeLock(); } BankAccountThreadSafeis a class that doesn't allow a bank account to be overdrawn (that is, have a negative balance), and it's an example of thread-safe class. The final fields are set in the constructors, hence implicitly thread-safe. The balance field, on the other hand, is managed in a thread-safe way by the setBalance,getBalance, creditOperation, and debitOperation methods. In other words, this class is correctly programmed, concurrency-wise. The problem is that wherever we would like to have those characteristics, we have to write the same code (especially the finally block containing the lock's release). We can solve that by writing an aspect that carries out that task for us. A state modification is execution(void com.mycompany.BankAccount.set*(*)) A safe read is execution(* com.mycompany.BankAccount.getBalance()) package org.springaop.chapter.five.concurrent; import java.util.concurrent.locks.Lock; import java.util.concurrent.locks.ReadWriteLock; import java.util.concurrent.locks.ReentrantReadWriteLock; import org.aspectj.lang.annotation.After; import org.aspectj.lang.annotation.Aspect; import org.aspectj.lang.annotation.Before; import org.aspectj.lang.annotation.Pointcut; @Aspect public class BankAccountAspect { /*pointcuts*/ @Pointcut("execution(* org.springaop.chapter.five.concurrent.BankAccount.getBalance())") public void safeRead(){} @Pointcut("execution(* org.springaop.chapter.five.concurrent.BankAccount.set*(*))") public void stateModification(){} @Pointcut( "execution(* org.springaop.chapter.five.concurrent.BankAccount.getId())") public void getId(){} @Pointcut("execution(* org.springaop.chapter.five.concurrent.BankAccount.getStartDate())) public void getStartDate(){} /*advices*/ @Before("safeRead()") public void beforeSafeRead() { rLock.lock(); } @After("safeRead()") public void afterSafeRead() { rLock.unlock(); } @Before("stateModification()") public void beforeSafeWrite() { wLock.lock(); } @After("stateModification()") public void afterSafeWrite() { wLock.unlock(); } private final ReadWriteLock lock = new ReentrantReadWriteLock(); private final Lock rLock = lock.readLock(); private final Lock wLock = lock.writeLock(); } The BankAccountAspect class applies the crosscutting functionality. In this case, the functionality is calling the lock and unlock methods on the ReadLock and the WriteLock. The before methods apply the locks with the @Before annotation, while the after methods release the locks as if they were in the final block, with the @After annotation that is always executed (after-finally advice). In this way the BankAccount class can become much easier, clearer, and briefer. It wouldn't have any perception that it can be executed in a thread-safe manner. package org.springaop.chapter.five.concurrent; import java.util.Date; public class BankAccount { public BankAccount(Integer id) { this.id = id; this.balance = new Float(0); this.startDate = new Date(); } public BankAccount(Integer id, Float balance) { this.id = id; this.balance = balance; this.startDate = new Date(); } public BankAccount(Integer id, Float balance, Date start) { this.id = id; this.balance = balance; this.startDate = start; } public boolean debitOperation(Float debit) { float balance = getBalance(); if (balance < debit) { return false; } else { setBalance(balance - debit); return true; } } public void creditOperation(Float credit) { setBalance(getBalance() + credit); } private void setBalance(Float balance) { this.balance = balance; } public Float getBalance() { return balance; } public Integer getId() { return id; } public Date getStartDate() { return (Date) startDate.clone(); } private Float balance; private final Integer id; private final Date startDate; } Another good design choice, together with the use of ReadWriteLock when necessary, is: using objects that once built are immutable, and therefore, not corruptible and can be easily shared between threads. Transparent caching with AOP Often, the objects that compose applications perform the same operations with the same arguments and obtain the same results. Sometimes, these operations are costly in terms of CPU usage, or may be there is a lot of I/O going on while executing those operations. To get better results in terms of speed and resources used, it's suggested to use a cache. We can store in it the results corresponding to the methods' invocations as a key-value pair: method and arguments as key and return object as value. Once you decide to use a cache you're just halfway. In fact, you must decide which part of the application is going to use the cache. Let's think about a web application backed by a database. Such web application usually involves Data Access Objects (DAOs), which access the relational database. Such objects are usually a bottleneck in the application as there is a lot of I/O going on. In other words, a cache can be used there. The cache can also be used by the business layer that has already aggregated and elaborated data retrieved from repositories, or it can be used by the presentation layer putting formatted presentation templates in the cache, or even by the authentication system that keeps roles according to an authenticated username. There are almost no limits as to how you can optimize an application and make it faster. The only price you pay is having RAM to dedicate the objects that are to be kept in memory, besides paying attention to the rules on how to manage life of the objects in cache. After these preliminary remarks, using a cache could seem common and obvious. A cache essentially acts as a hash into which key-value pairs are put. The keys are useful to retrieve objects from the cache. Caching usually has configuration parameters that'll allow you to change its behavior. Now let's have a look at an example with ehcache (http://ehcache.sourceforge.net). First of all let's configure it with the name methodCache so that we have at the most 1000 objects. The objects are inactive for a maximum of five minutes, with a maximum life of 10 minutes. If the objects count is over 1000, ehcache saves them on the filesystem, in java.io.tmpdir. <ehcache> ... <diskStore path="java.io.tmpdir"/> ... <defaultCache maxElementsInMemory="10000" eternal="false" timeToIdleSeconds="120" timeToLiveSeconds="120" overflowToDisk="true" diskPersistent="false" diskExpiryThreadIntervalSeconds="120" /> ... <cache name="methodCache" maxElementsInMemory="1000" eternal="false" overflowToDisk="false" timeToIdleSeconds="300" timeToLiveSeconds="600" /> </ehcache> Now let's create an CacheAspect . Let's define the cacheObject to which the ProceedingJoinPoint is passed. Let's recover an unambiguous key from the ProceedingJoinPoint with the method getCacheKey. We will use this key to put the objects in cache and to recover them. Once we have obtained the key, we ask to cache the Element with the instruction cache.get(cacheKey). The Element has to be evaluated because it may be null if the cache didn't find an Element with the passed cacheKey. If the Element is null, advice invokes the method proceed(), and puts in the cache the Element with the key corresponding to the invocation. Otherwise, if the Element recovered from the cache is not null, the method isn't invoked on the target class, and the value taken from the cache is given back to the caller. package org.springaop.chapter.five.cache; import it.springaop.utils.Constants; import net.sf.ehcache.Cache; import net.sf.ehcache.Element; import org.apache.log4j.Logger; import org.aspectj.lang.ProceedingJoinPoint; public class CacheAspect { public Object cacheObject(ProceedingJoinPoint pjp) throws Throwable { Object result; String cacheKey = getCacheKey(pjp); Element element = (Element) cache.get(cacheKey); logger.info(new StringBuilder("CacheAspect invoke:").append("n get:") .append(cacheKey).append(" value:").append(element).toString()); if (element == null) { result = pjp.proceed(); element = new Element(cacheKey, result); cache.put(element); logger.info(new StringBuilder("n put:").append(cacheKey).append( " value:").append(result).toString()); } return element.getValue(); } public void flush() { cache.flush(); } private String getCacheKey(ProceedingJoinPoint pjp) { String targetName = pjp.getTarget().getClass().getSimpleName(); String methodName = pjp.getSignature().getName(); Object[] arguments = pjp.getArgs(); StringBuilder sb = new StringBuilder(); sb.append(targetName).append(".").append(methodName); if ((arguments != null) && (arguments.length != 0)) { for (int i = 0; i < arguments.length; i++) { sb.append(".").append(arguments[i]); } } return sb.toString(); } public void setCache(Cache cache) { this.cache = cache; } private Cache cache; private Logger logger = Logger.getLogger(Constants.LOG_NAME); } Here is applicationContext.xml <beans xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd"> ... <bean id="rockerCacheAspect" class="org.springaop.chapter.five.cache.CacheAspect" > <property name="cache"> <bean id="bandCache" parent="cache"> <property name="cacheName" value="methodCache" /> </bean> </property> </bean> <!-- CACHE config --> <bean id="cache" abstract="true" class="org.springframework.cache.ehcache.EhCacheFactoryBean"> <property name="cacheManager" ref="cacheManager" /> </bean> <bean id="cacheManager" class="org.springframework.cache.ehcache. EhCacheManagerFactoryBean"> <property name="configLocation" value="classpath:org/springaop/chapter/five/cache/ehcache.xml" /> </bean> ... </beans> The idea about the caching aspect is to avoid repetition in our code base and have a consistent strategy for identifying objects (for example using the hash code of an object) so as to avoid objects from ending up in the cache twice. Employing an around advice, we can use the cache to make the method invocations give back the cached result of a previous invocation of the same method in a totally transparent way. In fact, to the methods of the classes defined in the interception rules in pointcuts will be given back the return values drawn from the cache or, if they are not present, they will be invoked and inserted in the cache. In this way, the classes and methods don't have any cognition of obtaining values retrieved from the cache.
Read more
  • 0
  • 0
  • 18570

article-image-mailing-spring-mail
Packt
04 Jun 2015
19 min read
Save for later

Mailing with Spring Mail

Packt
04 Jun 2015
19 min read
In this article, by Anjana Mankale, author of the book Mastering Spring Application Development we shall see how we can use the Spring mail template to e-mail recipients. We shall also demonstrate using Spring mailing template configurations using different scenarios. (For more resources related to this topic, see here.) Spring mail message handling process The following diagram depicts the flow of a Spring mail message process. With this, we can clearly understand the process of sending mail using a Spring mailing template. A message is created and sent to the transport protocol, which interacts with internet protocols. Then, the message is received by the recipients. The Spring mail framework requires a mail configuration, or SMTP configuration, as the input and message that needs to be sent. The mail API interacts with internet protocols to send messages. In the next section, we shall look at the classes and interfaces in the Spring mail framework. Interfaces and classes used for sending mails with Spring The package org.springframework.mail is used for mail configuration in the spring application. The following are the three main interfaces that are used for sending mail: MailSender: This interface is used to send simple mail messages. JavaMailSender: This interface is a subinterface of the MailSender interface and supports sending mail messages. MimeMessagePreparator: This interface is a callback interface that supports the JavaMailSender interface in the preparation of mail messages. The following classes are used for sending mails using Spring: SimpleMailMessage: This is a class which has properties such as to, from, cc, bcc, sentDate, and many others. The SimpleMailMessage interface sends mail with MailSenderImp classes. JavaMailSenderImpl: This class is an implementation class of the JavaMailSender interface. MimeMessageHelper: This class helps with preparing MIME messages. Sending mail using the @Configuration annotation We shall demonstrate here how we can send mail using the Spring mail API. First, we provide all the SMTP details in the .properties file and read it to the class file with the @Configuration annotation. The name of the class is MailConfiguration. mail.properties file contents are shown below: mail.protocol=smtp mail.host=localhost mail.port=25 mail.smtp.auth=false mail.smtp.starttls.enable=false mail.from=me@localhost mail.username= mail.password=   @Configuration @PropertySource("classpath:mail.properties") public class MailConfiguration { @Value("${mail.protocol}") private String protocol; @Value("${mail.host}") private String host; @Value("${mail.port}") private int port; @Value("${mail.smtp.auth}") private boolean auth; @Value("${mail.smtp.starttls.enable}") private boolean starttls; @Value("${mail.from}") private String from; @Value("${mail.username}") private String username; @Value("${mail.password}") private String password;   @Bean public JavaMailSender javaMailSender() {    JavaMailSenderImpl mailSender = new JavaMailSenderImpl();    Properties mailProperties = new Properties();    mailProperties.put("mail.smtp.auth", auth);    mailProperties.put("mail.smtp.starttls.enable", starttls);    mailSender.setJavaMailProperties(mailProperties);    mailSender.setHost(host);    mailSender.setPort(port);    mailSender.setProtocol(protocol);    mailSender.setUsername(username);    mailSender.setPassword(password);    return mailSender; } } The next step is to create a rest controller to send mail; to do so, click on Submit. We shall use the SimpleMailMessage interface since we don't have any attachment. @RestController class MailSendingController { private final JavaMailSender javaMailSender; @Autowired MailSubmissionController(JavaMailSender javaMailSender) {    this.javaMailSender = javaMailSender; } @RequestMapping("/mail") @ResponseStatus(HttpStatus.CREATED) SimpleMailMessage send() {    SimpleMailMessage mailMessage = new SimpleMailMessage();    mailMessage.setTo("packt@localhost");    mailMessage.setReplyTo("anjana@localhost");    mailMessage.setFrom("Sonali@localhost");    mailMessage.setSubject("Vani veena Pani");  mailMessage.setText("MuthuLakshmi how are you?Call      Me Please [...]");    javaMailSender.send(mailMessage);    return mailMessage; } } Sending mail using MailSender and Simple Mail Message with XML configuration "Simple mail message" means the e-mail sent will only be text-based with no HTML formatting, no images, and no attachments. In this section, consider a scenario where we are sending a welcome mail to the user as soon as the user gets their order placed in the application. In this scenario, the mail will be sent after the database insertion operation is successful. Create a separate folder, called com.packt.mailService, for the mail service. The following are the steps for sending mail using the MailSender interface and SimpleMailMessage class. Create a new Maven web project with the name Spring4MongoDB_MailChapter3. We have also used the same Eshop db database with MongoDB for CRUD operations on Customer, Order, and Product. We have also used the same mvc configurations and source files. Use the same dependencies as used previously. We need to add dependencies to the pom.xml file: <dependency> <groupId>org.springframework.integration</groupId> <artifactId>spring-integration-mail</artifactId> <version>3.0.2.RELEASE</version> <scope>runtime</scope> </dependency> <dependency> <groupId>javax.activation</groupId> <artifactId>activation</artifactId> <version>1.1-rev-1</version> <scope>runtime</scope> </dependency> <dependency> <groupId>javax.mail</groupId> <artifactId>mail</artifactId> <version>1.4.3</version> </dependency> Compile the Maven project. Create a separate folder called com.packt.mailService for the mail service. Create a simple class named MailSenderService and autowire the MailSender and SimpleMailMessage classes. The basic skeleton is shown here: public class MailSenderService { @Autowired private MailSender mailSender; @AutoWired private SimpleMailMessage simplemailmessage; public void sendmail(String from, String to, String    subject, String body){    /*Code */ }   } Next, create an object of SimpleMailMessage and set mail properties, such as from, to, and subject to it. public void sendmail(String from, String to, String subject, String body){ SimpleMailMessage message=new SimpleMailMessage(); message.setFrom(from); message.setSubject(subject); message.setText(body); mailSender.send(message); } We need to configure the SMTP details. Spring Mail Support provides this flexibility of configuring SMTP details in the XML file. <bean id="mailSender" class="org.springframework.mail.javamail. JavaMailSenderImpl"> <property name="host" value="smtp.gmail.com" /> <property name="port" value="587" /> <property name="username" value="username" /> <property name="password" value="password" />   <property name="javaMailProperties"> <props>    <prop key="mail.smtp.auth">true</prop>    <prop key="mail.smtp.starttls.enable">true</prop> </props> </property> </bean>   <bean id="mailSenderService" class=" com.packt.mailserviceMailSenderService "> <property name="mailSender" ref="mailSender" /> </bean>   </beans> We need to send mail to the customer after the order has been placed successfully in the MongoDB database. Update the addorder() method as follows: @RequestMapping(value = "/order/save", method = RequestMethod.POST) // request insert order recordh public String addorder(@ModelAttribute("Order")    Order order,Map<String, Object> model) {    Customer cust=new Customer();    cust=customer_respository.getObject      (order.getCustomer().getCust_id());      order.setCustomer(cust);    order.setProduct(product_respository.getObject      (order.getProduct().getProdid()));    respository.saveObject(order);    mailSenderService.sendmail      ("anjana.mprasad@gmail.com",cust.getEmail(),      "Dear"+cust.getName()+"Your order      details",order.getProduct().getName()+"-price-"+order      .getProduct().getPrice());    model.put("customerList", customerList);    model.put("productList", productList);    return "order"; } Sending mail to multiple recipients If you want to intimate the user regarding the latest products or promotions in the application, you can create a mail sending group and send mail to multiple recipients using Spring mail sending support. We have created an overloaded method in the same class, MailSenderService, which will accept string arrays. The code snippet in the class will look like this: public class MailSenderService { @Autowired private MailSender mailSender; @AutoWired private SimpleMailMessage simplemailmessage; public void sendmail(String from, String to, String subject,    String body){    /*Code */ }   public void sendmail(String from, String []to, String subject,    String body){    /*Code */ }   } The following is the code snippet for listing the set of users from MongoDB who have subscribed to promotional e-mails: public List<Customer> getAllObjectsby_emailsubscription(String    status) {    return mongoTemplate.find(query(      where("email_subscribe").is("yes")), Customer.class); } Sending MIME messages Multipurpose Internet Mail Extension (MIME) allows attachments to be sent over the Internet. This class just demonstrates how we can send mail with MIME messages. Using a MIME message sender type class is not advisible if you are not sending any attachments with the mail message. In the next section, we will look at the details of how we can send mail with attachments. Update the MailSenderService class with another method. We have used the MIME message preparator and have overridden the prepare method() to set properties for the mail. public class MailSenderService { @Autowired private MailSender mailSender; @AutoWired private SimpleMailMessage simplemailmessage;   public void sendmail(String from, String to, String subject,    String body){    /*Code */ } public void sendmail(String from, String []to, String subject,    String body){    /*Code */ } public void sendmime_mail(final String from, final String to,    final String subject, final String body) throws MailException{    MimeMessagePreparator message = new MimeMessagePreparator() {      public void prepare(MimeMessage mimeMessage)        throws Exception {        mimeMessage.setRecipient(Message.RecipientType.TO,new          InternetAddress(to));        mimeMessage.setFrom(new InternetAddress(from));        mimeMessage.setSubject(subject);        mimeMessage.setText(msg);    } }; mailSender.send(message); } Sending attachments with mail We can also attach various kinds of files to the mail. This functionality is supported by the MimeMessageHelper class. If you just want to send a MIME message without an attachment, you can opt for MimeMesagePreparator. If the requirement is to have an attachment to be sent with the mail, we can go for the MimeMessageHelper class with file APIs. Spring provides a file class named org.springframework.core.io.FileSystemResource, which has a parameterized constructor that accepts file objects. public class SendMailwithAttachment { public static void main(String[] args)    throws MessagingException {    AnnotationConfigApplicationContext ctx =      new AnnotationConfigApplicationContext();    ctx.register(AppConfig.class);    ctx.refresh();    JavaMailSenderImpl mailSender =      ctx.getBean(JavaMailSenderImpl.class);    MimeMessage mimeMessage = mailSender.createMimeMessage();    //Pass true flag for multipart message    MimeMessageHelper mailMsg = new MimeMessageHelper(mimeMessage,      true);    mailMsg.setFrom("ANJUANJU02@gmail.com");    mailMsg.setTo("RAGHY03@gmail.com");    mailMsg.setSubject("Test mail with Attachment");    mailMsg.setText("Please find Attachment.");    //FileSystemResource object for Attachment    FileSystemResource file = new FileSystemResource(new      File("D:/cp/ GODGOD. jpg"));    mailMsg.addAttachment("GODGOD.jpg", file);    mailSender.send(mimeMessage);    System.out.println("---Done---"); }   } Sending preconfigured mail In this example, we shall provide a message that is to be sent in the mail, and we will configure it in an XML file. Sometimes when it comes to web applications, you may have to send messages on maintenance. Think of a scenario where the content of the mail changes, but the sender and receiver are preconfigured. In such a case, you can add another overloaded method to the MailSender class. We have fixed the subject of the mail, and the content can be sent by the user. Think of it as "an application which sends mails to users whenever the build fails". <?xml version="1.0" encoding="UTF-8"?> <beans xsi_schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/ context/spring-context-3.0.xsd"> <context:component-scan base-package="com.packt" /> <!-- SET default mail properties --> <bean id="mailSender" class= "org.springframework.mail.javamail.JavaMailSenderImpl"> <property name="host" value="smtp.gmail.com"/> <property name="port" value="25"/> <property name="username" value="anju@gmail.com"/> <property name="password" value="password"/> <property name="javaMailProperties"> <props>    <prop key="mail.transport.protocol">smtp</prop>    <prop key="mail.smtp.auth">true</prop>    <prop key="mail.smtp.starttls.enable">true</prop>    <prop key="mail.debug">true</prop> </props> </property> </bean>   <!-- You can have some pre-configured messagess also which are ready to send --> <bean id="preConfiguredMessage" class= "org.springframework.mail.SimpleMailMessage"> <property name="to" value="packt@gmail.com"></property> <property name="from" value="anju@gmail.com"></property> <property name="subject" value="FATAL ERROR- APPLICATION AUTO    MAINTENANCE STARTED-BUILD FAILED!!"/> </bean> </beans> Now we shall sent two different bodies for the subjects. public class MyMailer { public static void main(String[] args){    try{      //Create the application context      ApplicationContext context = new        FileSystemXmlApplicationContext(        "application-context.xml");        //Get the mailer instance      ApplicationMailer mailer = (ApplicationMailer)        context.getBean("mailService");      //Send a composed mail      mailer.sendMail("nikhil@gmail.com", "Test Subject",        "Testing body");    }catch(Exception e){      //Send a pre-configured mail      mailer.sendPreConfiguredMail("build failed exception occured        check console or logs"+e.getMessage());    } } } Using Spring templates with Velocity to send HTML mails Velocity is the templating language provided by Apache. It can be integrated into the Spring view layer easily. The latest Velocity version used during this book is 1.7. In the previous section, we demonstrated using Velocity to send e-mails using the @Bean and @Configuration annotations. In this section, we shall see how we can configure Velocity to send mails using XML configuration. All that needs to be done is to add the following bean definition to the .xml file. In the case of mvc, you can add it to the dispatcher-servlet.xml file. <bean id="velocityEngine" class= "org.springframework.ui.velocity.VelocityEngineFactoryBean"> <property name="velocityProperties"> <value>    resource.loader=class    class.resource.loader.class=org.apache.velocity    .runtime.resource.loader.ClasspathResourceLoader </value> </property> </bean> Create a new Maven web project with the name Spring4MongoDB_Mail_VelocityChapter3. Create a package and name it com.packt.velocity.templates. Create a file with the name orderconfirmation.vm. <html> <body> <h3> Dear Customer,<h3> <p>${customer.firstName} ${customer.lastName}</p> <p>We have dispatched your order at address.</p> ${Customer.address} </body> </html> Use all the dependencies that we have added in the previous sections. To the existing Maven project, add this dependency: <dependency> <groupId>org.apache.velocity</groupId> <artifactId>velocity</artifactId> <version>1.7</version> </dependency> To ensure that Velocity gets loaded on application startup, we shall create a class. Let's name the class VelocityConfiguration.java. We have used the annotations @Configuration and @Bean with the class. import java.io.IOException; import java.util.Properties;   import org.apache.velocity.app.VelocityEngine; import org.apache.velocity.exception.VelocityException; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.ui.velocity.VelocityEngineFactory; @Configuration public class VelocityConfiguration { @Bean public VelocityEngine getVelocityEngine() throws VelocityException, IOException{    VelocityEngineFactory velocityEngineFactory = new      VelocityEngineFactory();    Properties props = new Properties();    props.put("resource.loader", "class");    props.put("class.resource.loader.class",      "org.apache.velocity.runtime.resource.loader." +      "ClasspathResourceLoader");    velocityEngineFactory.setVelocityProperties(props);    return factory.createVelocityEngine(); } } Use the same MailSenderService class and add another overloaded sendMail() method in the class. public void sendmail(final Customer customer){ MimeMessagePreparator preparator = new    MimeMessagePreparator() {    public void prepare(MimeMessage mimeMessage)    throws Exception {      MimeMessageHelper message =        new MimeMessageHelper(mimeMessage);      message.setTo(user.getEmailAddress());      message.setFrom("webmaster@packt.com"); // could be        parameterized      Map model = new HashMap();      model.put("customer", customer);      String text =        VelocityEngineUtils.mergeTemplateIntoString(        velocityEngine, "com/packt/velocity/templates/        orderconfirmation.vm", model);      message.setText(text, true);    } }; this.mailSender.send(preparator); } Update the controller class to send mail using the Velocity template. @RequestMapping(value = "/order/save", method = RequestMethod.POST) // request insert order recordh public String addorder(@ModelAttribute("Order") Order order,Map<String, Object> model) { Customer cust=new Customer(); cust=customer_respository.getObject(order.getCustomer()    .getCust_id());   order.setCustomer(cust); order.setProduct(product_respository.getObject    (order.getProduct().getProdid())); respository.saveObject(order); // to send mail using velocity template. mailSenderService.sendmail(cust);   return "order"; } Sending Spring mail over a different thread There are other options for sending Spring mail asynchronously. One way is to have a separate thread to the mail sending job. Spring comes with the taskExecutor package, which offers us a thread pooling functionality. Create a class called MailSenderAsyncService that implements the MailSender interface. Import the org.springframework.core.task.TaskExecutor package. Create a private class called MailRunnable. Here is the complete code for MailSenderAsyncService: public class MailSenderAsyncService implements MailSender{ @Resource(name = "mailSender") private MailSender mailSender;   private TaskExecutor taskExecutor;   @Autowired public MailSenderAsyncService(TaskExecutor taskExecutor){    this.taskExecutor = taskExecutor; } public void send(SimpleMailMessage simpleMessage) throws    MailException {    taskExecutor.execute(new MailRunnable(simpleMessage)); }   public void send(SimpleMailMessage[] simpleMessages)    throws MailException {    for (SimpleMailMessage message : simpleMessages) {      send(message);    } }   private class SimpleMailMessageRunnable implements    Runnable {    private SimpleMailMessage simpleMailMessage;    private SimpleMailMessageRunnable(SimpleMailMessage      simpleMailMessage) {      this.simpleMailMessage = simpleMailMessage;    }      public void run() {    mailSender.send(simpleMailMessage);    } } private class SimpleMailMessagesRunnable implements    Runnable {    private SimpleMailMessage[] simpleMessages;    private SimpleMailMessagesRunnable(SimpleMailMessage[]      simpleMessages) {      this.simpleMessages = simpleMessages;    }      public void run() {      mailSender.send(simpleMessages);    } } } Configure the ThreadPool executor in the .xml file. <bean id="taskExecutor" class="org.springframework. scheduling.concurrent.ThreadPoolTaskExecutor" p_corePoolSize="5" p_maxPoolSize="10" p_queueCapacity="100"    p_waitForTasksToCompleteOnShutdown="true"/> Test the source code. import javax.annotation.Resource;   import org.springframework.mail.MailSender; import org.springframework.mail.SimpleMailMessage; import org.springframework.test.context.ContextConfiguration;   @ContextConfiguration public class MailSenderAsyncService { @Resource(name = " mailSender ") private MailSender mailSender; public void testSendMails() throws Exception {    SimpleMailMessage[] mailMessages = new      SimpleMailMessage[5];      for (int i = 0; i < mailMessages.length; i++) {      SimpleMailMessage message = new SimpleMailMessage();      message.setSubject(String.valueOf(i));      mailMessages[i] = message;    }    mailSender.send(mailMessages); } public static void main (String args[]){    MailSenderAsyncService asyncservice=new      MailSenderAsyncService();    Asyncservice. testSendMails(); } } Sending Spring mail with AOP We can also send mails by integrating the mailing functionality with Aspect Oriented Programming (AOP). This can be used to send mails after the user registers with an application. Think of a scenario where the user receives an activation mail after registration. This can also be used to send information about an order placed on an application. Use the following steps to create a MailAdvice class using AOP: Create a package called com.packt.aop. Create a class called MailAdvice. public class MailAdvice { public void advice (final ProceedingJoinPoint    proceedingJoinPoint) {    new Thread(new Runnable() {    public void run() {      System.out.println("proceedingJoinPoint:"+        proceedingJoinPoint);      try {        proceedingJoinPoint.proceed();      } catch (Throwable t) {        // All we can do is log the error.         System.out.println(t);      }    } }).start(); } } This class creates a new thread and starts it. In the run method, the proceedingJoinPoint.proceed() method is called. ProceddingJoinPoint is a class available in AspectJ.jar. Update the dispatcher-servlet.xml file with aop configurations. Update the xlmns namespace using the following code: advice"> <aop:around method="fork"    pointcut="execution(* org.springframework.mail    .javamail.JavaMailSenderImpl.send(..))"/> </aop:aspect> </aop:config> Summary In this article, we demonstrated how to create a mailing service and configure it using Spring API. We also demonstrated how to send mails with attachments using MIME messages. We also demonstrated how to create a dedicated thread for sending mails using ExecutorService. We saw an example in which mail can be sent to multiple recipients, and saw an implementation of using the Velocity engine to create templates and send mails to recipients. In the last section, we demonstrated how the Spring framework supported mails can be sent using Spring AOP and threads. Resources for Article: Further resources on this subject: Time Travelling with Spring [article] Welcome to the Spring Framework [article] Creating a Spring Application [article]
Read more
  • 0
  • 0
  • 18567

article-image-securing-your-elastix-system
Packt
31 Mar 2015
19 min read
Save for later

Securing your Elastix System

Packt
31 Mar 2015
19 min read
In the Article by Gerardo Barajas Puente, author of Elastix Unified Communications Server Cookbook, we will discuss some topics regarding security in our Elastix Unified Communications System. We will share some recommendations to ensure our system's availability, privacy, and correct performance. Attackers' objectives may vary from damaging data, to data stealing, to telephonic fraud, to denial of service. This list is intended to minimize any type of attack, but remember that there are no definitive arguments about security; it is a constantly changing subject with new types of attacks, challenges, and opportunities. (For more resources related to this topic, see here.) The recipes covered in this article are as follows: Using Elastix's embedded firewall Using the Security Advanced Settings menu to enable security features Recording and monitoring calls Recording MeetMe rooms (conference rooms) Recording queues' calls Monitoring recordings Upgrading our Elastix system Generating system backups Restoring a backup from one server to another Using Elastix's embedded firewall Iptables is one of the most powerful tools of Linux's kernel. It is used largely in servers and devices worldwide. Elastix's security module incorporates iptables' main features into its webGUI in order to secure our Unified Communications Server. This module is available in the Security | Firewall menu. In this module's main screen, we can check the status of the firewall (Activated or Deactivated). We will also notice the status of each rule of the firewall with the following information: Order: This column represents the order in which rules will be applied Traffic: The rule will be applied to any ingoing or outgoing packet Target: This option allows, rejects, or drops a packet Interface: This represents the network interface on which the rule will be used Source Address: The firewall will search for this IP source address and apply the rule. Destination Address: We can apply a firewall rule if the destination address is matched Protocol: We can apply a rule depending on the IP protocol of the packet (TCP, UDP, ICMP, and so on) Details: In this column, the details or comments regarding this rule may appear in order to remind us of why this rule is being applied By default, when the firewall is applied, Elastix will allow the traffic from any device to use the ports that belong to the Unified Communications Services. The next image shows the state of the firewall. We can review this information in the Define Ports section as shown in the next image: In this section, we can delete, define a new rule (or port), or search for a specific port. If we click on the View link, we will be redirected to the editing page for the selected rule as shown in the next picture. This is helpful whenever we would like to change the details of a rule. How to do it… To add a new rule, click on the Define Port link and add the following information as shown in the next image:     Name: Name for this port.     Protocol: We can choose the IP protocol to use. The options are as follows: TCP, ICMP, IP, and UDP.     Port: We can enter a single port or a range of ports. To enter a port we just enter the port number in the first text field before the ":" character. If we'd like to enter a range, we must use the two text areas. The first one is for the first port of the range, and the second one is for the last port of the range.     Comment: We can enter a comment for this port. The next image shows the creation of a new port for GSM-Solution. This solution will use the TCP protocol from port 5000 to 5002. Having our ports defined, we proceed to activate the firewall by clicking on Save. As soon as the firewall service is activated, we will see the status of every rule. A message will be displayed, informing us that the service has been activated. When the service has been started, we will be able to edit, eliminate or change the execution order of a certain rule or rules. To add a new rule, click on the New Rule button (as shown in the next picture) and we will be redirected to a new web page. The information we need to enter is as follows:     Traffic: This option sets the rule for incoming (INPUT), outgoing (OUTPUT), or redirecting (FORWARD) packets.     Interface IN: This is the interface used for the rule. All the available network interfaces will be listed. The options ANY and LOOPBACK are also available     Source Address: We can apply a rule for any specified IP address. For example, we can block all the incoming traffic from the IP address 192.168.1.1. It is important to specify its netmask.     Destination Address: This is the destination IP address for the rule. It is important to specify its netmask.     Protocol: We can choose the protocol we would like to filter or forward. The options are TCP, UDP, ICMP, IP, and STATE.     Source Port: In this section, we can choose any option previously configured in the Port Definition section for the source port.     Destination Port: Here, we can select any option previously configured in the Port Definition section for the source port.     Target: This is the action to perform for any packet that matches any of the conditions set in the previous fields The next image shows the application of a new firewall's rule based on the ports we defined previously: We can also check the user's activity by using the Audit menu. This module can be found in the Security menu. To enhance our system's security we also recommend using Elastix's internal Port Knocking feature. Using the Security Advanced Settings menu to enable security features The Advanced Settings option will allow us to perform the following actions: Enable or disable direct access to FreePBX's webGUI. Enable or disable anonymous SIP calls. Change the database and web administration password for FreePBX. How to do it… Click on the Security | Advanced Settings menu and these options are shown as in the next screenshot. Recording and monitoring calls Whenever we have the need for recording the calls that pass through our system, Elastix, and taking advantage of FreePBX's and Asterisk's features. In this section, we will show the configuration steps to record the following types of calls: Extension's inbound and outbound calls MeetMe rooms (conference rooms) Queues Getting ready... Go to PBX | PBX Configuration | General Settings. In the section called Dialing Options, add the values w and W to the Asterisk Dial command options and the Asterisk Outbound Dial command options. These values will allow the users to start recording after pressing *1. The next screenshot shows this configuration. The next step is to set the options from the Call Recording section as follows: Extension recording override: Disabled. If enabled, this option will ignore all automatic recording settings for all extensions. Call recording format: We can choose the audio format that the recording files will have. We recommend the wav49 format because it is compact and the voice is understandable despite the audio quality. Here is a brief description for the audio file format: WAV: This is the most popular good quality recording format, but its size will increase by 1 MB per minute. WAV49: This format results from a GSM codec recording under the WAV encapsulation making the recording file smaller: 100 KB per minute. Its quality is similar to that of a mobile phone call. ULAW/ALAW: This is the native codec (G.711) used between TELCOS and users, but the file size is very large (1 MB per minute). SLN: SLN means SLINEAR format, which is Asterisk's native format. It is an 8-kHz, 16-bit signer linear raw format. GSM: This format is used for recording calls by using the GSM codec. The recording file size will be increased at a rate of 100 KB per minute. Recording location: We leave this option blank. This option specifies the folder where our recordings will be stored. By default, our system is configured to record calls in the /var/spool/asterisk/monitor folder. Run after record: We also leave this option blank. This is for running a script after a recording has been done. For more information about audio formats, visit: http://www.voip-info.org/wiki/view/Convert+WAV+audio+files+for+use+in+Asterisk Apply the changes. All these options are shown in the next screenshot: How to do it… To record all the calls that are generated or received from or to extensions go to the extension's details in the module: PBX | PBX Configuration. We have to click on the desired extension we would like to activate its call recording. In the Recording Options section, we have two options:     Record Incoming     Record Outgoing Depending on the type of recording, select from one of the following options:     On Demand: In this option, the user must press *1 during a call to start recording it. This option only lasts for the current call. When this call is terminated, if the user wants to record another, the digits *1 must be pressed again. If *1 is pressed during a call that is being recorded, the recording will be stopped.     Always: All the calls will be recorded automatically.     Never: This option disables all call recording. These options are shown in the next image. Recording MeetMe rooms If we need to record the calls that go to a conference room, Elastix allows us to do this. This feature is very helpful whenever we need to remember the topics discussed in a conference. How to do it… To record the calls of a conference room, enable it at the conference's details. These details are found in the menu: PBX | PBX Configuration | Conferences. Click on the conference we would like to record and set the Record Conference option to Yes. Save and apply the changes. These steps are shown in the next image. Recording queues' calls Most of the time, the calls that arrive in a queue must be recorded for quality and security purposes. In this recipe, we will show how to enable this feature. How to do it… Go to PBX | PBX Configuration | Queues. Click on a queue to record its calls. Search for the Call Recording option. Select the recording format to use (wav49, wav, gsm). Save and apply the changes. The following image shows the configuration of this feature. Monitoring recordings Now that we know how to record calls, we will show how to retrieve them in order to listen them. How to do it… To visualize the recorded calls, go to PBX | Monitoring. In this menu, we will be able to see the recordings stored in our system. The displayed columns are as follows:     Date: Date of call     Time: Time of call     Source: Source of call (may be an internal or external number)     Destination: Destination of call (may be an internal or external number)     Duration: Duration of call     Type: Incoming or outgoing     Message: This column sets the Listen and Download links to enable you to listen or download the recording files. To listen to a recording, just click on the Message link and a new window will popup in your web browser. This window will have the options to playback the selected recording. It is important to enable our web browser to reproduce audio. To download a recording, we click on the Download link. To delete a recording or group of recordings, just select them and click on the Delete button. To search for a recording or set of recordings, we can do it by date, source, destination, or type, by clicking on the Show Filter button. If click on the Download button, we can download the search or report of the recording files in any of the following formats: CSV, Excel, or Text. It is very important to regularly check the Hard Disk status to prevent it from getting full of recording files and therefore have insufficient space to allow the main services work efficiently. Encrypting voice calls In Elastix/Asterisk, the SIP calls can be encrypted in two ways: encrypting the SIP protocol signaling and encrypting the RTP voice flow. To encrypt the SIP protocol signal, we will use the Transport Layer Security (TLS) protocol. How to do it… Create security keys and certificates. For this example, we will store our keys and certificates in the /etc/asterisk/keys folder. To create this folder, enter the mkdir /etc/asterisk/keys command. Change the owner of the folder from the user root to the user asterisk: chown asterisk:asterisk /etc/asterisk/keys Generate the keys and certificates by going to the following folder: cd /usr/share/doc/asterisk-1.8.20.0/contrib/scripts/   ./ast_tls_cert -C 10.20.30.70 -O "Our Company" -d /etc/asterisk/keys Where the options are as follows:     -C is used to set the host (DNS name) or IP address of our Elastix server.     -O is the organizational name or description.     -d is the folder where keys will be stored. Generate a pair of keys for a pair of extensions (extension 7002 and extension 7003, for example):     For extension 7002: ./ast_tls_cert -m client -c /etc/asterisk/keys/ca.crt -k /etc/asterisk/keys/ca.key -C 10.20.31.107 -O "Elastix Company" -d /etc/asterisk/keys -o 7002     And for extension 7003 ./ast_tls_cert -m client -c /etc/asterisk/keys/ca.crt -k /etc/asterisk/keys/ca.key -C 10.20.31.106 -O "Elastix Company" -d /etc/asterisk/keys -o 7003 where:     -m client: This option sets the program to create a client certificate.     -c /etc/asterisk/keys/ca.crt: This option specifies the Certificate Authority to use (our IP-PBX).     -k /etc/asterisk/keys/ca.key: Provides the key file to the *.crt file.     -C: This option defines the hostname or IP address of our SIP device.     -O: This option defines the organizational name (same as above).     -d: This option specifies the directory where the keys and certificates will be stored.     -o: This is the name of the key and certificate we are creating. When creating the client's keys and certificates, we must enter the same password set when creating the server's certificates. Configure the IP-PBX to support TLS by editing the sip_general_custom.conf file located in the /etc/asterisk/ folder. Add the following lines: tlsenable=yes tlsbindaddr=0.0.0.0 tlscertfile=/etc/asterisk/keys/asterisk.pem tlscafile=/etc/asterisk/keys/ca.crt tlscipher=ALL tlsclientmethod=tlsv1 tlsdontverifyserver=yes     These lines are in charge of enabling the TLS support in our IP-PBX. They also specify the folder where the certificates and the keys are stored and set the ciphering option and client method to use. Add the line transport=tls to the extensions we would like to use TLS in the sip_custom.conf file located at /etc/asterisk/. This file should look like: [7002](+) encryption=yes transport=tls   [7003](+) encryption=yes transport=tls Reload the SIP module in the Asterisk service. This can be done by using the command: asterisk -rx 'sip reload' Configure our TLS-supporting IP phones. This configuration varies from model to model. It is important to mention that the port used for TLS and SIP is port 5061; therefore, our devices must use TCP/UDP port 5061. After our devices are registered and we can call each other, we can be sure this configuration is working. If we issue the command asterisk -rx 'sip show peer 7003', we will see that the encryption is enabled. At this point, we've just enabled the encryption at the SIP signaling level. With this, we can block any unauthorized user depending on which port the media (voice or/and video) is being transported or steal a username or password or eavesdrop a conversation. Now, we will proceed to enable the audio/video (RTP) encryption. This term is also known as Secure Real Time Protocol (SRTP). To do this, we only enable on the SIP peers the encryption=yes option. The screenshot after this shows an SRTP call between peers 7002 and 7003. This information can be displayed with the command: asterisk -rx 'sip show channel [the SIP channel of our call] The line RTP/SAVP informs us that the call is secure, and the call in the softphone shows an icon with the form of a lock confirming that the call is secure. The following screenshot shows the icon of a lock, informing us that the current call is secured through SRTP: We can have the SRTP enabled without enabling TLS, and we can even activate TLS support between SIP trunks and our Elastix system. There is more… To enable the IAX encryption in our extensions and IAX trunks, add the following line to their configuration file (/etc/asterisk/iax_general_ custom.conf): encryption=aes128 Reload the IAX module with the command: iax2 reload If we would like to see the encryption in action, configure the debug output in the logger.conf file and issue the following CLI commands: CLI> set debug 1 Core debug is at least 1 CLI> iax2 debug IAX2 Debugging Enabled Generating system backups Generating system backups is a very important activity that helps us to restore our system in case of an emergency or failure. The success of our Elastix platform depends on how quickly we can restore our system. In this recipe, we will cover the generation of backups. How to do it… To perform a backup on our Elastix UCS, go to the System | Backup/Restore menu. When entering this module, the first screen that we will see shows all the backup files available and stored in our system, the date they have been created, and the possibility to restore any of them. If we click on any of them, we can download it on to our laptop, tablet, or any device that will allow us to perform a full backup restore, in the event of a disaster. The next screenshot shows the list of backups available on a system. If we select a backup file from the main view, we can delete it by clicking on the Delete button. To create a backup, click on the Perform a Backup button. Select what modules (with their options) will be saved. Click on the Process button to start the backup process on our Elastix box. When done, a message will be displayed informing us that the process has been completed successfully. We can automate this process by clicking on Set Automatic Backup after selecting this option when this process will be started: Daily, Weekly, or Monthly. Restoring a backup from one server to another If we have a backup file, we can copy it to another recently installed Elastix Unified Communications Server, if we'd like to restore it. For example, Server A is a production server, but we'd like to use a brand new server with more resources (Server B). How to do it… After having Elastix installed in Server B, perform a backup, irrespective of whether there is no configuration in it and create a backup in Server A as well. Then, we copy the backup (*.tar file) from Server A to Server B with the console command (being in Server A's console): scp /var/www/backup/back-up-file.tar root@ip-address-of-server-b:/var/www/backup/ Log into Server B's console and change the ownership of the backup file with the command: chown asterisk:asterisk /var/www/backup/back-up-file.tar Restore the copied backup in Server B by using the System | Backup/Restore menu. When this process is being done, Elastix's webGUI will alert us of a restoring process being performed and it will show if there is any software difference between the backup and our current system. We recommend the use of the same Admin and Root passwords and the same telephony hardware in both servers. After this operation is done, we have to make sure that all configurations are working on the new server, before going on production. There is more… If we click on the FTP Backup option, we can drag and drop any selected backup to upload it to a remote FTP server or we can download it locally. We only need to set up the correct data to log us into the remote FTP server. The data to enter are as follows: Server FTP: IP address or domain name of the remote FTP server Port: FTP port User: User Password: Password Path Server FTP: Folder or directory to store the backup The next screenshot shows the FTP-Backup menu and options: Although securing systems is a very important and sometimes difficult area that requires a high level of knowledge, in this article, we discussed the most common but effective tasks that should be done in order to keep your Elastix Unified Communications System healthy and secure. Summary The main objective of this article is to give you all the necessary tools to configure and support an Elastix Unified Communications Server. We will look at these tools through Cookbook recipes, just follow the steps to get an Elastix System up and running. Although a good Linux and Asterisk background is required, this article is structured to help you grow from a beginner to an advanced user. Resources for Article: Further resources on this subject: Lync 2013 Hybrid and Lync Online [article] Creating an Apache JMeter™ test workbench [article] Innovation of Communication and Information Technologies [article]
Read more
  • 0
  • 0
  • 18566
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-acting-proxy-httpproxymodule
Packt
24 Dec 2013
9 min read
Save for later

Acting as a proxy (HttpProxyModule)

Packt
24 Dec 2013
9 min read
(For more resources related to this topic, see here.) The HttpProxyModule allows Nginx to act as a proxy and pass requests to another server. location / {   proxy_pass        http://app.localhost:8000; } Note when using the HttpProxyModule (or even when using FastCGI), the entire client request will be buffered in Nginx before being passed on to the proxy server. Explaining directives Some of the important directives of the HttpProxyModule are as follows. proxy_pass The proxy_pass directive sets the address of the proxy server and the URI to which the location will be mapped. The address may be given as a hostname or an address and port, for example: proxy_pass http://localhost:8000/uri/; Or, the address may be given as an UNIX socket path: proxy_pass http://unix:/path/to/backend.socket:/uri/; path is given after the word unix between two colons. You can use the proxy_pass directive to forward headers from the client request to the proxied server. proxy_set_header Host $host; While passing requests, Nginx replaces the location in the URI with the location specified by the proxy_pass directive. If inside the proxied location, URI is changed by the rewrite directive and this configuration will be used to process the request. For example: location  /name/ {   rewrite      /name/([^/] +)  /users?name=$1  break;   proxy_pass   http://127.0.0.1; } A request URI is passed to the proxy server after normalization as follows: Double slashes are replaced by a single slash Any references to current directory like "./" are removed Any references to the previous directory like "../" are removed. If proxy_pass is specified without a URI (for example in "http://example.com/request",/request is the URI part), the request URI is passed to the server in the same form as sent by a client location /some/path/ {   proxy_pass http://127.0.0.1; } If you need the proxy connection to an upstream server group to use SSL, your proxy_pass rule should use https:// and you will also have to set your SSL port explicitly in the upstream definition. For example: upstream https-backend {   server 10.220.129.20:443; }   server {   listen 10.220.129.1:443;   location / {     proxy_pass https://backend-secure;   } } proxy_pass_header The proxy_pass_header directive allows transferring header lines forbidden for response. For example: location / {   proxy_pass_header X-Accel-Redirect; } proxy_connect_timeout The proxy_connect_timeout directive sets a connection timeout to the upstream server. You can't set this timeout value to be more than 75 seconds. Please remember that this is not the response timeout, but only a connection timeout. This is not the time until the server returns the pages which is configured through proxy_read_timeout directive. If your upstream server is up but hanging, this statement will not help as the connection to the server has been made. proxy_next_upstream The proxy_next_upstream directive determines in which cases the request will be transmitted to the next server: error: An error occurred while connecting to the server, sending a request to it, or reading its response timeout: The timeout occurred during the connection with the server, transferring the request, or while reading the response from the server invalid_header: The server returned an empty or incorrect response http_500: The server responded with code 500 http_502: The server responded with code 502 http_503: The server responded with code 503 http_504: The server responded with code 504 http_404: The server responded with code 404 off: Disables request forwarding Transferring the request to the next server is only possible if there is an error sending the request to one of the servers. If the request sending was interrupted due to an error or some other reason, the transfer of request will not take place. proxy_redirect The proxy_redirect directive allows you to manipulate the HTTP redirection by replacing the text in the response from the upstream server. Specifically, it replaces text in the Location and Refresh headers. The HTTP Location header field is returned in response from a proxied server for the following reasons: To indicate that a resource has moved temporarily or permanently. To provide information about the location of a newly created resource. This could be the result of an HTTP PUT. Let us suppose that the proxied server returned the following: Location: http://localhost:8080/images/new_folder If you have the proxy_redirect directive set to the following: proxy_redirect http://localhost:8080/images/ http://xyz/; The Location text will be rewritten to be similar to the following: Location: http://xyz/new_folder/. It is possible to use some variables in the redirected address: proxy_redirect http://localhost:8000/ http://$location:8000; You can also use regular expressions in this directive: proxy_redirect ~^(http://[^:]+):d+(/.+)$ $1$2; The value off disables all the proxy_redirect directives at its level. proxy_redirect off; proxy_set_header The proxy_set_header directive allows you to redefine and add new HTTP headers to the request sent to the proxied server. You can use a combination of static text and variables as the value of the proxy_set_header directive. By default, the following two headers will be redefined: proxy_set_header Host $proxy_host; proxy_set_header Connection Close; You can forward the original Host header value to the server as follows: proxy_set_header Host $http_host; However, if this header is absent in the client request, nothing will be transferred. It is better to use the variable $host; its value is equal to the request header Host or to the basic name of the server in case the header is absent from the client request. proxy_set_header Host $host; You can transmit the name of the server together with the port of the proxied server: proxy_set_header Host $host:$proxy_port; If you set the value to an empty string, the header is not passed to the upstream proxied server. For example, if you want to disable the gzip compression on upstream, you can do the following: proxy_set_header  Accept-Encoding  ""; proxy_store The proxy_store directive sets the path in which upstream files are stored, with paths corresponding to the directives alias or root. The off directive value disables local file storage. Please note that proxy_store is different from proxy_cache. It is just a method to store proxied files on disk. It may be used to construct cache-like setups (usually involving error_page-based fallback). This proxy_store directive parameter is off by default. The value can contain a mix of static strings and variables. proxy_store   /data/www$uri; The modification date of the file will be set to the value of the Last-Modified header in the response. A response is first written to a temporary file in the path specified by proxy_temp_path and then renamed. It is recommended to keep this location path and the path to store files the same to make sure it is a simple renaming instead of creating two copies of the file. Example: location /images/ {   root                 /data/www;   error_page           404 = @fetch; }   location /fetch {   internal;   proxy_pass           http://backend;   proxy_store          on;   proxy_store_access   user:rw  group:rw  all:r;   proxy_temp_path      /data/temp;   alias                /data/www; } In this example, proxy_store_access defines the access rights of the created file. In the case of an error 404, the fetch internal location proxies to a remote server and stores the local copies in the /data/temp folder. proxy_cache The proxy_cache directive either turns off caching when you use the value off or sets the name of the cache. This name can then be used subsequently in other places as well. Let's look at the following example to enable caching on the Nginx server: http {   proxy_cache_path  /var/www/cache levels=1:2 keys_zone=my-     cache:8m max_size=1000m inactive=600m;   proxy_temp_path /var/www/cache/tmp;   server {     location / {       proxy_pass http://example.net;       proxy_cache my-cache;       proxy_cache_valid  200 302  60m;       proxy_cache_valid  404      1m;     }   } } The previous example creates a named cache called my-cache. It sets up the validity of the cache for response codes 200 and 302 to 60m, and for 404 to 1m, respectively. The cached data is stored in the /var/www/cache folder. The levels parameter sets the number of subdirectory levels in the cache. You can define up to three levels. The name of key_zone is followed by an inactive interval. All the inactive items in my-cache will be purged after 600m. The default value for inactive intervals is 10 minutes.   Chapter 5 of the book, Creating Your Own Module, is inspired by the work from Mr. Evan Miller which can be found at http://www.evanmiller.org/nginx-modules-guide.html.   Summary In this article we looked at several standard HTTP modules. These modules provide a very rich set of functionalities by default. You can disable these modules if you please at the time of configuration. However, they will be installed by default if you don't. The list of modules and their directives in this chapter is by no means exhaustive. Nginx's online documentation can provide you with more details. Resources for Article: Introduction to nginx [Article] Nginx Web Services: Configuration and Implementation [Article] Using Nginx as a Reverse Proxy [Article]
Read more
  • 0
  • 2
  • 18563

article-image-chatgpt-prompting-basics-finding-your-ip-address
Clint Bodungen
18 Oct 2023
6 min read
Save for later

ChatGPT Prompting Basics: Finding Your IP Address

Clint Bodungen
18 Oct 2023
6 min read
Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!This article is an excerpt from the book, ChatGPT for Cybersecurity Cookbook, by Clint Bodungen. Master ChatGPT and the OpenAI API, and harness the power of cutting-edge generative AI and large language models to revolutionize the way you perform penetration testing, threat detection, and risk assessment.IntroductionIn this article, we will explore the basics of ChatGPT prompting using the ChatGPT interface, which is different from the OpenAI Playground we used in the previous recipe. The advantage of using the ChatGPT interface is that it does not consume account credits and is better suited for generating formatted output, such as writing code or creating tables. Getting ready To use the ChatGPT interface, you will need to have an active OpenAI account. If you haven't already, please set up your ChatGPT account. How to do it… In this recipe, we'll guide you through using the ChatGPT interface to generate a Python script that retrieves a user's public IP address. By following these steps, you'll learn how to interact with ChatGPT in a conversation-like manner and receive context-aware responses, including code snippets. Now, let's proceed with the steps in this recipe: 1. In your browser, go to https://chat.openai.com and click “Log in” 2. Log in using your OpenAI credentials. 3. Once you are logged in, you will be taken to the ChatGPT interface. The interface is similar to a chat application, with a text box at the bottom where you can enter your prompts.  Figure – The ChatGPT interface 4. ChatGPT uses a conversation-based approach, so you can simply type your prompt as a message and press "Enter" or click the       button to receive a response from the model. For example, you can ask ChatGPT to generate a piece of Python code to find the public IP address of a user:  Figure – Entering a prompt ChatGPT will generate a response containing the requested Python code, along with a thorough explanation.  Figure – ChatGPT response with code 5. Continue the conversation by asking follow-up questions or providing additional information, and ChatGPT will respond accordingly.  Figure – ChatGPT contextual follow-up response 6. Run the ChatGPT generated code by clicking on “Copy code”, paste it into your code editor of choice (I personally use Visual Studio Code), save it as a “.py” Python script, and run from a terminal. PS D:\GPT\ChatGPT for Cybersecurity Cookbook> python .\my_ip.py Your public IP address is:  Your local network IP address is: 192.168.1.105 Figure – Running the ChatGPT generated script  How it works… By using the ChatGPT interface to enter prompts, you can generate context-aware responses and content that continues over the course of an entire conversation like a chatbot. The conversation-based approach allows for more natural interactions and the ability to ask follow-up questions or provide additional context. The responses can even include complex formatting such as code snippets or tables (more on tables later). There’s more… As you become more familiar with ChatGPT, you can experiment with different prompt styles, instructions, and contexts to obtain the desired output for your cybersecurity tasks. You can also compare the results generated through the ChatGPT interface and the OpenAI Playground to determine which approach best fits your needs. Tip:You can further refine the generated output by providing very clear and specific instructions or using roles. It also helps to divide complex prompts into several smaller prompts, giving ChatGPT one instruction per prompt, building on the previous prompts as you go. In the upcoming recipes, we will delve into more advanced prompting techniques that utilize these techniques to help you get the most accurate and detailed responses from ChatGPT. As you interact with ChatGPT, your conversation history is automatically saved in the left panel of the ChatGPT interface. This feature allows you to easily access and review your previous prompts and responses. By leveraging the conversation history feature, you can keep track of your interactions with ChatGPT and quickly reference previous responses for your cybersecurity tasks or other projects.  Figure – Conversation history in the ChatGPT interface To view a saved conversation, simply click on the desired conversation in the left panel. You can also create new conversations by clicking on the "+ New chat" button located at the top of the conversation list. This enables you to separate and organize your prompts and responses based on specific tasks or topics. Caution Keep in mind that when you start a new conversation, the model loses the context of the previous conversation. If you want to reference any information from a previous conversation, you will need to include that context in your new prompt. ConclusionIn conclusion, this article has unveiled the power of ChatGPT and its conversation-driven approach, making complex tasks like retrieving your public IP address a breeze. With step-by-step guidance, you've learned to harness ChatGPT's capabilities and enjoy context-aware responses, all while keeping your account credits intact. As you dive deeper into the world of ChatGPT, you'll discover its versatility in various applications and the potential to optimize your cybersecurity endeavors. By mastering ChatGPT's conversational prowess, you're on the path to seamless, productive interactions and a future filled with AI-driven possibilities.Author BioClint Bodungen is a cybersecurity professional with 25+ years of experience and the author of Hacking Exposed: Industrial Control Systems. He began his career in the United States Air Force and has since many of the world's largest energy companies and organizations, working for notable cybersecurity companies such as Symantec, Kaspersky Lab, and Booz Allen Hamilton. He has published multiple articles, technical papers, and training courses on cybersecurity and aims to revolutionize cybersecurity education using computer gaming (“gamification”) and AI technology. His flagship product, ThreatGEN® Red vs. Blue, is the world’s first online multiplayer cybersecurity simulation game, designed to teach real-world cybersecurity.
Read more
  • 0
  • 0
  • 18545

article-image-introducing-opensips
Packt
30 Oct 2015
14 min read
Save for later

Introducing OpenSIPS

Packt
30 Oct 2015
14 min read
This article by Flavio E Goncalves and Bogdan-Andrei Iancu, the authors of the book Building Telephony Systems with OpenSIPS - Second Edition, will help you to understand who can use OpenSIPS. This is followed by a description of the OpenSIPS design. It is essential to fully understand OpenSIPS—from the outside and inside perspectives—before using it. In this article, you will see the following: The different usage scenarios for OpenSIPS or when to use it The OpenSIPS internal design and structure (For more resources related to this topic, see here.) Who's using OpenSIPS? A wide range of Session Initiation Protocol (SIP) oriented service providers choose OpenSIPS to power their platform, mainly because of its throughput and capabilities and also because of its reliability as a stable and mature open source software project. Voice over Internet Protocol (VoIP) service providers take advantage of the rich feature set of OpenSIPS to build attractive and competitive services for residential or enterprise customers. The flexibility of OpenSIPS (in terms of service creation) and the fast releasing cycle give them an advantage in creating cutting-edge services in the competitive VoIP markets. The ability to pipe large amounts of traffic through OpenSIPS in a reliable and precise fashion allows providers for trunking services, Direct Inward Dialing (DID), or termination services to scale and become more efficient. The dynamic nature of these services and the interoperability concerns are key aspects that make OpenSIPS the perfect candidate for the job. Currently, OpenSIPS has exceeded its original Class 4 status and is now able to offer Class 5 signaling features. So, hosted/virtual Private Branch Exchange (PBX) providers use OpenSIPS as a core component to emulate PBX-like services. OpenSIPS is flexible enough to integrate all the complex Class 5 features and even more in order to interface with external SIP engines (such as media servers). The ability of OpenSIPS to frontend existing platforms for the load balancing and clustering (even geo distribution) core components opens the markets populated by existing players who offer PBX services or termination services. Such players can now take advantage of OpenSIPS too. Besides calls, OpenSIPS provides you with present and instant messaging capabilities, which are essential to buildservices such as Rich Communication Services (RCS). In the area of advanced SIP services, Local Number Portability (LNP) providers, Canonical Name (CNAME) providers, fraud detection providers are relying on the OpenSIPS skills to interface with various DB engines and handle huge amounts of data (typical for their services). OpenSIPS can offer both, an efficient SIP frontend and a powerful backend to connect to DBs and other external tools. With the OpenSIPS evolution in the Class 5 area, new capabilities such as Back-2-Back User Agent and Call Queuing were exploited by call center providers (inbound and outbound termination). When compared with the existing solutions, OpenSIPS-based call centers were able to handle large amounts of calls and offer, in a geo-distributed fashion, an all-in-one platform: call queuing, DID management, Public Switched Telephone Network (PSTN) routing, SIP peering, and others. Nevertheless, being an open software, OpenSIPS offers a perfect study and research platform. Universities and research centers are using OpenSIPS to familiarize students with the VoIP/SIP world to build projects or research cases. It is important to mention here that this relation with the universities and research institutes is in the benefit of both parties as the OpenSIPS project gets a lot of traction and expansions from these study/research projects. The project manages a list of OpenSIPS users, a list that is continuously growing: http://www.opensips.org/About/WhoIsUsing The OpenSIPS design Architecturally speaking, OpenSIPS is formed of two logical components: the core and modules. The core is the application itself and it provides the low-level functionality of OpenSIPS, the definition of various interfaces and some generic resources. The modules are shared libraries loaded on demand at the startup time. Each module implements a well-defined functionality for a specific routing algorithm or authentication method. There are mainly two types of modules in OpenSIPS: The modules providing functionalities and functions directly for the routing script The modules implementing a core defined interface (such as a module implementing the SQL interface will become a backend to a certain SQL server) The architecture of OpenSIPS is shown in the following figure: The OpenSIPS core The OpenSIPS core is a minimal application. By itself, it is only able to proxy the SIP requests and reply in a stateless mode with very basic scripting capabilities. In most of the cases, the core is used in conjunction with several modules. The OpenSIPS core provides the following: The SIP transport layer The SIP factory—the message parser and builder The routing script parser and interpreter The memory and locking manager The core script functions and variables A SQL interface definition (implementation provided by modules and not by the core itself) A NoSQL interface definition An AAA interface definition A management interface An events interface A statistics interface The SIP transport layer implements various SIP transport protocols. Currently, OpenSIPS supports UDP, TCP, Transport Layer Security (TLS), and WebSocket. The transport protocols that are to be used depend on the SIP listeners defined in the routing script. Multiple transport protocols may be used at the same time. The SIP factory layer provides you with functions to parse and build the SIP messages. OpenSIPS implements a lazy but efficient SIP parser—this means that the parsing is done on demand (OpenSIPS parses as far as requested) and selectively. (The header bodies are parsed only if requested; otherwise, only the header name is parsed.) The parsing process is transparent at the script level; each function (core or modules) does its own parsing internally. When it comes to changing the message, it is very important to know that the changes you make in the script are not applied in real time over the message but stored and applied only when all the message processing is done. Due to this offline handling of changes, you are not able to see your own changes over the message. For example, if you add a new SIP header and test its presence, you will not find it, or if you remove a SIP header and look again for it, you will find it. Therouting script parser and interpreter is loading and parsing (to the memory) the routing script at the startup time; after this, the file is no longer needed. The routing script cannot be reloaded at runtime—OpenSIPS must be restarted in order to read the routing script again. Aside from various global settings, the routing script contains the routing logic—this logic is defined via a C/Shell-like language, custom to OpenSIPS. Configuring the routing logic in OpenSIPS is more like doing simple programming. This approach (writing your own program to route the traffic) gives OpenSIPS a tremendous flexibility when it comes to routing. The memory and locking manager is a global resource in OpenSIPS. For performance reasons, OpenSIPS implements its own internal manager for the memory allocation and locking operations. This part is not visible at the route scripting level, but it may be configured at compiling time. (OpenSIPS has several implementations for the memory and locking managers.) The OpenSIPS core provides its own script functions and script variables to be used from the routing script. When compared with the functions and variables exported by the modules, the core set is quite limited in number and functionality. The online manual lists and documents all the functions and variables provided by the OpenSIPS core. The SQL interface is defined by the core but not implemented. The definition is done here in the OpenSIPS core for standardization reasons. Modules can use SQL services via the SQL interface without being aware of the underlying SQL driver. Additionally, the other modules may implement the SQL interface, offering drivers to various SQL DBs as MySQL, Postgres, Oracle, Berkeley, unixODBC, and many others. (See all the modules starting with the db_ prefix.) Even if the SQL interface is for internal usage (between modules), it is partially exposed at the script level by the avpops module: avp_db_query("select first_name, last_name from subscribers where username='$rU' and domain='$rd'", "$avp(fname);$avp(lname)"); Similar, the NoSQL interface is defined to standardize the operations with the NoSQL databases. The OpenSIPS modules currently implement drivers to Redis, CouchBase, Cassandra, MongoDB, Memcached, and other databases. (See all the modules starting with the cachedb_ prefix.) The NoSQL interface is directly exposed by the OpenSIPS core via a simple set of functions: cache_store("redis:cluster1", "key1", "$var(my_val)", 1200); cache_fetch("redis:cluster1", "key1", "$var(my_val)"); cache_remove("redis:cluster1", "key1"); The AAA Interface defines the interfacing to the AAA servers in a similar way. Currently, OpenSIPS supports the RADIUS driver for the AAA interface, the Diameter driver being under heavy rework at this time. The AAA interface is not exposed at all at the routing script level, being exclusively used internally between modules. The Management Interface (MI) is an OpenSIPS interface that allows the external applications to trigger predefined commands in OpenSIPS. Such commands typically allow an external application/script to do the following: Push data in OpenSIPS (such as setting a debug level, registering a contact, and so on) Fetch data from OpenSIPS (see registered users, ongoing calls, get statistics, and so on) Trigger an internal action in OpenSIPS (reloading the data, sending a message, and so on) The MI commands are provided by the OpenSIPS core (See the online documentation at http://www.opensips.org/Documentation/Interface-CoreMI-2-1.) and also by the modules. (Check the online module documentation to see the commands provided by each module at http://www.opensips.org/Documentation/Modules-2-1.) For the MI, OpenSIPS supports the following drivers: XMLRPC, FIFO file, Datagrams, Json RPC, and HTTP. (See all the modules starting with the mi_ prefix.) A simple example of interacting with OpenSIPS via MI interfaces is using the opensipsctl utility—it uses the FIFO or XMLRPC protocols to push MI commands into OpenSIPS. The opensipsctl utility allows you to run an MI command explicitly via the fifo file: opensipsctl fifo ps opensipsctl fifo debug 4 The following is a simple program in Python to trigger in order to run an MI command in OpenSIPS via the XMLRPC protocol: #!/usr/bin/python import xmlrpclib opensips = xmlrpclib.ServerProxy('http://127.0.0.1:8080/RPC2') print opensips.ps(); The event interface is an OpenSIPS interface that provides you with different ways to notify the external applications about certain events triggered in OpenSIPS. In order to notify an external application about OpenSIPS' internal events, the event interface provides the following functions: Managing the exported events Managing the subscriptions from different applications Exporting generic functions to raise an event (regardless of the transport protocol used) Communicating with different transport protocols to send the events Events can be triggered by the core activities (See the online documentation at http://www.opensips.org/Documentation/Interface-CoreEvents-2-1.), module activities (See the Exported Events section in the module documentation at http://www.opensips.org/Documentation/Modules-2-1.), or explicitly from the routing script via the raise_event() function: raise_event("E_SCRIPT_EVENT", $avp(attributes), $avp(values)); To deliver the events, the OpenSIPS modules implement the following drivers: datagram, RabbitMQ, and XMLRPC. (See all the modules starting with the event_ prefix.) The statistics interface provides you with access to various internal statistics of OpenSIPS. It provides valuable information on what is going on in OpenSIPS. This can be used by the external applications for the monitoring purposes, load evaluation, and real-time integration with other services. The values of statistics variables are exclusively numerical. OpenSIPS provides two types of statistics variables: Counter-like: Variables that keep counting things that happened in OpenSIPS, such as received requests, processed dialogs, failed DB queries, and so on Computed values: Variables that are calculated in real-time, such as how much memory is used, the current load, active dialogs, active transactions, and so on Refer to http://www.opensips.org/Documentation/Interface-Statistics-1-10 for more information. In OpenSIPS, the statistics variables are grouped in different sets depending on their purposes. For example, the OpenSIPS core provides shmem, load, net, and other groups (http://www.opensips.org/Documentation/Interface-CoreStatistics-2-1), while each OpenSIPS module provides its own group. (Typically, the group has the same name as the module.) The statistics can be easily queried via the MI with the get_statistics command: # get various statistic variables, by list of names > opensipsctl fifo get_statistics rcv_requests inuse_transactions > core:rcv_requests = 453 > tm:inuse_transactions = 10 The OpenSIPS modules Each OpenSIPS module is a dynamic library that may be loaded on demand at OpenSIPS startup, if instructed so in the routing script. The modules are identified by their names and they typically export various resources to be used in/from the routing script: A set of module parameters (optional)—they allow the module to be configured at startup A set of script functions (optional)—functions to be used from the routing script A set of asynchronous script functions (optional)—functions to be used from the routing script, but in an asynchronous way via a resume route A set of variables (optional)—variables that can be used from the routing script A set of statistics variables (optional)—statistics specific to the module that can be read via the MI A set of MI commands (optional)—commands specific to the module to be triggered via the MI outside OpenSIPS A set of events (optional)—events that can be triggered by the module (and delivered by the event modules to the external applications) Based on what they implement, there are three types of modules in OpenSIPS: Modules implementing the functionalities to be used from the routing script, that is, authentication methods, routing algorithms, registration handling, and so on. Modules implementing one of the interfaces defined by the core in order to provide a driver for a certain communication protocol, that is, MySQL driver for the SQL interface, RADIUS driver for the AAA interface, and others. Modules implementing their own particular API; such an API is to be used directly (bypassing the core) by the other modules. This allows modules to use other modules, independent of the core. Such modules are the b2b_entities, the dialog module, TM module, and so on. We can conclude that even if most of the modules provide functionalities to the routing script, there are modules providing functionalities to other modules via the core interfaces or their own APIs. These are some examples: The db_mysql module implements the SQL interface defined by the core; the auth_db module using the SQL interface to perform the SQL operations can transparently use any module implementing the SQL interface The b2b_entities module defines and implements its own API (to manage SIP UASs and UACs); the b2b_logic module uses the API from the b2b_entities module directly in order to build even more complex functionalities (that are used later from the routing script) This leads to the concept of module dependencies and an OpenSIPS module may depend on the following: An external library at linking time, that is, b2b_logicdepends on the xml2 library The core interface at startup time—if a module uses the SQL interface, you need to load one or more modules implementing an SQL driver Another module—one module depends directly and explicitly on another module (that is, b2b_logic depends on b2b_entities) The documentation of each module (the README file) contains the information on the module's dependencies, what functions or parameters are exposed, and what MI commands or events are available. See the online module documentations at http://www.opensips.org/Documentation/Modules-2-1. Summary In this article, you have seen who is using OpenSIPS. From the internal perspective, you learned the design/structure of OpenSIPS, such as the OpenSIPS core and OpenSIPS modules and what they can provide. Resources for Article: Further resources on this subject: Architecture of FreeSWITCH [article] Innovation of Communication and Information Technologies [article] Cisco Unified Communications Manager 8: Call Routing, Dial Plan, and E.164 [article]
Read more
  • 0
  • 0
  • 18535

article-image-programming-linux
Packt
02 Jan 2017
20 min read
Save for later

Programming with Linux

Packt
02 Jan 2017
20 min read
In this article by Edward Snajder, the author of Raspberry Pi Zero Cookbook, we pick up from having our operating system installed and our Raspberry Pi Zero on our home network. We can now dive into some basic Linux commands. You will find knowing these commands useful any time you are working on a Linux machine. In this article, we'll start prepping with some Linux recipes: (For more resources related to this topic, see here.) Navigating a filesystem and viewing and searching the contents of a directory Creating a new file, editing it in an editor, and changing ownership Renaming and copying/moving the file/folder into a new directory Installing and uninstalling a program Navigating a filesystem and viewing and searching the contents of a directory If you aren’t already a Linux or Mac user, getting around the filesystem can seem pretty alien at first. Truly, if you’ve only used Windows Explorer, this is going to seem like a strange, alien process. Once you start getting the hang of things, though, you’ll find that getting around the Linux filesystem is easy and fun. Getting ready The only thing you need to get started is a client connection to your Raspberry Pi Zero. I like to use SSH, but you can certainly connect using the serial connection or a terminal in X Windows. How to do it… If you want to find out where you are in the filesystem, use pwd: pi@rpz14101:~$ pwd /home/pi This tells me I’m in the /home/pi directory, which is the default home directory for the pi user. Generally, every user you create should get a /home/username directory to keep their own files in. This can be done automatically with user creation and the adduser command. To look at the contents of the directory you are in, use the ls command: pi@rpz14101:~$ ls Desktop Downloads Pictures python_games share Videos Documents Music Public Scratch Templates To look in another directory, simply specify the directory you want to list (you may need to use sudo depending on where you are looking): pi@rpz14101:~$ sudo ls /opt/ cookbook.share pigpio sonic-pi vc minecraft-pi share testsudo.deleteme Wolfram This is a nice quick summary of what files are in the directory, but you will usually want a little more information about the files and directories. The ls command has a ton of options, all of which can be displayed with ls –help and explained in more detail with man ls. Some of the best ones to know are as follows: -a show all files (regular and hidden) -l show long format (more file information, in columns) -h human readable (turns bytes into MB or GB as appropriate) -t or -tr order in time order, or reverse time order My typical command when I start looking in a directory is this one: ls -ltrh This produces all non-hidden files, with human-readable sizes, in column format, and the newest file at the bottom: pi@rpz14101:~$ ls -ltrh /opt/ total 513M drwxr-xr-x 7 root root 4.0K May 27 04:11 vc drwxr-xr-x 3 root root 4.0K May 27 04:32 Wolfram drwxr-xr-x 3 root root 4.0K May 27 04:34 pigpio drwxr-xr-x 4 root root 4.0K May 27 04:36 minecraft-pi drwxr-xr-x 5 root root 4.0K May 27 04:36 sonic-pi -rw-r--r-- 1 root root 0 Jul 4 13:41 testsudo.deleteme drwxr-xr-x 2 root root 4.0K Jul 9 13:05 share -rwxr-xr-x 1 root root 512M Jul 24 17:53 cookbook.share One last trick: if you need this format but there are a lot of files in the directory you are searching, you will see a ton of text scroll by. Maybe you just need the most recent or largest files? We can do this with a pipe (|) and the tail command. Let’s take a directory with a lot of files, such as /usr/lib/. To list the five most recently modified files, I can pipe ls -ltrh to the tail command: pi@rpz14101:~$ ls -ltrh /usr/lib/ | tail -5 lrwxrwxrwx 1 root root 22 May 27 04:40 libwiringPiDev.so -> libwiringPiDev.so.2.32 drwxr-xr-x 2 root root 4.0K Jun 5 10:38 samba drwxr-xr-x 3 root root 4.0K Jul 4 22:48 pppd drwxr-xr-x 65 root root 60K Jul 24 15:48 arm-linux-gnueabihf drwxr-xr-x 2 root root 4.0K Jul 24 15:48 tmpfiles.d What about the five largest files? Instead of the t in -ltrh, I can use S: pi@rpz14101:~$ ls -lSrh /usr/lib/ | tail -5 -rw-r--r-- 1 root root 2.8M Sep 17 2014 libmozjs185.so.1.0.0 -rw-r--r-- 1 root root 2.8M Sep 30 2014 libqscintilla2.so.11.3.0 -rw-r--r-- 1 root root 2.9M Jun 5 2014 libcmis-0.4.so.4.0.1 -rw-r--r-- 1 root root 3.4M Jun 12 2015 libv8.so.3.14.5 -rw-r--r-- 1 root root 5.1M Aug 18 2014 libmwaw-0.3.so.3.0.1 A little creative piping and you can find exactly the file you are looking for. If not, another great tool for exploring the filesystem is tree. This gives a pseudo-graphical tree that shows how the files are structured in the system. It produces a lot of text, especially if you have it print an entire directory tree. If just looking into directory structures, you can use tree with the -d flag for directories only. The -L flag will reduce how deep you dive into nested directories: pi@rpz14101:~$ tree -d -L 2 /opt/ /opt/ ├── minecraft-pi │ ├── api │ └── data ├── pigpio │ └── cgi ├── share ├── sonic-pi │ ├── app │ ├── bin │ └── etc ├── vc │ ├── bin │ ├── include │ ├── lib │ ├── sbin │ └── src └── Wolfram └── WolframEngine Last, we will look at a couple of searching utilities, find and grep. The find command is a powerful function that finds files in whatever directories you specify. It is great for trying to find that mystery piece of software that installed itself in an odd place or the needle-in-a-haystack file in a directory that contains hundreds of files. For example, if I were to run tree in the /opt/sonic-pi/ directory, it would run on for several seconds, and thousands of files would shoot by. I, however, am only interested in finding files with cowbell in the name. I can use the find command to look for it: pi@rpz14101:~$ find /opt/sonic-pi/ -name *cowbell* /opt/sonic-pi/etc/samples/drum_cowbell.flac When looking for anything with cowbell in the filename, the find command returns the exactly location of anything that matches. There are tons of options for using the find command; start with find –help, and then try man find when you want to get really deep. The grep command can be used in a couple different ways when searching for files, and it is one of those commands you will find yourself using constantly while both loving and hating its awesome power. Let’s say you need to find something inside of a file—grep is the tool for you. It can also find things like find can, but generally, find is more efficient at finding filename patterns than grep is. If I use grep to look for cowbells in my sonic-pi directory, I’ll get a different, and more colorful, output: We don’t see the file with cowbell in the name like we did using find, but we find every file that contains cowbell inside of it. The -r flag tells grep to delve into subdirectories, and -i tells it to ignore cases with cowbells (so Cowbell and cowbell are both found, as shown in the screenshot). As you use Linux more often, both find and grep become regularly used tools for administration and file management. This won’t be the last time you use them! Creating a new file, editing it in an editor, and changing ownership There are a lot of different text editors to use on a Linux system from the command line. The program vi is the Ubuntu default, and the program you will find installed on pretty much any Linux system. Emacs is another popular editor, and lots of Linux users get quite passionate about which one is better. My preference is vim, which is generally known as vi improved. The nano text editor is another one that is commonly installed on Linux distros, and it is one of the most lightweight editors available. Getting ready For this recipe, we will work with vi, since that’s definitely going to be installed on your system. If you want to try out vim, you can install it using this: sudo apt-get vim How to do it… First we will go to our share directory: cd /home/pi/share Then, we will create an empty file using the touch command: touch ch3_touchfile.txt If you use the ls command from the previous directory, you can see that the size of the file is 0. You can also display the contents of the file with the cat command, which will return nothing in this case. The touch command is a great way to test whether you have permissions to create files in a specific directory. You can also create a new file with the editor itself: vi ch3_vifile.txt This will open the vi editor with a blank file named ch3_vifile.txt: Using vi or vim (or Emacs) for the first time is completely different from using something like OpenOffice or Microsoft Word. Vi works in two modes: insert (or edit) and command. Once you learn how to use command mode, vi becomes a very efficient editor for working on scripts in bash or Python. Edit mode, more or less, is the mode where you can type and edit text like a regular WYSIWYG editor. There are books written on becoming a power user of vi, well beyond the scope of this book. Getting a handle on the basics is the best place to start: With the empty file, you can jump into edit mode by pressing the i or a keys. The editor will switch to insert mode, as shown by the -- INSERT -- in the bottom left of the screen. Then you can you start typing in your text: To get out of insert mode, press the Esc key. The :w command will save the file, and the :q command will quit. You can combine them, so :wq saves the file and quits. You can verify that the contents were saved with the cat command: pi@rpz14101:~$ cat ch3_vifile.txt Hello from the Raspberry Pi Zero Cookbook! Let’s take another look at the ls command and some of the information the -l format includes. We will take a look at the files we’ve created so far in this recipe: pi@rpz14101:~$ ls -ltrh *.txt -rw-r--r-- 1 pi pi 43 Jul 25 11:23 ch3_vifile.txt -rw-r--r-- 1 pi pi 0 Jul 25 11:24 ch3_touchfile.txt File Permissions Number of links Owner:Group Size Modification Date File Name -rw-r--r-- 1 pi:pi 43 Jul 25 11:23 ch3_vifile.txt We can see that since we made the files as the pi user, the owner of the file and the group owner are pi. By default, when a new user is created, a group container is created as well, so root has a root group, user rpz has an rpz group, and so on. We can change the ownership settings of a file with the chown command. Be careful, since you can take away your own access, though you can always sudo your way back. The chmod command will change who is allowed to do what with a file. Let’s look at ownership changes and what impact they will have with a few examples: pi@rpz14101:~ $ ls -ltrh *.txt -rw-r--r-- 1 pi pi 0 Jul 25 13:28 ch3_touchfile.txt -rwx------ 1 pi pi 43 Jul 25 13:28 ch3_vifile.txt pi@rpz14101:~ $ cat ch3_vifile.txt Hello from the Raspberry Pi Zero Cookbook! pi@rpz14101:~ $ sudo chown rpz:rpz ch3_vifile.txt pi@rpz14101:~ $ cat ch3_vifile.txt cat: ch3_vifile.txt: Permission denied pi@rpz14101:~ $ sudo cat ch3_vifile.txt Hello from the Raspberry Pi Zero Cookbook! pi@rpz14101:~ $ sudo chown rpz:pi ch3_vifile.txt pi@rpz14101:~ $ cat ch3_vifile.txt cat: ch3_vifile.txt: Permission denied pi@rpz14101:~ $ sudo chmod 750 ch3_vifile.txt pi@rpz14101:~ $ cat ch3_vifile.txt Hello from the Raspberry Pi Zero Cookbook! pi@rpz14101:~ $ sudo chown root:root ch3_vifile.txt pi@rpz14101:~ $ cat ch3_vifile.txt cat: ch3_vifile.txt: Permission denied pi@rpz14101:~ $ sudo chmod 755 ch3_vifile.txt pi@rpz14101:~ $ cat ch3_vifile.txt Hello from the Raspberry Pi Zero Cookbook! The chmod values are documented very well, and with a little practice, you can get your file permissions and ownership set up in a way that is both secure and easy to work with. Renaming and copying/moving the file/folder into a new directory A common activity on any filesystem is the practice of copying and moving files, and even directories, from one place to another. You might do it to make a backup copy of something, or you might decide that the contents should live in a more appropriate location. This recipe will explore how to manipulate files in the Raspbian system. Getting ready If you are still in your terminal from the last recipe, we are going to use the same files from the previous recipe. We should have the ownership back to pi:pi; if not, run the following: sudo chown pi:pi /home/pi/*.txt How to do it… First, let’s make a new directory. We’ll put it under the /home/pi/share/ folder so it is accessible to other computers on your home network. To make a directory, use the mkdir command: pi@rpz14101:~$ mkdir /home/pi/share/ch3 We can look at the new directory with the ls command: pi@rpz14101:~$ ls -ltrh /home/pi/share/ total 4.0K -rw-r--r-- 1 pi pi 0 Jul 24 15:56 helloNetwork.yes drwxr-xr-x 2 pi pi 4.0K Jul 25 13:06 ch3 A great flag to go with the mkdir command is -p. This will allow you to create directories and subdirectories in one command. Without it, if I try to create a subdirectory that doesn’t already exist, I’ll get an error: pi@rpz14101:~$ mkdir /home/pi/share/ch3/nested/folders mkdir: cannot create directory ‘/home/pi/share/ch3/nested/folders’: No such file or directory With the -p flag, it works without a problem: pi@rpz14101:~$ mkdir -p /home/pi/share/ch3/nested/folders The tree command shows the structure of our ch3 directory: pi@rpz14101:~$ tree /home/pi/share /home/pi/share ├── ch3 │ └── nested │ └── folders └── helloNetwork.yes 3 directories, 1 file Now, let’s move our files to our new ch3 directory. The copy and move commands—cp and mv, respectively—are the tools we will use. Copying a file from one place to another is as simple as indicating the file's source and destination. The following command will make a copy of vifile.txt and save it as vifile.txt.copy in the /home/pi/share/ch3/ directory: cp /home/pi/ch3_vifile.txt /home/pi/share/ch3/ch3_vifile.txt.copy We can copy files as well as directories and their contents as long as you have enough disk space. To move or rename a file, we use the mv command. This takes the file given in the source and moves it to the destination provided. As simple as the cp command, let’s move all of our files to the share directory: mv /home/pi/ch3_vifile.txt /home/pi/share/ch3/ch3_vifile.txt.moved mv /home/pi/ch3_touchfile.txt /home/pi/share/ch3/ch3_touchfile.txt If we look at the tree of our share directory, we will see everything nicely organized: pi@rpz14101:~$ tree /home/pi/share/ /home/pi/share/ ├── ch3 │ ├── ch3_touchfile.txt │ ├── ch3_vifile.txt.copy │ ├── ch3_vifile.txt.moved │ └── nested │ └── folders └── helloNetwork.yes 3 directories, 4 files Installing and uninstalling a program We’ve installed a few programs throughout the book so far, but have yet to delve into the apt-get command and the family of software-installation utilities. Now, we will learn how to install and uninstall any program available for Raspbian as well as how to search for new software and run updates. Getting ready Stay in your terminal window, and get ready to install some applications! How to do it… The apt-* commands are a suite of utilities that allow you to do various things with installed packages. To install a package, we use the apt-get tool, and the install command, like this: sudo apt-get install <packagename> Let’s install something cool—how about a Matrix screensaver? It is super easy and works great from the command line. To look for a package, we use the apt-cache search command. apt-cache is another tool in the apt-* family of utilities, and it checks the software database for matches. Running sudo apt-cache search matrix results tons of results! The word "matrix" is a little too popular for us computer and math nerds—we have matrixes everywhere! It would take forever to go through that list to find what we are looking for. Fortunately, we can take advantage of grep, which we touched on in an earlier recipe, to narrow down our results. One of the fun things about using Linux and the command line is the ways you can chain commands to do cool things: pi@rpz14101:~ $ sudo apt-cache search matrix | grep "The Matrix" cmatrix - simulates the display from "The Matrix" wmmatrix - View The Matrix in a Window Maker dock application That’s a bit more manageable! We could also have narrowed the list using this command: sudo apt-cache search “The Matrix” This returns fewer results than before, but a few more than the grep command. Whichever way you find it, we see that the cmatrix package is the one we are looking for. Installing is as simple as running this: pi@rpz14101:~ $ ` Reading package lists... Done Building dependency tree Reading state information... Done Suggested packages: cmatrix-xfont The following NEW packages will be installed: cmatrix 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 16.2 kB of archives. After this operation, 27.6 kB of additional disk space will be used. Get:1 http://mirrordirector.raspbian.org/raspbian/ jessie/main cmatrix armhf 1.2a-5 [16.2 kB] Fetched 16.2 kB in 1s (15.3 kB/s) Selecting previously unselected package cmatrix. (Reading database ... 121906 files and directories currently installed.) Preparing to unpack .../cmatrix_1.2a-5_armhf.deb ... Unpacking cmatrix (1.2a-5) ... Processing triggers for man-db (2.7.0.2-5) ... Setting up cmatrix (1.2a-5) ... After that, we are ready to go! Channel your inner Neo and run this command: cmatrix -s -b You should be in the Matrix! Try it on your serial and SSH connections, and even in the terminal on VNC: you’ll notice differences in the rendering behavior. There are literally thousands of software packages available to install in the repositories of our awesome open source communities. Pretty much anything you think a computer should be able to do, someone, or a group of people, has worked on a solution and pushed it out to the repositories.  We will be using apt-get a lot throughout this cookbook; it is one of the commands you’ll find yourself using all the time as you get more interested in Raspberry Pis and the Linux operating system. Running sudo apt-get update will check all repositories to see whether there are any version updates available. Here, you can see all of the locations it checks to see whether there is anything new for Raspbian: pi@rpz14101:~ $ sudo apt-get update Get:1 http://archive.raspberrypi.org jessie InRelease [13.2 kB] Get:2 http://mirrordirector.raspbian.org jessie InRelease [14.9 kB] Get:3 http://archive.raspberrypi.org jessie/main armhf Packages [144 kB] Get:4 http://mirrordirector.raspbian.org jessie/main armhf Packages [8,981 kB] Hit http://archive.raspberrypi.org jessie/ui armhf Packages Ign http://archive.raspberrypi.org jessie/main Translation-en_GB Get:5 http://mirrordirector.raspbian.org jessie/contrib armhf Packages [37.5 kB] Ign http://archive.raspberrypi.org jessie/main Translation-en Get:6 http://mirrordirector.raspbian.org jessie/non-free armhf Packages [70.3 kB] Ign http://archive.raspberrypi.org jessie/ui Translation-en_GB Ign http://archive.raspberrypi.org jessie/ui Translation-en Get:7 http://mirrordirector.raspbian.org jessie/rpi armhf Packages [1,356 B] Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en_GB Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en Ign http://mirrordirector.raspbian.org jessie/main Translation-en_GB Ign http://mirrordirector.raspbian.org jessie/main Translation-en Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en_GB Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en_GB Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en Fetched 9,263 kB in 34s (272 kB/s) Reading package lists... Done After updating, apt-get upgrade will look at the versions of everything you have installed and upgrade anything to the latest version if there is one available. Depending on how many updates you have, this can take quite a while: pi@rpz14101:~ $ sudo apt-get upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following packages will be upgraded: dpkg-dev gir1.2-gdkpixbuf-2.0 initramfs-tools libavcodec56 libavformat56 libavresample2 libavutil54 libdevmapper-event1.02.1 libdevmapper1.02.1 libdpkg-perl python-picamera python3-picamera raspberrypi-kernel raspberrypi-net-mods ssh tzdata xarchiver 40 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 57.0 MB of archives. After this operation, 415 kB of additional disk space will be used. Do you want to continue? [Y/n] y Get:1 http://archive.raspberrypi.org/debian/ jessie/main nodered armhf 0.14.5 [5,578 kB] … Adding 'diversion of /boot/overlays/w1-gpio.dtbo to /usr/share/rpikernelhack/overlays/w1-gpio.dtbo by rpikernelhack' … run-parts: executing /etc/kernel/postrm.d/initramfs-tools 4.4.11-v7+ /boot/kernel7.img Preparing to unpack .../raspberrypi-net-mods_1.2.3_armhf.deb ... Unpacking raspberrypi-net-mods (1.2.3) over (1.2.2) ... Processing triggers for man-db (2.7.0.2-5) ... … Setting up libssl1.0.0:armhf (1.0.1t-1+deb8u2) ... Setting up libxml2:armhf (2.9.1+dfsg1-5+deb8u2) ... … Removing 'diversion of /boot/overlays/w1-gpio-pullup.dtbo to /usr/share/rpikernelhack/overlays/w1-gpio-pullup.dtbo by rpikernelhack' … Setting up raspberrypi-net-mods (1.2.3) ... Modified /etc/network/interfaces detected. Leaving unchanged and writing new file as interfaces.new. Processing triggers for libc-bin (2.19-18+deb8u4) ... Processing triggers for initramfs-tools (0.120+deb8u2) ... You don’t really have to understand the details of what’s going on during the upgrade, and it will let you know if there were any problems at the end (and often what to do to fix them). Regularly updating and upgrading will keep all of your software current with all of the latest bug fixes and security patches. There’s more… You can also add and remove software from the GUI. If you log on to your Pi, either directly to a monitor or over VNC Server (a recipe we covered earlier), you can find the Add / Remove Software option under Menu | Preferences: The PiPackages utility makes it very easy to find software when you only have a general idea of what you are looking for. While you can do the same things with the apt commands, if you are browsing, this is a little easier on the eyes. The utility provides categorizations so you don’t have to scroll through every package. Clicking on a package provides a detailed description: Simply check the box and click on Apply or OK, and the software will be installed. Now you can install software on your Raspberry Pi Zero from the command line or GUI. Summary In this article, we looked at the basic file manipulation functionalities of Linux on a Raspberry Pi. We saw how to navigate the filesystem and create, edit, rename, copy, and move files and folders. We also saw how to change ownership of a file and install and uninstall programs. Resources for Article: Further resources on this subject: Sending Notifications using Raspberry Pi Zero [Article] Raspberry Pi LED Blueprints [Article] Hacking a Raspberry Pi project? Understand electronics first! [Article]
Read more
  • 0
  • 0
  • 18524
article-image-puppet-announces-the-public-beta-of-project-nebula
Savia Lobo
10 Oct 2019
3 min read
Save for later

Puppet announces the public beta of Project Nebula

Savia Lobo
10 Oct 2019
3 min read
Today, Puppet announced the public beta of their Project Nebula at the Puppetize PDX, a two-day event (October 9-10) featuring user-focused DevOps and infrastructure delivery talks, and hands-on workshops. Project Nebula is a simplified workflow automation for the continuous deployment of cloud-native applications and infrastructure. It is designed for teams that are adopting cloud-native and serverless technologies and need an end-to-end workflow management system. Also Read: Puppet’s 2019 State of DevOps Report highlight security integration into DevOps practices result into higher business outcome Why Project Nebula? Puppet has worked closely with its private beta participants to understand their deployment workflows and pain points. On interviewing these participants they realized that they want to adopt cloud native technologies; however, they face multiple challenges in adopting containers, serverless infrastructure, microservices, and observability for even simple cloud-native applications. They also said that a major roadblock is a lack of simple automation today to easily compose multiple tools together for infrastructure provisioning, application deployment, and notifications into an end-to-end deployment. Another roadblock highlighted is the lack of a cohesive platform that multiple teams can use to share workflows and best practices and build them into their own deployments. “In-house efforts to build a deployment platform like this can take years, incur large maintenance and support costs, and often require specialized skill sets that many companies do not have today,” the company states. Project Nebula tries to help users in eliminating these roadblocks and gives teams a consistent, easy-to-use experience for deploying cloud-native apps in a safe, secure and continuous manner. Listen: Puppet’s VP of Ecosystem Engineering Nigel Kersten talks about key DevOps challenges [Podcast] Few features in Project Nebula With the focus on ease of use and improved productivity, Project Nebula also provides a single place to build, provision, and deploy cloud-native applications. Other notable features include: Built-in example Workflows: This will help users to get started with their deployments. Don’t start with a blank slate. Support for 20+: This is one of the most popular cloud-native deployment tools as configurable steps within your deployment, including Terraform, CloudFormation, Helm, Kubectl, Kustomize and more. Intuitive visualization: This provides a bird’s eye view of the entire deployment workflow. Easy to compose deployment workflows: One can easily compose deployment workflows that are checked into the source control repository and eliminate the process of writing messy, ad-hoc bash scripts. Know more about Puppet’s Project Nebula in detail, on its official website. Puppet launches Puppet Remediate, a vulnerability remediation solution for IT Ops Puppet announces updates in a bid to help organizations manage their “automation footprint” “This is John. He literally wrote the book on Puppet” – An Interview with John Arundel
Read more
  • 0
  • 0
  • 18497

article-image-why-are-experts-worried-about-microsofts-billion-dollar-bet-in-openais-agi-pipe-dream
Sugandha Lahoti
23 Jul 2019
6 min read
Save for later

Why are experts worried about Microsoft's billion dollar bet in OpenAI's AGI pipe dream?

Sugandha Lahoti
23 Jul 2019
6 min read
Microsoft has invested $1 billion in OpenAI with the goal of building next-generation supercomputers and a platform within Microsoft Azure which will scale to AGI (Artificial General Intelligence). This is a multiyear partnership with Microsoft becoming OpenAI’s preferred partner for commercializing new AI technologies. Open AI will become a big Azure customer, porting its services to run on Microsoft Azure. The $1 billion is a cash investment into OpenAI LP, which is Open AI’s for-profit corporate subsidiary. The investment will follow a standard capital commitment structure which means OpenAI can call for it, as they need it. But the company plans to spend it in less than five years. Per the official press release, “The companies will focus on building a computational platform in Azure for training and running advanced AI models, including hardware technologies that build on Microsoft’s supercomputing technology. These will be implemented in a safe, secure and trustworthy way and is a critical reason the companies chose to partner together.” They intend to license some of their pre-AGI technologies, with Microsoft becoming their preferred partner. “My goal in running OpenAI is to successfully create broadly beneficial A.G.I.,” Sam Altman, who co-founded Open AI with Elon Musk, said in a recent interview. “And this partnership is the most important milestone so far on that path.” Musk left the company in February 2019, to focus on Tesla and because he didn’t agree with some of what OpenAI team wanted to do. What does this partnership mean for Microsoft and Open AI OpenAI may benefit from this deal by keeping their innovations private which may help commercialization, raise more funds and get to AGI faster. For OpenAI this means the availability of resources for AGI, while potentially allowing founders and other investors with the opportunity to either double-down on OpenAI or reallocate resources to other initiatives However, this may also lead to them not disclosing progress, papers with details, and open source code as much as in the past. https://twitter.com/Pinboard/status/1153380118582054912 As for Microsoft, this deal is another attempt in quietly taking over open source. First, with the acquisition of GitHub and the subsequent launch of GitHub Sponsors, and now with becoming OpenAI’s ‘preferred partner’ for commercialization. Last year at an Investor conference, Nadella said, “AI is going to be one of the trends that is going to be the next big shift in technology. It's going to be AI at the edge, AI in the cloud, AI as part of SaaS applications, AI as part of in fact even infrastructure. And to me, to be the leader in it, it's not enough just to sort of have AI capability that we can exercise—you also need the ability to democratize it so that every business can truly benefit from it. That to me is our identity around AI.” Partnership with OpenAI seems to be a part of this plan. This deal can also possibly help Azure catch up with Google and Amazon both in hardware scalability and Artificial Intelligence offerings. A hacker news user comments, “OpenAI will adopt and make Azure their preferred platform. And Microsoft and Azure will jointly "develop new Azure AI supercomputing technologies", which I assume is advancing their FGPA-based deep learning offering. Google has a lead with TensorFlow + TPUs and this is a move to "buy their way in", which is a very Microsoft thing to do.” https://twitter.com/soumithchintala/status/1153308199610511360 It is also likely that Microsoft is investing money which will eventually be pumped back into its own company, as OpenAI buys computing power from the tech giant. Under the terms of the contract, Microsoft will eventually become the sole cloud computing provider for OpenAI, and most of that $1 billion will be spent on computing power, Altman says. OpenAI, who were previously into building ethical AI will now pivot to build cutting edge AI and move towards AGI. Sometimes even neglecting ethical ramifications, wanting to deploy tech at the earliest which is what Microsoft would be interested in monetizing. https://twitter.com/CadeMetz/status/1153291410994532352 I see two primary motivations: For OpenAI—to secure funding and to gain some control over hardware which in turn helps differentiate software. For MSFT—to elevate Azure in the minds of developers for AI training. - James Wang, Analyst at ARKInvest https://twitter.com/jwangARK/status/1153338174871154689 However, the news of this investment did not go down well with some experts in the field who saw this as a pure commercial deal and questioned whether OpenAI’s switch to for-profit research undermines its claims to be “democratizing” AI. https://twitter.com/fchollet/status/1153489165595504640 “I can't really parse its conversion into an LP—and Microsoft's huge investment—as anything but a victory for capital” - Robin Sloan, Author https://twitter.com/robinsloan/status/1153346647339876352 “What is OpenAI? I don't know anymore.” - Stephen Merity, Deep learning researcher https://twitter.com/Smerity/status/1153364705777311745 https://twitter.com/SamNazarius/status/1153290666413383682 People are also speculating whether creating AGI is really even possible. In a recent survey experts estimated that there was a 50 percent chance of creating AGI by the year 2099. Pet New York Times, most experts believe A.G.I. will not arrive for decades or even centuries. Even Altman admits OpenAI may never get there. But the race is on nonetheless. Then why is Microsoft delivering the $1 billion over five years considering that is neither enough money nor enough time to produce AGI. Although, OpenAI has certainly impressed the tech community with its AI innovations. In April, OpenAI’s new algorithm that is trained to play the complex strategy game, Dota 2, beat the world champion e-sports team OG at an event in San Francisco, winning the first two matches of the ‘best-of-three’ series. The competition included a human team of five professional Dota 2 players and AI team of five OpenAI bots. In February, they released a new AI model GPT-2, capable of generating coherent paragraphs of text without needing any task-specific training. However experts felt that the move signalled towards ‘closed AI’ and propagated the ‘fear of AI’ for its ability to write convincing fake news from just a few words. Github Sponsors: Could corporate strategy eat FOSS culture for dinner? Microsoft is seeking membership to Linux-distros mailing list for early access to security vulnerabilities OpenAI: Two new versions and the output dataset of GPT-2 out!
Read more
  • 0
  • 0
  • 18493

article-image-command-line-tools
Packt
06 Jul 2017
9 min read
Save for later

Command-Line Tools

Packt
06 Jul 2017
9 min read
In this article by Aaron Torres, author of the book, Go Cookbook, we will cover the following recipes: Using command-line arguments Working with Unix pipes An ANSI coloring application (For more resources related to this topic, see here.) Using command-line arguments This article will expand on other uses for these arguments by constructing a command that supports nested subcommands. This will demonstrate Flagsets and also using positional arguments passed into your application. This recipe requires a main function to run. There are a number of third-party packages for dealing with complex nested arguments and flags, but we'll again investigate doing so using only the standard library. Getting ready You need to perform the following steps for the installation: Download and install Go on your operating system at https://golang.org/doc/install and configure your GOPATH. Open a terminal/console application. Navigate to your GOPATH/src and create a project directory, for example, $GOPATH/src/github.com/yourusername/customrepo. All code will be run and modified from this directory. Optionally, install the latest tested version of the code using the go get github.com/agtorre/go-cookbook/ command. How to do it... From your terminal/console application, create and navigate to the chapter2/cmdargs directory. Copy tests from https://github.com/agtorre/go-cookbook/tree/master/chapter2/cmdargs or use this as an exercise to write some of your own. Create a file called cmdargs.go with the following content: package main import ( "flag" "fmt" "os" ) const version = "1.0.0" const usage = `Usage: %s [command] Commands: Greet Version ` const greetUsage = `Usage: %s greet name [flag] Positional Arguments: name the name to greet Flags: ` // MenuConf holds all the levels // for a nested cmd line argument type MenuConf struct { Goodbye bool } // SetupMenu initializes the base flags func (m *MenuConf) SetupMenu() *flag.FlagSet { menu := flag.NewFlagSet("menu", flag.ExitOnError) menu.Usage = func() { fmt.Printf(usage, os.Args[0]) menu.PrintDefaults() } return menu } // GetSubMenu return a flag set for a submenu func (m *MenuConf) GetSubMenu() *flag.FlagSet { submenu := flag.NewFlagSet("submenu", flag.ExitOnError) submenu.BoolVar(&m.Goodbye, "goodbye", false, "Say goodbye instead of hello") submenu.Usage = func() { fmt.Printf(greetUsage, os.Args[0]) submenu.PrintDefaults() } return submenu } // Greet will be invoked by the greet command func (m *MenuConf) Greet(name string) { if m.Goodbye { fmt.Println("Goodbye " + name + "!") } else { fmt.Println("Hello " + name + "!") } } // Version prints the current version that is // stored as a const func (m *MenuConf) Version() { fmt.Println("Version: " + version) } Create a file called main.go with the following content: package main import ( "fmt" "os" "strings" ) func main() { c := MenuConf{} menu := c.SetupMenu() menu.Parse(os.Args[1:]) // we use arguments to switch between commands // flags are also an argument if len(os.Args) > 1 { // we don't care about case switch strings.ToLower(os.Args[1]) { case "version": c.Version() case "greet": f := c.GetSubMenu() if len(os.Args) < 3 { f.Usage() return } if len(os.Args) > 3 { if.Parse(os.Args[3:]) } c.Greet(os.Args[2]) default: fmt.Println("Invalid command") menu.Usage() return } } else { menu.Usage() return } } Run the go build command. Run the following commands and try a few other combinations of arguments: $./cmdargs -h Usage: ./cmdargs [command] Commands: Greet Version $./cmdargs version Version: 1.0.0 $./cmdargs greet Usage: ./cmdargs greet name [flag] Positional Arguments: name the name to greet Flags: -goodbye Say goodbye instead of hello $./cmdargs greet reader Hello reader! $./cmdargs greet reader -goodbye Goodbye reader! If you copied or wrote your own tests go up one directory and run go test, and ensure all tests pass. How it works... Flagsets can be used to set up independent lists of expected arguments, usage strings, and more. The developer is required to do validation on a number of arguments, parsing in the right subset of arguments to commands, and defining usage strings. This can be error prone and requires a lot of iteration to get it completely correct. The flag package makes parsing arguments much easier and includes convenience methods to get the number of flags, arguments, and more. This recipe demonstrates basic ways to construct a complex command-line application using arguments, including a package-level config, required positional arguments, multi-leveled command usage, and how to split these things into multiple files or packages if needed. Working with Unix pipes Unix pipes are useful when passing the output of one program to the input of another. Consider the following example: $ echo "test case" | wc -l 1 In a Go application, the left-hand side of the pipe can be read in using os.Stdin and acts like a file descriptor. To demonstrate this, this recipe will take an input on the left-hand side of a pipe and return a list of words and their number of occurrences. These words will be tokenized on white space. Getting ready Refer to the Getting Ready section of the Using command-line arguments recipe. How to do it... From your terminal/console application, create a new directory, chapter2/pipes. Navigate to that directory and copy tests from https://github.com/agtorre/go-cookbook/tree/master/chapter2/pipes or use this as an exercise to write some of your own. Create a file called pipes.go with the following content: package main import ( "bufio" "fmt" "os" ) // WordCount takes a file and returns a map // with each word as a key and it's number of // appearances as a value func WordCount(f *os.File) map[string]int { result := make(map[string]int) // make a scanner to work on the file // io.Reader interface scanner := bufio.NewScanner(f) scanner.Split(bufio.ScanWords) for scanner.Scan() { result[scanner.Text()]++ } if err := scanner.Err(); err != nil { fmt.Fprintln(os.Stderr, "reading input:", err) } return result } func main() { fmt.Printf("string: number_of_occurrencesnn") for key, value := range WordCount(os.Stdin) { fmt.Printf("%s: %dn", key, value) } }   Run echo "some string" | go run pipes.go. You may also run: go build echo "some string" | ./pipes You should see the following output: $ echo "test case" | go run pipes.go string: number_of_occurrences test: 1 case: 1 $ echo "test case test" | go run pipes.go string: number_of_occurrences test: 2 case: 1 If you copied or wrote your own tests, go up one directory and run go test, and ensure that all tests pass. How it works... Working with pipes in go is pretty simple, especially if you're familiar with working with files. This recipe uses a scanner to tokenize the io.Reader interface of the os.Stdin file object. You can see how you must check for errors after completing all of the reads. An ANSI coloring application Coloring an ANSI terminal application is handled by a variety of code before and after a section of text that you want colored. This recipe will explore a basic coloring mechanism to color the text red or keep it plain. For a more complete application, take a look at https://github.com/agtorre/gocolorize, which supports many more colors and text types implements the fmt.Formatter interface for ease of printing. Getting ready Refer to the Getting Ready section of the Using command line arguments recipe. How to do it... From your terminal/console application, create and navigate to the chapter2/ansicolor directory. Copy tests from https://github.com/agtorre/go-cookbook/tree/master/chapter2/ansicolor or use this as an exercise to write some of your own. Create a file called color.go with the following content: package ansicolor import "fmt" //Color of text type Color int const ( // ColorNone is default ColorNone = iota // Red colored text Red // Green colored text Green // Yellow colored text Yellow // Blue colored text Blue // Magenta colored text Magenta // Cyan colored text Cyan // White colored text White // Black colored text Black Color = -1 ) // ColorText holds a string and its color type ColorText struct { TextColor Color Text string } func (r *ColorText) String() string { if r.TextColor == ColorNone { return r.Text } value := 30 if r.TextColor != Black { value += int(r.TextColor) } return fmt.Sprintf("33[0;%dm%s33[0m", value, r.Text) } Create a new directory named example. Navigate to example and then create a file named main.go with the following content. Ensure that you modify the ansicolor import to use the path you set up in step 1: package main import ( "fmt" "github.com/agtorre/go-cookbook/chapter2/ansicolor" ) func main() { r := ansicolor.ColorText{ansicolor.Red, "I'm red!"} fmt.Println(r.String()) r.TextColor = ansicolor.Green r.Text = "Now I'm green!" fmt.Println(r.String()) r.TextColor = ansicolor.ColorNone r.Text = "Back to normal..." fmt.Println(r.String()) } Run go run main.go. Alternatively, you may also run the following: go build ./example You should see the following with the text colored if your terminal supports the ANSI coloring format: $ go run main.go I'm red! Now I'm green! Back to normal... If you copied or wrote your own tests, go up one directory and run go test, and ensure that all the tests pass. How it works... This application makes use of a struct keyword to maintain state of the colored text. In this case, it stores the color of the text and the value of the text. The final string is rendered when you call the String() method, which will either return colored text or plain text depending on the values stored in the struct. By default, the text will be plain. Summary In this article, we demonstrated basic ways to construct a complex command-line application using arguments, including a package-level config, required positional arguments, multi-leveled command usage, and how to split these things into multiple files or packages if needed. We saw how to work with Unix pipes and explored a basic coloring mechanism to color text red or keep it plain. Resources for Article: Further resources on this subject: Building a Command-line Tool [article] A Command-line Companion Called Artisan [article] Scaffolding with the command-line tool [article]
Read more
  • 0
  • 0
  • 18487
article-image-writing-fully-native-application
Packt
05 May 2015
15 min read
Save for later

Writing a Fully Native Application

Packt
05 May 2015
15 min read
In this article written by Sylvain Ratabouil, author of Android NDK Beginner`s Guide - Second Edition, we have breached Android NDK's surface using JNI. But there is much more to find inside! The NDK includes its own set of specific features, one of them being Native Activities. Native activities allow creating applications based only on native code, without a single line of Java. No more JNI! No more references! No more Java! (For more resources related to this topic, see here.) In addition to native activities, the NDK brings some APIs for native access to Android resources, such as display windows, assets, device configuration. These APIs help in getting rid of the tortuous JNI bridge often necessary to embed native code. Although there is a lot still missing, and not likely to be available (Java remains the main platform language for GUIs and most frameworks), multimedia applications are a perfect target to apply them. Here we initiate a native C++ project developed progressively throughout this article: DroidBlaster. Based on a top-down viewpoint, this sample scrolling shooter will feature 2D graphics, and, later on, 3D graphics, sound, input, and sensor management. We will be creating its base structure and main game components. Let's now enter the heart of the Android NDK by: Creating a fully native activity Handling main activity events Accessing display window natively Retrieving time and calculating delays Creating a native Activity The NativeActivity class provides a facility to minimize the work necessary to create a native application. It lets the developer get rid of all the boilerplate code to initialize and communicate with native code and concentrate on core functionalities. This glue Activity is the simplest way to write applications, such as games without a line of Java code. The resulting project is provided with this book under the name DroidBlaster_Part1. Time for action – creating a basic native Activity We are now going to see how to create a minimal native activity that runs an event loop. Create a new hybrid Java/C++ project:      Name it DroidBlaster.      Turn the project into a native project. Name the native module droidblaster.      Remove the native source and header files that have been created by ADT.      Remove the reference to the Java src directory in Project Properties | Java Build Path | Source. Then, remove the directory itself on disk.      Get rid of all layouts in the res/layout directory.      Get rid of jni/droidblaster.cpp if it has been created. In AndroidManifest.xml, use Theme.NoTitleBar.Fullscreen as the application theme. Declare a NativeActivity that refers to the native module named droidblaster (that is, the native library we will compile) using the meta-data property android.app.lib_name: <?xml version="1.0" encoding="utf-8"?> <manifest    package="com.packtpub.droidblaster2d" android_versionCode="1"    android_versionName="1.0">    <uses-sdk        android_minSdkVersion="14"        android_targetSdkVersion="19"/>      <application android_icon="@drawable/ic_launcher"        android_label="@string/app_name"        android_allowBackup="false"        android:theme        ="@android:style/Theme.NoTitleBar.Fullscreen">        <activity android_name="android.app.NativeActivity"            android_label="@string/app_name"            android_screenOrientation="portrait">            <meta-data android_name="android.app.lib_name"                android:value="droidblaster"/>            <intent-filter>                <action android:name ="android.intent.action.MAIN"/>                <category                    android_name="android.intent.category.LAUNCHER"/>            </intent-filter>        </activity>    </application> </manifest> Create the file jni/Types.hpp. This header will contain common types and the header cstdint: #ifndef _PACKT_TYPES_HPP_ #define _PACKT_TYPES_HPP_   #include <cstdint>   #endif Let's write a logging class to get some feedback in the Logcat.      Create jni/Log.hpp and declare a new class Log.      Define the packt_Log_debug macro to allow the activating or deactivating of debug messages with a simple compile flag: #ifndef _PACKT_LOG_HPP_ #define _PACKT_LOG_HPP_   class Log { public:    static void error(const char* pMessage, ...);    static void warn(const char* pMessage, ...);    static void info(const char* pMessage, ...);    static void debug(const char* pMessage, ...); };   #ifndef NDEBUG    #define packt_Log_debug(...) Log::debug(__VA_ARGS__) #else    #define packt_Log_debug(...) #endif   #endif Implement the jni/Log.cpp file and implement the info() method. To write messages to Android logs, the NDK provides a dedicated logging API in the android/log.h header, which can be used similarly as printf() or vprintf() (with varArgs) in C: #include "Log.hpp"   #include <stdarg.h> #include <android/log.h>   void Log::info(const char* pMessage, ...) {    va_list varArgs;    va_start(varArgs, pMessage);    __android_log_vprint(ANDROID_LOG_INFO, "PACKT", pMessage,        varArgs);    __android_log_print(ANDROID_LOG_INFO, "PACKT", "n");    va_end(varArgs); } ... Write other log methods, error(), warn(), and debug(), which are almost identical, except the level macro, which are respectively ANDROID_LOG_ERROR, ANDROID_LOG_WARN, and ANDROID_LOG_DEBUG instead. Application events in NativeActivity can be processed with an event loop. So, create jni/EventLoop.hpp to define a class with a unique method run(). Include the android_native_app_glue.h header, which defines the android_app structure. It represents what could be called an applicative context, where all the information is related to the native activity; its state, its window, its event queue, and so on: #ifndef _PACKT_EVENTLOOP_HPP_ #define _PACKT_EVENTLOOP_HPP_   #include <android_native_app_glue.h>   class EventLoop { public:    EventLoop(android_app* pApplication);      void run();   private:    android_app* mApplication; }; #endif Create jni/EventLoop.cpp and implement the activity event loop in the run() method. Include a few log events to get some feedback in Android logs. During the whole activity lifetime, the run() method loops continuously over events until it is requested to terminate. When an activity is about to be destroyed, the destroyRequested value in the android_app structure is changed internally to indicate to the client code that it must exit. Also, call app_dummy() to ensure the glue code that ties native code to NativeActivity is not stripped by the linker. #include "EventLoop.hpp" #include "Log.hpp"   EventLoop::EventLoop(android_app* pApplication):        mApplication(pApplication) {}   void EventLoop::run() {    int32_t result; int32_t events;    android_poll_source* source;      // Makes sure native glue is not stripped by the linker.    app_dummy();      Log::info("Starting event loop");    while (true) {        // Event processing loop.        while ((result = ALooper_pollAll(-1, NULL, &events,                (void**) &source)) >= 0) {            // An event has to be processed.            if (source != NULL) {                source->process(mApplication, source);            }            // Application is getting destroyed.            if (mApplication->destroyRequested) {                Log::info("Exiting event loop");                return;            }        }    } } Finally, create jni/Main.cpp to define the program entry point android_main(), which runs the event loop in a new file Main.cpp: #include "EventLoop.hpp" #include "Log.hpp"   void android_main(android_app* pApplication) {    EventLoop(pApplication).run(); } Edit the jni/Android.mk file to define the droidblaster module (the LOCAL_MODULE directive). Describe the C++ files to compile the LOCAL_SRC_FILES directive with the help of the LS_CPP macro. Link droidblaster with the native_app_glue module (the LOCAL_STATIC_LIBRARIES directive) and android (required by the Native App Glue module), as well as the log libraries (the LOCAL_LDLIBS directive): LOCAL_PATH := $(call my-dir)   include $(CLEAR_VARS)   LS_CPP=$(subst $(1)/,,$(wildcard $(1)/*.cpp)) LOCAL_MODULE := droidblaster LOCAL_SRC_FILES := $(call LS_CPP,$(LOCAL_PATH)) LOCAL_LDLIBS := -landroid -llog LOCAL_STATIC_LIBRARIES := android_native_app_glue   include $(BUILD_SHARED_LIBRARY)   $(call import-module,android/native_app_glue)   Create jni/Application.mk to compile the native module for multiple ABIs. We will use the most basic ones, as shown in the following code: APP_ABI := armeabi armeabi-v7a x86 What just happened? Build and run the application. Of course, you will not see anything tremendous when starting this application. Actually, you will just see a black screen! However, if you look carefully at the LogCat view in Eclipse (or the adb logcat command), you will discover a few interesting messages that have been emitted by your native application in reaction to activity events. We initiated a Java Android project without a single line of Java code! Instead of referencing a child of Activity in AndroidManifest, we referenced the android.app.NativeActivity class provided by the Android framework. NativeActivity is a Java class, launched like any other Android activity and interpreted by the Dalvik Virtual Machine like any other Java class. However, we never faced it directly. NativeActivity is in fact a helper class provided with Android SDK, which contains all the necessary glue code to handle application events (lifecycle, input, sensors, and so on) and broadcasts them transparently to native code. Thus, a native activity does not eliminate the need for JNI. It just hides it under the cover! However, the native C/C++ module run by NativeActivity is executed outside Dalvik boundaries in its own thread, entirely natively (using the Posix Thread API)! NativeActivity and native code are connected together through the native_app_glue module. The Native App Glue has the responsibility of: Launching the native thread, which runs our own native code Receiving events from NativeActivity Routing these events to the native thread event loop for further processing The Native glue module code is located in ${ANDROID_NDK}/sources/android/native_app_glue and can be analyzed, modified, or forked at will. The headers related to native APIs such as, looper.h, can be found in ${ANDROID_NDK}/platforms/<Target Platform>/<Target Architecture>/usr/include/android/. Let's see in more detail how it works. More about the Native App Glue Our own native code entry point is declared inside the android_main() method, which is similar to the main methods in desktop applications. It is called only once when NativeActivity is instantiated and launched. It loops over application events until NativeActivity is terminated by the user (for example, when pressing a device's back button) or until it exits by itself. The android_main() method is not the real native application entry point. The real entry point is the ANativeActivity_onCreate() method hidden in the android_native_app_glue module. The event loop we implemented in android_main() is in fact a delegate event loop, launched in its own native thread by the glue module. This design decouples native code from the NativeActivity class, which is run on the UI thread on the Java side. Thus, even if your code takes a long time to handle an event, NativeActivity is not blocked and your Android device still remains responsive. The delegate native event loop in android_main() is itself composed, in our example, of two nested while loops. The outer one is an infinite loop, terminated only when activity destruction is requested by the system (indicated by the destroyRequested flag). It executes an inner loop, which processes all pending application events. ... int32_t result; int32_t events; android_poll_source* source; while (true) {    while ((result = ALooper_pollAll(-1, NULL, &events,            (void**) &source)) >= 0) {        if (source != NULL) {            source->process(mApplication, source);        }        if (mApplication->destroyRequested) {            return;        }    } } ... The inner For loop polls events by calling ALooper_pollAll(). This method is part of the Looper API, which can be described as a general-purpose event loop manager provided by Android. When timeout is set to -1, like in the preceding example, ALooper_pollAll() remains blocked while waiting for events. When at least one is received, ALooper_pollAll() returns and the code flow continues. The android_poll_source structure describing the event is filled and is then used by client code for further processing. This structure looks as follows: struct android_poll_source {    int32_t id; // Source identifier  struct android_app* app; // Global android application context    void (*process)(struct android_app* app,            struct android_poll_source* source); // Event processor }; The process() function pointer can be customized to process application events manually. As we saw in this part, the event loop receives an android_app structure in parameter. This structure, described in android_native_app_glue.h, contains some contextual information as shown in the following table: void* userData Pointer to any data you want. This is essential in giving some contextual information to the activity or input event callbacks. void (*pnAppCmd)(…) and int32_t (*onInputEvent)(…) These member variables represent the event callbacks triggered by the Native App Glue when an activity or an input event occurs. ANativeActivity* activity Describes the Java native activity (its class as a JNI object, its data directories, and so on) and gives the necessary information to retrieve a JNI context. AConfiguration* config Describes the current hardware and system state, such as the current language and country, the current screen orientation, density, size, and so on. void* savedState size_t and savedStateSize Used to save a buffer of data when an activity (and thus its native thread) is destroyed and later restored. AInputQueue* inputQueue Provides input events (used internally by the native glue). ALooper* looper Allows attaching and detaching event queues used internally by the native glue. Listeners poll and wait for events sent on a communication pipe. ANativeWindow* window and ARect contentRect Represents the "drawable" area on which graphics can be drawn. The ANativeWindow API, declared in native_window.h, allows retrieval of the window width, height, and pixel format, and the changing of these settings. int activityState Current activity state, that is, APP_CMD_START, APP_CMD_RESUME, APP_CMD_PAUSE, and so on. int destroyRequested When equal to 1, it indicates that the application is about to be destroyed and the native thread must be terminated immediately. This flag has to be checked in the event loop. The android_app structure also contains some additional data for internal use only, which should not be changed. Knowing all these details is not essential to program native programs but can help you understand what's going on behind your back. Let's now see how to handle these activity events. Summary The Android NDK allows us to write fully native applications without a line of Java code. NativeActivity provides a skeleton to implement an event loop that processes application events. Associated with the Posix time management API, the NDK provides the required base to build complex multimedia applications or games. In summary, we created NativeActivity that polls activity events to start or stop native code accordingly. We accessed the display window natively, like a bitmap, to display raw graphics. Finally, we retrieved time to make the application adapt to device speed using a monotonic clock. Resources for Article: Further resources on this subject: Android Native Application API [article] Organizing a Virtual Filesystem [article] Android Fragmentation Management [article]
Read more
  • 0
  • 0
  • 18485

article-image-implementing-concurrency-with-kotlin-tutorial
Savia Lobo
06 Oct 2018
10 min read
Save for later

Implementing Concurrency with Kotlin [Tutorial]

Savia Lobo
06 Oct 2018
10 min read
Correct concurrent code is one that allows for a certain variability in the order of its execution while still being deterministic on the result. For this to be possible, different parts of the code need to have some level of independence, and some degree of orchestration may also be required. This tutorial is an excerpt taken from the book, Learning Concurrency in Kotlin written by Miguel Angel Castiblanco Torres. In this book, you will learn how to handle errors and exceptions, as well as how to leverage multi-core processing. In this article, we will be introduced to concurrency and how Kotlin approaches concurrency challenges. The best way to understand concurrency is by comparing sequential code with concurrent code. Let's start by looking at some non-concurrent code: fun getProfile(id: Int) : Profile { val basicUserInfo = getUserInfo(id) val contactInfo = getContactInfo(id) return createProfile(basicUserInfo, contactInfo) } If I ask you what is going to be obtained first, the user information or the contact information – assuming no exceptions – you will probably agree with me that 100% of the time the user information will be retrieved first. And you will be correct. That is, first and foremost, because the contact information is not being requested until the contact information has already been retrieved: Timeline of getProfile And that's the beauty of sequential code: you can easily see the exact order of execution, and you will never have surprises on that front. But sequential code has two big issues: It may perform poorly compared to concurrent code It may not take advantage of the hardware that it's being executed on Let's say, for example, that both getUserInfo and getContactInfo call a web service, and each service will constantly take around one second to return a response. That means that getProfile will take not less than two seconds to finish, always. And since it seems like getContactInfo doesn't depend on getUserInfo, the calls could be done concurrently, and by doing so it would be possible can halve the execution time of getProfile. Let's imagine a concurrent implementation of getProfile: suspend fun getProfile(id: Int) { val basicUserInfo = asyncGetUserInfo(id) val contactInfo = asyncGetContactInfo(id) createProfile(basicUserInfo.await(), contactInfo.await()) } In this updated version of the code, getProfile() is a suspending function – notice the suspend modifier in its definition – and the implementation of asyncGetUserInfo() and asyncGetContactInfo() are asynchronous – which will not be shown in the example code to keep things simple. Because asyncGetUserInfo() and asyncGetContactInfo() are written to run in different threads, they are said to be concurrent. For now, let's think of it as if they are being executed at the same time – we will see later that it's not necessarily the case, but will do for now. This means that the execution of asyncGetContactInfo() will not depend on the completion of asyncGetUserInfo(), so the requests to the web services could be done around at the same time. And since we know that each service takes around one second to return a response, createProfile() will be called around one second after getProfile() is started, sooner than it could ever be in the sequential version of the code, where it will always take at least two seconds to be called. Let's take a look at how this may look: Concurrent timeline for getProfile But in this updated version of the code, we don't really know if the user information will be obtained before the contact information. Remember, we said that each of the web services takes around one second, and we also said that both requests will be started at around the same time. This means that if asyncGetContactInfo is faster than asyncGetUserInfo, the contact information will be obtained first; but the user information could be obtained first if asyncGetUserInfo returns first; and since we are at it, it could also happen that both of them return the information at the same time. This means that our concurrent implementation of getProfile, while possibly performing twice as fast as the sequential one, has some variability in its execution. That's the reason there are two await() calls when calling createProfile(). What this is doing is suspending the execution of getProfile() until both asyncGetUserInfo() and asyncGetContactInfo() have completed. Only when both of them have completed createProfile() will be executed. This guarantees that regardless of which of the concurrent call ends first, the result of getProfile() will be deterministic. And that's where the tricky part of concurrency is. You need to guarantee that no matter the order in which the semi-independent parts of the code are completed, the result needs to be deterministic. For this example, what we did was suspend part of the code until all the moving parts completed, but as we will see later in the book, we can also orchestrate our concurrent code by having it communicate between coroutines. Concurrency in Kotlin Now that we have covered the basics of concurrency, it's a good time to discuss the specifics of concurrency in Kotlin. This section will showcase the most differentiating characteristics of Kotlin when it comes to concurrent programming, covering both philosophical and technical topics. Non-blocking Threads are heavy, expensive to create, and limited—only so many threads can be created—So when a thread is blocked it is, in a way, being wasted. Because of this, Kotlin offers what is called Suspendable Computations; these are computations that can suspend their execution without blocking the thread of execution. So instead of, for example, blocking thread X to wait for an operation to be made in a thread Y, it's possible to suspend the code that has to wait and use thread X for other computations in the meantime. Furthermore, Kotlin offers great primitives like channels, actors, and mutual exclusions, which provide mechanisms to communicate and synchronize concurrent code effectively without having to block a thread. Being explicit Concurrency needs to be thought about and designed for, and because of that, it's important to make it explicit in terms of when a computation should run concurrently. Suspendable computations will run sequentially by default. Since they don't block the thread when suspended, there's no direct drawback: fun main(args: Array<String>) = runBlocking { val time = measureTimeMillis { val name = getName() val lastName = getLastName() println("Hello, $name $lastName") } println("Execution took $time ms") } suspend fun getName(): String { delay(1000) return "Susan" } suspend fun getLastName(): String { delay(1000) return "Calvin" } In this code, main() executes the suspendable computations getName() and getLastName() in the current thread, sequentially. Executing main() will print the following: This is convenient because it's possible to write non-concurrent code that doesn't block the thread of execution. But after some time and analysis, it becomes clear that it doesn't make sense to have getLastName() wait until after getName() has been executed since the computation of the latter has no dependency on the former. It's better to make it concurrent: fun main(args: Array<String>) = runBlocking { val time = measureTimeMillis { val name = async { getName() } val lastName = async { getLastName() } println("Hello, ${name.await()} ${lastName.await()}") } println("Execution took $time ms") } Now, by calling async {...} it's clear that both of them should run concurrently, and by calling await() it's requested that main() is suspended until both computations have a result: Readable Concurrent code in Kotlin is as readable as sequential code. One of the many problems with concurrency in other languages like Java is that often it's difficult to read, understand, and/or debug concurrent code. Kotlin's approach allows for idiomatic concurrent code: suspend fun getProfile(id: Int) { val basicUserInfo = asyncGetUserInfo(id) val contactInfo = asyncGetContactInfo(id) createProfile(basicUserInfo.await(), contactInfo.await()) } By convention, a function that is going to run concurrently by default should indicate this in its name, either by starting with async or ending in Async. This suspend method calls two methods that will be executed in background threads and waits for their completion before processing the information. Reading and debugging this code is as simple as it would be for sequential code. In many cases, it is better to write a suspend function and call it inside an async {} or launch {} block, rather than writing functions that are already async. This is because it gives more flexibility to the callers of the function to have a suspend function; that way the caller can decide when to run it concurrently, for example. In other cases, you may want to write both the concurrent and the suspend function. Leveraged Creating and managing threads is one of the difficult parts of writing concurrent code in many languages. As seen before, it's important to know when to create a thread, and almost as important to know how many threads are optimal. It's also important to have threads dedicated to I/O operations, while also having threads to tackle CPU-bound operations. And communicating/syncing threads is a challenge in itself. Kotlin has high-level functions and primitives that make it easier to implement concurrent code: To create a thread it's enough to call newSingleThreadContext(), a function that only takes the name of the thread. Once created, that thread can be used to run as many coroutines as needed. Creating a pool of threads is as easy, by calling newFixedThreadPoolContext() with the size and the name of the pool. CommonPool is a pool of threads optimal for CPU-bound operations. Its maximum size is the amount of cores in the machine minus one. The runtime will take charge of moving a coroutine to a different thread when needed. There are many primitives and techniques to communicate and synchronize coroutines, such as channels, mutexes, and thread confinement. Flexible Kotlin offers many different primitives that allow for simple-yet-flexible concurrency. You will find that there are many ways to do concurrent programming in Kotlin. Here is a list of some of the topics we will look at throughout the book: Channels: Pipes that can be used to safely send and receive data between coroutines. Worker pools: A pool of coroutines that can be used to divide the processing of a set of operations in many threads. Actors: A wrapper around a state that uses channels and coroutines as a mechanism to offer the safe modification of a state from many different threads. Mutual exclusions (Mutexes): A synchronization mechanism that allows the definition of a critical zone so that only one thread can execute at a time. Any coroutine trying to access the critical zone will be suspended until the previous coroutine leaves. Thread confinement: The ability to limit the execution of a coroutine so that it always happens in a specified thread. Generators (Iterators and sequences): Data sources that can produce information on demand and be suspended when no new information is required. All of these are tools that are at your fingertips when writing concurrent code in Kotlin, and their scope and use will help you to make the right choices when implementing concurrent code. To summarize, we learned the basics of concurrency and how to implement concurrency in Kotlin programming language. If you've enjoyed reading this excerpt, head over to the book, Learning Concurrency in Kotlin to learn how to use the Machine Learning Toolkit and best practices and tips to help you implement Splunk services effectively and efficiently. KotlinConf 2018: Kotlin 1.3 RC out and Kotlin/Native hits beta Kotlin 1.3 RC1 is here with compiler and IDE improvements Kotlin 1.3 M1 arrives with coroutines, and new experimental features like unsigned integer types
Read more
  • 0
  • 0
  • 18469
Modal Close icon
Modal Close icon