How-To Tutorials

article-image-loading-data-creating-app-and-adding-dashboards-and-reports-splunk

31 Oct 2014

13 min read

Loading data, creating an app, and adding dashboards and reports in Splunk

31 Oct 2014

0
0
10125

article-image-getting-started-with-amazon-machine-learning-workflow-tutorial

Melisha Dsouza

02 Sep 2018

14 min read

Getting started with Amazon Machine Learning workflow [Tutorial]

Melisha Dsouza

02 Sep 2018

14 min read

Amazon Machine Learning is useful for building ML models and generating predictions. It also enables the development of robust and scalable smart applications. The process of building ML models with Amazon Machine Learning consists of three operations: data analysis model training evaluation. The code files for this article are available on Github. This tutorial is an excerpt from a book written by Alexis Perrier titled Effective Amazon Machine Learning. The Amazon Machine Learning service is available at https://console.aws.amazon.com/machinelearning/. The Amazon ML workflow closely follows a standard Data Science workflow with steps: Extract the data and clean it up. Make it available to the algorithm. Split the data into a training and validation set, typically a 70/30 split with equal distribution of the predictors in each part. Select the best model by training several models on the training dataset and comparing their performances on the validation dataset. Use the best model for predictions on new data. As shown in the following Amazon ML menu, the service is built around four objects: Datasource ML model Evaluation Prediction The Datasource and Model can also be configured and set up in the same flow by creating a new Datasource and ML model. Let us take a closer look at each one of these steps. Understanding the dataset used We will use the simple Predicting Weight by Height and Age dataset (from Lewis Taylor (1967)) with 237 samples of children's age, weight, height, and gender, which is available at https://v8doc.sas.com/sashtml/stat/chap55/sect51.htm. This dataset is composed of 237 rows. Each row has the following predictors: sex (F, M), age (in months), height (in inches), and we are trying to predict the weight (in lbs) of these children. There are no missing values and no outliers. The variables are close enough in range and normalization is not required. We do not need to carry out any preprocessing or cleaning on the original dataset. Age, height, and weight are numerical variables (real-valued), and sex is a categorical variable. We will randomly select 20% of the rows as the held-out subset to use for prediction on previously unseen data and keep the other 80% as training and evaluation data. This data split can be done in Excel or any other spreadsheet editor: By creating a new column with randomly generated numbers Sorting the spreadsheet by that column Selecting 190 rows for training and 47 rows for prediction (roughly a 80/20 split) Let us name the training set LT67_training.csv and the held-out set that we will use for prediction LT67_heldout.csv, where LT67 stands for Lewis and Taylor, the creator of this dataset in 1967. As with all datasets, scripts, and resources mentioned in this book, the training and holdout files are available in the GitHub repository at https://github.com/alexperrier/packt-aml. It is important for the distribution in age, sex, height, and weight to be similar in both subsets. We want the data on which we will make predictions to show patterns that are similar to the data on which we will train and optimize our model. Loading the data on S3 Follow these steps to load the training and held-out datasets on S3: Go to your s3 console at https://console.aws.amazon.com/s3. Create a bucket if you haven't done so already. Buckets are basically folders that are uniquely named across all S3. We created a bucket named aml.packt. Since that name has now been taken, you will have to choose another bucket name if you are following along with this demonstration. Click on the bucket name you created and upload both the LT67_training.csv and LT67_heldout.csv files by selecting Upload from the Actions drop-down menu: Both files are small, only a few KB, and hosting costs should remain negligible for that exercise. Note that for each file, by selecting the Properties tab on the right, you can specify how your files are accessed, what user, role, group or AWS service may download, read, write, and delete the files, and whether or not they should be accessible from the Open Web. When creating the datasource in Amazon ML, you will be prompted to grant Amazon ML access to your input data. You can specify the access rules to these files now in S3 or simply grant access later on. Our data is now in the cloud in an S3 bucket. We need to tell Amazon ML where to find that input data by creating a datasource. We will first create the datasource for the training file ST67_training.csv. Declaring a datasource Go to the Amazon ML dashboard, and click on Create new... | Datasource and ML model. We will use the faster flow available by default: As shown in the following screenshot, you are asked to specify the path to the LT67_training.csv file {S3://bucket}{path}{file}. Note that the S3 location field automatically populates with the bucket names and file names that are available to your user: Specifying a Datasource name is useful to organize your Amazon ML assets. By clicking on Verify, Amazon ML will make sure that it has the proper rights to access the file. In case it needs to be granted access to the file, you will be prompted to do so as shown in the following screenshot: Just click on Yes to grant access. At this point, Amazon ML will validate the datasource and analyze its contents. Creating the datasource An Amazon ML datasource is composed of the following: The location of the data file: The data file is not duplicated or cloned in Amazon ML but accessed from S3 The schema that contains information on the type of the variables contained in the CSV file: Categorical Text Numeric (real-valued) Binary It is possible to supply Amazon ML with your own schema or modify the one created by Amazon ML. At this point, Amazon ML has a pretty good idea of the type of data in your training dataset. It has identified the different types of variables and knows how many rows it has: Move on to the next step by clicking on Continue, and see what schema Amazon ML has inferred from the dataset as shown in the next screenshot: Amazon ML needs to know at that point which is the variable you are trying to predict. Be sure to tell Amazon ML the following: The first line in the CSV file contains te column name The target is the weight We see here that Amazon ML has correctly inferred the following: sex is categorical age, height, and weight are numeric (continuous real values) Since we chose a numeric variable as the target Amazon ML, will use Linear Regression as the predictive model. For binary or categorical values, we would have used Logistic Regression. This means that Amazon ML will try to find the best a, b, and c coefficients so that the weight predicted by the following equation is as close as possible to the observed real weight present in the data: predicted weight = a * age + b * height + c * sex Amazon ML will then ask you if your data contains a row identifier. In our present case, it does not. Row identifiers are useful when you want to understand the prediction obtained for each row or add an extra column to your dataset later on in your project. Row identifiers are for reference purposes only and are not used by the service to build the model. You will be asked to review the datasource. You can go back to each one of the previous steps and edit the parameters for the schema, the target and the input data. Now that the data is known to Amazon ML, the next step is to set up the parameters of the algorithm that will train the model. Understanding the model We select the default parameters for the training and evaluation settings. Amazon ML will do the following: Create a recipe for data transformation based on the statistical properties it has inferred from the dataset Split the dataset (ST67_training.csv) into a training part and a validation part, with a 70/30 split. The split strategy assumes the data has already been shuffled and can be split sequentially. The recipe will be used to transform the data in a similar way for the training and the validation datasets. The only transformation suggested by Amazon ML is to transform the categorical variable sex into a binary variable, where m = 0 and f = 1 for instance. No other transformation is needed. The default advanced settings for the model are shown in the following screenshot: We see that Amazon ML will pass over the data 10 times, shuffle splitting the data each time. It will use an L2 regularization strategy based on the sum of the square of the coefficients of the regression to prevent overfitting. We will evaluate the predictive power of the model using our LT67_heldout.csv dataset later on. Regularization comes in 3 levels with a mild (10^-6), medium (10^-4), or aggressive (10^-02) setting, each value stronger than the previous one. The default setting is mild, the lowest, with a regularization constant of 0.00001 (10^-6) implying that Amazon ML does not anticipate much overfitting on this dataset. This makes sense when the number of predictors, three in our case, is much smaller than the number of samples (190 for the training set). Clicking on the Create ML model button will launch the model creation. This takes a few minutes to resolve, depending on the size and complexity of your dataset. You can check its status by refreshing the model page. In the meantime, the model status remains pending. At that point, Amazon ML will split our training dataset into two subsets: a training and a validation set. It will use the training portion of the data to train several settings of the algorithm and select the best one based on its performance on the training data. It will then apply the associated model to the validation set and return an evaluation score for that model. By default, Amazon ML will sequentially take the first 70% of the samples for training and the remaining 30% for validation. It's worth noting that Amazon ML will not create two extra files and store them on S3, but instead create two new datasources out of the initial datasource we have previously defined. Each new datasource is obtained from the original one via a Data rearrangement JSON recipe such as the following: { "splitting": { "percentBegin": 0, "percentEnd": 70 } } You can see these two new datasources in the Datasource dashboard. Three datasources are now available where there was initially only one, as shown by the following screenshot: While the model is being trained, Amazon ML runs the Stochastic Gradient algorithm several times on the training data with different parameters: Varying the learning rate in increments of powers of 10: 0.01, 0.1, 1, 10, and 100. Making several passes over the training data while shuffling the samples before each path. At each pass, calculating the prediction error, the Root Mean Squared Error (RMSE), to estimate how much of an improvement over the last pass was obtained. If the decrease in RMSE is not really significant, the algorithm is considered to have converged, and no further pass shall be made. At the end of the passes, the setting that ends up with the lowest RMSE wins, and the associated model (the weights of the regression) is selected as the best version. Once the model has finished training, Amazon ML evaluates its performance on the validation datasource. Once the evaluation itself is also ready, you have access to the model's evaluation. Evaluating the model Amazon ML uses the standard metric RMSE for linear regression. RMSE is defined as the sum of the squares of the difference between the real values and the predicted values: Here, ŷ is the predicted values, and y the real values we want to predict (the weight of the children in our case). The closer the predictions are to the real values, the lower the RMSE is. A lower RMSE means a better, more accurate prediction. Making batch predictions We now have a model that has been properly trained and selected among other models. We can use it to make predictions on new data. A batch prediction consists in applying a model to a datasource in order to make predictions on that datasource. We need to tell Amazon ML which model we want to apply on which data. Batch predictions are different from streaming predictions. With batch predictions, all the data is already made available as a datasource, while for streaming predictions, the data will be fed to the model as it becomes available. The dataset is not available beforehand in its entirety. In the Main Menu select Batch Predictions to access the dashboard predictions and click on Create a New Prediction: The first step is to select one of the models available in your model dashboard. You should choose the one that has the lowest RMSE: The next step is to associate a datasource to the model you just selected. We had uploaded the held-out dataset to S3 at the beginning of this chapter (under the Loading the data on S3 section) but had not used it to create a datasource. We will do so now.When asked for a datasource in the next screen, make sure to check My data is in S3, and I need to create a datasource, and then select the held-out dataset that should already be present in your S3 bucket: Don't forget to tell Amazon ML that the first line of the file contains columns. In our current project, our held-out dataset also contains the true values for the weight of the students. This would not be the case for "real" data in a real-world project where the real values are truly unknown. However, in our case, this will allow us to calculate the RMSE score of our predictions and assess the quality of these predictions. The final step is to click on the Verify button and wait for a few minutes: Amazon ML will run the model on the new datasource and will generate predictions in the form of a CSV file. Contrary to the evaluation and model-building phase, we now have real predictions. We are also no longer given a score associated with these predictions. After a few minutes, you will notice a new batch-prediction folder in your S3 bucket. This folder contains a manifest file and a results folder. The manifest file is a JSON file with the path to the initial datasource and the path to the results file. The results folder contains a gzipped CSV file: Uncompressed, the CSV file contains two columns, trueLabel, the initial target from the held-out set, and score, which corresponds to the predicted values. We can easily calculate the RMSE for those results directly in the spreadsheet through the following steps: Creating a new column that holds the square of the difference of the two columns. Summing all the rows. Taking the square root of the result. The following illustration shows how we create a third column C, as the squared difference between the trueLabel column A and the score (or predicted value) column B: As shown in the following screenshot, averaging column C and taking the square root gives an RMSE of 11.96, which is even significantly better than the RMSE we obtained during the evaluation phase (RMSE 14.4): The fact that the RMSE on the held-out set is better than the RMSE on the validation set means that our model did not overfit the training data, since it performed even better on new data than expected. Our model is robust. The left side of the following graph shows the True (Triangle) and Predicted (Circle) Weight values for all the samples in the held-out set. The right side shows the histogram of the residuals. Similar to the histogram of residuals we had observed on the validation set, we observe that the residuals are not centered on 0. Our model has a tendency to overestimate the weight of the students: In this tutorial, we have successfully performed the loading of the data on S3 and let Amazon ML infer the schema and transform the data. We also created a model and evaluated its performance. Finally, we made a prediction on the held -out dataset. To understand how to leverage Amazon's powerful platform for your predictive analytics needs, check out this book Effective Amazon Machine Learning. Four interesting Amazon patents in 2018 that use machine learning, AR, and robotics Amazon Sagemaker makes machine learning on the cloud easy Amazon ML Solutions Lab to help customers “work backwards” and leverage machine learning

0
0
10120

article-image-mission-running-eve-online

Packt

17 Oct 2012

14 min read

Mission Running in EVE Online

Packt

17 Oct 2012

14 min read

Mission types Missions in EVE fall into five general categories: Courier, Kill, Mining, Trade, and Special missions. Courier, Kill, Mining, and Trade missions are further categorized into five different levels: level I through level V. Let's take a closer look at each category to see what each type of mission entails. Courier missions Courier missions are simply delivery missions where you move items from one station to another. The type of items can range from common items found on the market to mission-specific cargo containers. The size and quantity of items to move can vary greatly and can either be moved using whatever ship you are currently flying or may require the use of an industrial ship. While the ISK reward for courier missions is a bit on the low side, it having no negative standing impact on opposing factions is a huge positive. This, added to the fact that courier missions are quite easy and can often be completed very quickly, means they are generally worth running. Every so often you will receive a courier mission that takes you into Low-Sec space and with the risks involved in Low-Sec, it is best to decline these missions. You are able to decline one mission every four hours per agent without taking a negative standing hit with that agent. Be very careful when declining missions, since if your standing falls too low with an agent you will no longer be able to receive mission offers from that agent. Kill missions By far the most common, the most profitable, and let's be honest, the most fun missions to run are kill missions. The only thing better than flying beautiful space ships is seeing beautiful space ships blow up. There are currently two types of kill missions. In one, you warp to a set of coordinates in space where you engage NPC targets and in the other you warp to an acceleration gate which takes you into a series of pockets all connected by more acceleration gates. Think of this second type of kill mission like a dungeon with multiple rooms, in which you fight NPC targets in each of the rooms. Kill missions are a great way to make ISK, especially when you are able to access higher-level agents. However, as you can guess, kill missions also get more and more difficult the higher the level of the agent. Another thing that you need to be very careful about when running kill missions is that you run the risk of having negative standing impact on factions opposed to the faction you are running missions for. For example, if you were running missions for the Caldari state and you completed a mission that had you destroy ships belonging to or friendly with the Gallente Federation, then you would lose standing with the Gallente. A great way to negate the standing loss is to run missions for as many agents as you can find and to decline any mission that will have you attack the ships of another empire. But remember you can only decline one mission per agent every four hours, hence having multiple agents. Agents and how they work Now would be a great time to take a closer look at the agent system of EVE. Each agent works in a specific division of a NPC corporation. The division that the agent works in determines the type of missions you will receive (more on this later), and the NPC corporation that the agent works for determines which faction your standing rewards will affect. An agent's information sheet is shown in the following image: As you can see, the agent Aariwa Tenanochi works for the Home Guard in their Security division and is a level 1 agent. Aariwa can be found in the Nonni system at the Home Guard Assembly Plant located at planet III Moon 1. You can also see that the only requirement to receive missions from Aariwa is that you meet his standing requirements. Doing the tutorial missions will allow you to access all level 1 agents for your chosen faction. All agents are rated level 1 to level 5. The higher the level of the agent the harder their missions are, and the harder the mission the better and bigger the reward. The difficulty level of each agent level can best be described as follows: Level 1: Very Easy. Designed for new players to learn about mission running and PvE in general. Frigate or destroyer for kill missions, short range and low cost items for courier and trade missions, and small amounts of low grade ore or minerals for mining missions. Level 2: Easy. Still designed beginner players, but will require you to start thinking tactically. Destroyer or cruiser for kill missions, slightly longer range and higher cost items for courier and trade missions, and higher amounts of low grade ore or minerals for mining missions. Level 3: Medium. You will need to have a firm understanding of how your ship works and have solid tactics for it. Battlecruiser or Heavy Assault Cruiser for kill missions, much longer range and higher cost items for courier and trade missions, and large amounts of low grade ore or minerals for mining missions. Level 4: Hard. You will need to understand your ship inside out, have solid tactics, and a solid fit for your ship. Battleship for kill missions, very long range and very costly items for courier and trade missions, and very large amounts of low grade ore or minerals for mining missions. Level 5: Very Hard. Same as level 4 agents but only found in Low-Sec systems and comes with all the risks of Low-Sec. It is a good idea to only do these missions in a fleet with a group of people. The standing requirements to access the different level of agents are shown in the following table: Level 1 Level 2 Level 3 Level 4 Level 5 Standing required - 1.00 3.00 5.00 7.00 While you can use larger ships to run lower level kill missions, it is generally not a good idea. First, there can be ship limitation for a mission that does not allow you to use your larger ship and second, while your larger ship will be able to handle the incoming damage much easier, the larger weapons that your ship uses will have a harder time hitting the smaller targets. And since ammo costs ISK, it cuts into your profit per mission. I generally like to use the cheapest ship I can get away with when running missions. This ensures that if I ever get scanned down and killed by player pirates, my loss is minimal. Understanding your foe The biggest key to success when running kill missions is understanding who your targets will be and how they like to fight. Whenever you are offered a kill mission by an agent, you can find out which faction your target belongs to inside the mission description, either in the text of the description or by an emblem representing your target's faction. It is important to know which faction your target belongs to because then you will know what kind of damage to expect from your target. In the mission offer shown in the following image you can see that your targets for this mission are Rogue Drones: There are four types of damage, EM, Thermal , Kinetic, and Explosive, and each faction has a preference toward specific damage types. Once you know what kind of damage to expect you can then tailor the defense of your ship for maximum protection towards that damage. Knowing which faction your target belongs to will also tell you which type of damage you should be doing to maximize your damage output. Each faction has weaknesses towards specific damage types. Along with what types of damage to defend against and what types of damage to utilize, knowing which faction your target belongs to will also tell you what kind of special tactic ships you may encounter in your mission. For example, the Caldari likes to use ECM to jam your targeting systems and the Amarr uses energy neutralizers to drain your ship's capacitor, leaving you unable to use weapons or even to warp away. The following table shows all the factions and the damage types to defend and use: NPC faction Damage type to defend Damage type to deal Guristas Kinetic, Thermal Kinetic, Thermal Serpentis Thermal, Kinetic Thermal Blood Raider EM, Thermal EM, Thermal Sansha's Nation EM, Thermal EM, Thermal Angel Cartel Explosive, Kinetic, Thermal, EM Explosive Mordu's Legion Kinetic, Thermal, Explosive, EM Thermal, Kinetic Mercenary Kinetic, Thermal Thermal, Kinetic Republic Fleet Explosive, Thermal, Kinetic, EM Explosive, Kinetic Caldari Navy Kinetic, Thermal Kinetic, Thermal Amarr Navy EM, Thermal, Kinetic EM, Thermal Federation Navy Thermal, Kinetic Kinetic, Thermal Rogue Drones Explosive, Kinetic, EM, Thermal EM Thukker Tribe Explosive, Thermal EM CONCORD EM, Thermal, Kinetic, Explosive Explosive, Kinetic Equilibrium of Mankind Kinetic, Thermal Kinetic, EM You will most likely have noticed that several factions use the same damage types but they are listed in different orders. The damage types are listed this way because the damage output or weakness is not divided evenly. So for the Serpentis pirate faction for example, they prefer a combination of Kinetic and Thermal damage with a higher percentage being Kinetic damage than Thermal damage. What ship to use to maximize earnings The first question that most new mission runners ask is "What race of ships should I use?" While each race has its own advantages and disadvantages and are all very good choices, in my opinion Gallente drone ships are the best ships to use for mission running, from a pure ISK making stand point. Gallente drone ships are the best when it comes to mission running because of the lack of ammo costs and the versatility they offer. With drones being your primary method of dealing damage you do not have to worry about the cost associated with needing a large supply of ammo like the Caldari or Minmatar. What about the Amarr you say? They don't need ammo. While it is true that the Amarr also do not require ammo, the heavy capacitor drain of lasers limits the amount of defense your ship can have. Drone ships do not need to fit weapons in their high slots and therefore you can fit your ship with the maximum amount of defense possible. Drone ships can also utilize the speed and range of drones to their advantage. By using modules such as the Drone Link Augmentor you can increase the range of your drones so that you can comfortably sit outside the attack range of your targets and let your drones do all the work. Being outside of the attack range also means that you are outside the range of electronic warfare, so it's a win-win situation. The best feature of drone ships is the ability to carry different types of drones. Light Drones for frigate-sized targets, Medium Drones for cruiser-sized targets, and Heavy Drones for battleship-sized targets. You can even carry Electronic Warfare Drones that can jam your targets, drain their capacitors, or even provide more defense for your ship by way of shield or armor repairs. How should I fit my ship? The second most common question is "How should I have my ship fitted for mission running?" Ship fitting is very subjective and almost an art form in itself. I can easily go on for pages about the different concepts behind ship fittings and why one way is better than another but there will always be people that will disagree because what works for one person does not necessarily work for you. The best thing to do is to visit websites such as eve.battleclinic.com/browse_loadouts.php to get ideas on how ship fittings should work and to use 3rd party software such as Python Fitting Assistant to come up with fittings that suit your style of play the best. When browsing fittings on BattleClinic, it is a good idea to make sure the filters at the bottom right are set for the most recent expansions. You wouldn't want to use a completely out-of-date fitting that will only get you killed. Basic tactics When you start a kill mission you will likely be faced with two different scenarios. The first is as soon as you come out of warp, targets will be on the offensive and immediately start attacking you. The second and more favorable scenario is to warp onto the field with multiple groups of targets in a semi-passive state. No matter the scenario you come across, the following are a few simple tactics will ensure everything goes that much smoother. These tactics are as follows: Zoom out. I know it's hard because your ship is so pretty, but zoom the camera out. You need to be able to have a high level of situational awareness at all times and you can't do that if you're zoomed in on your ship all the time. Finish each wave or group of targets before engaging other targets. This will ensure that you will never be too outnumbered and be able to more easily handle the incoming damage. Kill the tacklers first. Tacklers are ships that use electronic warfare to slow your ship or to prevent you from warping away. Since flying out of range or warping away is the best way of escape when things go bad, killing off the tacklers first will give you the best chance to escape intact. Kill the largest targets first. Taking out the largest targets first gives you two critical advantages. The first is you are taking a lot of the damage against you off the field and the second is that larger ships are much easier to hit. Save structures for last. If part of your mission is to shoot or destroy a structure, save that for last. The majority of times, firing on a structure will cause all the hostiles to attack you at once and may even spawn stationary defenses. This tactic does not apply to defensive towers, such as missile and tackling towers. You should always kill these towers as soon as possible. Mining missions Mining missions come in two flavors. The first will have you travel to a set of coordinates given to you by your agent, to mine a specific amount of a specific type of ore and then return to your agent with the ore you have mined. The second will simply have you supply your agent with an amount of ore or mineral. For the second type of mining mission you can either mine and refine the ore yourself or purchase it from the market. The ISK reward for mining missions is really bad and in general you will make more ISK if you had simply spent the time mining, but once again, having no negative standing impact on opposing factions is a huge plus. So if you are a career miner, it can be worth it for you to run these missions for the standings gain. After all, you will have to increase your standing in order to maximize your refine yield. It would be best to only accept the missions in which you already have the requested ore or mineral in your stock and to decline the rest. Trade missions Trade missions are simple missions that require you to provide items to your agent at a specific station. These items can either be made by you or purchased off the market. Like courier and mining missions, the only upside to trade missions is that they can be completed without the negative standing impact on opposing factions. But with the high amount of ISK needed to complete these missions and the time involved, it is best to avoid these missions. If you choose to do these missions, check to see if the item required is on sale at the destination station. If it is on sale there, you can often purchase it and complete the mission without ever leaving your current station.

0
0
10119

article-image-2018-new-year-resolutions-algorithmic-world-part-1-of-3

Sugandha Lahoti

03 Jan 2018

6 min read

2018 new year resolutions to thrive in an Algorithmic World - Part 1 of 3

Sugandha Lahoti

03 Jan 2018

6 min read

We often think of Data science and machine learning as skills essential to a niche group of researchers, data scientists, and developers. But the world as we know today revolves around data and algorithms, just as it used to revolve around programming a decade back. As data science and algorithms get integrated into all aspects of businesses across industries, data science like Microsoft Excel will become ubiquitous and will serve as a handy tool which makes you better at your job no matter what your job is. Knowing data science is key to having a bright career in this algoconomy (algorithm driven economy). If you are big on new year resolutions, make yourself a promise to carve your place in the algorithm-powered world by becoming data science savvy. Follow these three resolutions to set yourself up for a bright data-driven career. Get the foundations right: Start with the building blocks of data science, i.e. developing your technical skills. Stay relevant: Keep yourself updated on the latest developments in your field and periodically invest in reskilling and upskilling. Be mindful of your impact: Finally, always remember that your work has real-world implications. Choose your projects wisely and your project goals, hypotheses, and contributors with even more care. In this three-part series, we expand on how data professionals could go about achieving these three resolutions. But the principles behind the ideas are easily transferable to anyone in any job. Think of them as algorithms that can help you achieve your desired professional outcome! You simply need to engineer the features and fine-tune the hyperparameters specific to your industry and job role. 1st Resolution: Learn the building blocks of data science If you are interested in starting a career in data science or in one that involves data, here is a simple learning roadmap for you to develop your technical skills. Start off with learning a data-friendly programming language, one that you find easy and interesting. Next, brush up your statistics skills. Nothing fancy, just your high school math and stats would do nicely. Next, learn about algorithms - what they do, what questions they answer, how many types are there and how to write one. Finally, you can put all that learning to practice by building models on top of your choice of Machine Learning framework. Now let’s see, how you can accomplish each of these tasks 1. Learn Python or any another popular data friendly programming language you find interesting (Learning period: 1 week - 2 months) If you see yourself as a data scientist in the near future, knowing a programming language is one of the first things to check off your list. We suggest you learn a data-friendly programming language like Python or R. Python is a popular choice because of its strong, fast, and easy computational capabilities for the Data Science workflow. Moreover, because of a large and active community, the likelihood of finding someone in your team or your organization who knows Python is quite high, which is an added advantage. “Python has become the most popular programming language for data science because it allows us to forget about the tedious parts of programming and offers us an environment where we can quickly jot down our ideas and put concepts directly into action.” - Sebastian Raschka We suggest learning the basics from the book Learn Python in 7 days by Mohit, Bhaskar N. Das. Then you can move on to learning Python specifically for data science with Python Data Science Essentials by Alberto Boschetti. Additionally, you can learn R, which is a highly useful language when it comes to statistics and data. For learning R, we recommend R Data science Essentials by Raja B. Koushik. You can learn more about how Python and R stand against each other in the data science domain here. Although R and Python are the most popular choices for new developers and aspiring data scientists, you can also use Java for data science, if that is your cup of tea. Scala is another alternative. 2. Brush up on Statistics (Learning period: 1 week - 3 weeks) While you are training your programming muscle, we recommend that you brush through basic mathematics (probability and statistics). Remember, you already know everything to get started with data science from your high school days. You just need to refresh your memory with a little practice. A good place to start is to understand concepts like standard deviation, probability, mean, mode, variance, kurtosis among others. Now, your normal high-school books should be enough to get started, however, an in-depth understanding is required to leverage the power of data science. We recommend the book Statistics for Data Science by James D. Miller for this. 3. Learn what machine learning algorithms do and which ones to learn (Learning period: 1 month - 3 months) Machine Learning is a powerful tool to make predictions based on huge amounts of data. According to a recent study, in the next ten years, ML algorithms are expected to replace a quarter of the jobs across the world, in fields like transport, manufacturing, architecture, healthcare and many others. So the next step in your data science journey is learning about machine learning algorithms. There are new algorithms popping up almost every day. We’ve collated a list of top ten algorithms that you should learn to effectively design reliable and robust ML systems. But fear not, you don’t need to know all of them to get started. Start with some basic algorithms that are majorly used in the real world applications like linear regression, naive bayes, and decision trees. 4. Learn TensorFlow, Keras, or any other popular machine learning framework (Learning period: 1 month - 3 months) After you have familiarized yourself with some of the machine learning algorithms, it is time you put that learning to practice by building models based on those algorithms. While there are many cloud-based machine learning options that have click-based model building features available, the best way to learn a skill is to get your hands dirty. There is a growing range of frameworks that make it easy to build complex models while allowing for high degrees of customization. Here is a list of top 10 deep learning frameworks at your disposal to choose from. Our favorite pick is TensorFlow. It’s Python-based, backed by Google, has a very good documentation, and there are tons of tutorials and videos available on the internet to guide you. You can find a comprehensive list of books for learning Tensorflow here. We also recommend learning Keras, which is a good option if you have some knowledge of Python programming and want to get started with deep learning. Try the book Deep Learning with Keras, by Antonio Gulli and Sujit Pal, to get you started. If you find learning from multiple sources daunting, just learn from Sebastian Raschka’s Python machine learning book. Once you have got your fundamentals right, it is important to stay relevant through continuous learning and reskilling. Check out part 2 where we explore how you could about doing this in a systematic and time efficient manner. In part 3, we look at ways you can own your work and become aware of its outcome.

0
0
10111

article-image-effective-text-generation-editing-and-translation-with-chatgpt

Valentina Alto

13 Jun 2023

7 min read

Effective Text Generation, Editing and Translation with ChatGPT

Valentina Alto

13 Jun 2023

7 min read

This article is an excerpt from the book, Modern Generative AI with ChatGPT and OpenAI Models, by Valentina Alto. This book will help harness the power of AI with innovative, real-world applications, and unprecedented productivity boosts, powered by the latest advancements in AI technology like ChatGPT and OpenAI.In the world of natural language processing, ChatGPT stands as a powerful tool for various text-related tasks. From generating creative and coherent text to providing translations and editing assistance, ChatGPT offers a wide range of functionalities. In this article, we will explore how to harness the capabilities of ChatGPT to accomplish tasks such as generating engaging content, translating text between languages, and receiving helpful suggestions for editing. With practical examples and step-by-step instructions, we will unlock the potential of ChatGPT as a versatile text companion for developers and content creators alike.As a language model, ChatGPT is particularly suited for generating text based on users’ instructions. For example, you could ask ChatGPT to generate emails, drafts, or templates that target a specific audience:Figure 1: Example of an email generated by ChatGPTAnother example might be asking ChatGPT to create a pitch structure for a presentation you have to prepare: Figure 2 – Slideshow agenda and structure generated by ChatGPTYou can also generate blog posts or articles about trending topics this way. Here is an example:Figure 3 – blog post with relevant tags and SEO by ChatGPTWe can even get ChatGPT to reduce the size of the post to make it fit for a tweet. Here is how we can do this: Figure 4 – ChatGPT shrinks an article into a Twitter postFinally, ChatGPT can also generate video or theatre scripts, including the scenography and the suggested editing. The following figure shows an example of a theatre dialog between a person and ChatGPT: Figure 5– Theatre dialog with scenography generated by ChatGPTI only provided a truncated version to keep you in suspense regarding the ending…Improving writing skills and translationSometimes, rather than generating new content, you might want to revisit an existing piece of text. It this be for style improvement purposes, audience changes, language translation, and so on.Let’s look at some examples. Imagine that I drafted an email to invite a customer of mine to a webinar. I wrote two short sentences. Here, I want ChatGPT to improve the form and style of this email since the target audience will be executive-level:Figure 6 – Example of an email revisited by ChatGPT to target an executive audienceNow, let’s ask the same thing but with a different target audience: Figure 6 – Example of the same email with a different audience, generated by ChatGPTChatGPT can also give you some feedback about your writing style and structure.Imagine, for example, that you wrote a script with scenography for your YouTube channel. You included the speech as well as images, clips, and video editing activities. You also know that your typical audience is between 15 and 25 years old. You want feedback on your script and ask for this from ChatGPT: Figure 7 – Example of ChatGPT providing feedback on a video script As you can see, not only was ChatGPT able to give me feedback about the writing style, but also it suggested how I could improve the scenography of the whole video, by including more visuals.Again, imagine you wrote an introduction for an essay titled The History of Natural Language Processing and you want some feedback about the writing style and its consistency with the title: Figure 8 – Example of ChatGPT giving feedback on an introduction for an essayLet’s also ask ChatGPT to make concrete examples of the attention-grabbing anecdote it talked about in its response: Figure 9 – Example of ChatGPT elaborating on something it mentioned I’m also interested in knowing whether my introduction was consistent with the title or whether I’m taking the wrong direction:Figure 10 – ChatGPT provides feedback about the consistency of the introduction with the titleI was impressed by this last one. ChatGPT was smart enough to see that there was no specific mention of the history of NLP in my introduction. Nevertheless, it sets up the expectation about that topic to be treated later on. This means that ChatGPT also has expertise in terms of how an essay should be structured and it was very precise in applying its judgment, knowing that it was just an introduction.It is also impressive to note how the model can give different feedback, depending on the context. With the video script, ChatGPT’s feedback took into account that the final consumption of that content would have been on screen. On the other hand, the essay’s introduction lives in a more formal and academic context, with a specific structure, that ChatGPT was able to capture once more.Last but not least, ChatGPT is also an excellent tool for translation. It knows at least 95 languages (if you have doubts about whether your language is supported, you can always ask ChatGPT directly). Here, however, there is a consideration that might arise: what is the added value of ChatGPT for translation when we already have cutting-edge tools such as Google Translate?To answer this question, we have to consider some key differentiators and how we can leverage ChatGPT’s embedded translations capabilities:ChatGPT can capture the intent. This means that you could also bypass the translation phase since it is something that ChatGPT can do in the backend. For example, if you write a prompt to produce a social media post in French, you could write that prompt in any language you want – ChatGPT will automatically detect it (without the need to specify it in advance) and understand your intent:Figure 11 – Example of ChatGPT generating an output in a language that is different from the inputChatGPT can capture the more refined meaning of particular slang or idioms. This allows for a translation that is not literal so that it can preserve the underlying meaning. Namely, let’s consider the British expression It’s not my cup of tea, to indicate something that is not the type of thing you like. Let’s ask both ChatGPT and Google Translate to translate it into Italian: Figure 12 – Comparison between ChatGPT and Google Translate while translating from English into ItalianAs you can see, ChatGPT can provide several Italian idioms that are equivalent to the original one, also in their slang format. On the other hand, Google Translate performed a literal translation, leaving behind the real meaning of the idiom. As with any other task, you can always provide context to ChatGPT. So, if you want your translation to have a specific slang or style, you can always specify it in the prompt. Or, even funnier, you can ask ChatGPT to translate your prompt with a sarcastic touch: Figure 5.20 – Example of ChatGPT translating a prompt with a sarcastic touch.The original content from: OpenAI’s Wikipedia page: https://it.wikipedia.org/wiki/OpenAISummaryIn conclusion, ChatGPT is able not only to generate new text but also to manipulate existing material to tailor it to your needs. It has also proven to be very precise at translating between languages, also keeping the jargon and language-specific expressions intact.Author BioValentina Alto graduated in 2021 in Data Science. Since 2020 she has been working in Microsoft as Azure Solution Specialist and, since 2022, she focused on Data&AI workloads within the Manufacturing and Pharmaceutical industry. She has been working on customers’ projects closely with system integrators to deploy cloud architecture with a focus on datalake house and DWH, data integration and engineering, IoT and real-time analytics, Azure Machine Learning, Azure cognitive services (including Azure OpenAI Service), and PowerBI for dashboarding. She holds a BSc in Finance and an MSc degree in Data Science from Bocconi University, Milan, Italy. Since her academic journey she has been writing Tech articles about Statistics, Machine Learning, Deep Learning and AI on various publications. She has also written a book about the fundamentals of Machine Learning with Python. LinkedIn Medium

0
0
10101

article-image-using-your-smart-watch-control-networked-leds

Andrew Fisher

04 May 2015

15 min read

Using your smart watch to control networked LEDs

Andrew Fisher

04 May 2015

15 min read

Introduction In the middle of this year, the hacker’s watchmaker, Pebble, will launch their next products. These watches, and the non-color ones before them, are remarkably capable devices that can easily talk with other services - whether it’s to get notified of your latest mention on Twitter or get the latest weather report. Pebble have also made building applications easy by allowing you to build in JavaScript. In this post I’ll use JavaScript to show you how easy it is to build a watch application that can interact with an external service. I could build a simple web application that gets your latest train times or Yelp reviews, but where’s the fun in that? Instead I’m going to pair a Pebble with one of the other current darlings of the hacker community, the ESP8266 WiFi module. The ESP8266 wireless module made its entrance into the hardware community in late 2014 and things haven’t been quite the same at hacker spaces around the world. This is a device that has more memory than an Arduino, natively uses WiFi, can be programmed in C or Lua (with a custom firmware), has low power consumption and costs less than $5 per module. It is still early days in the ESP8266 community so the edges are a bit rough, but things are stable enough to get playing and what better combination of things to play with than making some LEDs controllable wirelessly using your smart watch as an interface. Design approach To make the wireless LED device I’m going to use an ESP8266 with the NodeMCU firmware on it with some custom code that exposes a web server. The web server will accept a post request that has values for red, green and blue (between 0 - off, and 255 - full on, for each channel). These values will be interpreted and used to set the color of a strip of WS2812 controllable LEDs (aka NeoPixels). Once the web service is exposed, the Pebble watch application simply needs to make an HTTP request to the ESP8266 web server, POSTing the data to it of the colour to turn the pixels. Bill of materials You’ll need the following to build this project Qty Item Price Notes 1 Pebble Watch $99 Any Pebble watch will work 1 USB to Serial programmer $15 A 3.3 and 5v switchable USB-Serial converter is really useful. http://www.dfrobot.com/index.php?route=product/product&product_id=147 This project requires a 3.3V one 1 ESP8266 $5 The ESP-01 is the most readily available board (even available on Amazon) http://www.amazon.com/ESP8266-ESP-01-Serial-Wireless-Transceiver/dp/B00NF0FCW8 Lots Jumper wires M-M, M-F $1 Get a mix 1 2xAA battery holder $1 http://www.jameco.com/1/1/1260-bh-321-4a-r-2xaa-battery-holder-wires-mounting-tabs.html 1 NeoPixel Strip $10 http://www.adafruit.com/neopixel(Strips work well of any size eg: http://www.adafruit.com/products/1426 or else the circular rings http://www.adafruit.com/products/1643) In addition you’ll probably also want the usual hacker tools of hot glue, solder and band aids. Prerequisites There are a few things you’ll need to set up before you get started which will be specific to your system. This may take a little while (30-60 minutes) but the documentation linked below is exhaustive for Linux, Mac and Windows. Sign up for CloudPebble and get your developer account (it’s free) Download the Pebble SDK and install it following these directions - you need this to be able to install your JS application on your watch Install ESP Tool (for flashing your ESP8266 module) - this uses python so you may need a python environment if you’re using Windows. Install ESPlora (for uploading your application to the ESP8266 module) - this uses Java If you want some additional background on the ESP8266 and the development process, this excellent presentation / documentation by Andy Gelme is well worth a read. Once your environment is ready to go, grab the files that go along with this post with the following command: mkdir ~/watch-led git clone https://gist.github.com/ee6fadcd837a0f46be8d.git ~/watch-led && cd ~/watch-led Configuring the ESP8266 In order to configure the wireless module you will need to do the following things: Wire the module up Flash a binary file with the NodeMCU firmware onto the module Configure the application and upload that to the NodeMCU environment Test that the code is working Wire the module The ESP-01 module is relatively simple inasmuch as it only has 8 pins, however the use of double pin header means you can’t just plug it into a breadboard. You have a few options here, many people make a converter that uses 2x4 female header and converts this to a 1x8 strip of male header that can be plugged into a breadboard. Others (like me) make custom cables for different applications - most people just use jumper wires to join to a breadboard or other modules. You choose whatever works best for you but the wiring diagram is below. ESP-01 USB Serial adapter 1 RXD <-- TXD 2 VCC 3.3V (make sure in range 3-3.5V) 3 GPIO 0 Connect to ground when flashing firmware 4 RESET 3.3V resets (Don’t connect) 5 GPIO 2 Signal line on WS2812 strip 6 CH_PD 3.3V (enables the module) 7 GND Ground 8 TXD --> RXD With the LEDs, connect VCC to 3.3V and ground to ground as well. These LEDs are happy to run off as little as 3V. Install NodeMCU firmware Before you install the firmware you need to put the module into “flash mode”. You do this by joining GPIO 0 (zero) to ground as you can see in the photo below illustrated by the pink wire. If you don’t do this, you can’t put new firmware on your module and esptool will tell you it can’t connect. Next, issue the following commands from a terminal (assuming esptool.py is in your path): cd ~/watch-led esptool.py -p <<PORT>> write_flash 0x00000 nodemcu_dev_0x00000.bin 0x10000 nodemcu_dev_0x10000.bin Note that you need to change <<PORT>> to the path to your serial port (eg /dev/ttyUSB0) - esptool will now go through the process of erasing and then flashing the NodeMCU firmware onto the module. ESPtool is very simple and is fully automated Assuming you have installed esptool and wired your module correctly you should not expect to see any errors in this process. If you do, check your wiring and your setup and then head to the GitHub repo for more information. Once the firmware upload is complete, power down your module, remove the jumper on GPIO0 to ground and then power the module back up again. The module is now using 9600 baud rate - if you connect using screen or minicom etc you will be dropped into a lua interpreter. Here you can issue commands and get responses. screen /dev/ttyUSB0 9600 Change /dev/ttyUSB0 for your port The firmware has a full lua interpreter you can play with. Once you’ve had your fill playing with a lua interpreter, disconnect your serial connection. Configure application Now the module has NodeMCU on it, it’s possible to use the ESPlora IDE to talk to it, upload files to the module and write applications using Lua. Open the ESPlora IDE and connect to your module. Select your connection from the marked area and then hit “open”. From here, you can upload files to the module. Open up the five Lua files in the code folder you downloaded earlier. You should have setup.lua, server.lua, init.lua, config.lua and application.lua - these files can be edited directly from the code screen and then uploaded onto the ESP8266. Two files that are of interest are config and server. Set your wireless details Switch to the config.lua tab and update the line: module.SSID["SSID"] = "PASSWORD" And change SSID and PASSWORD to the details for your network. You’ll notice when you hit save (CTRL+S) that it will save and then upload the file to the module as well and confirm that is complete. In addition you can hit the Save to ESP button which will upload it as well. Go ahead now and upload config.lua, setup.lua, init.lua and application.lua. Configure your LEDs Open up the server.lua file and look for the line: local NO_PIXELS = 17 Change this to however many LEDs you have on your strip (the maximum limit is probably 50 or so before you have memory issues). There are other config options at the top of this file as well which should be fairly self explanatory. Save this file to the module as well. Test application is working As a check before we test if the module is working properly, hit the “File List” button and you will get the files that are uploaded. Check that all five are there and if they are hit the “Reset ESP” button. This will cause the module to restart. You’ll see a bootup sequence and after 10-15 seconds you’ll get a message showing the application has started, and the IP address of the module and the server. You can now issue commands from curl: curl --data "red=255&green=0&blue=255" http://<IP> Where <<IP>> is the IP address of the module. You should at this point have some LEDs turning a nice shade of magenta. If not, backtrack a few steps and make sure each component is working - the debug log will tell you if there’s any serious issues and you can use print() statements in server.lua to print out messages. Also try things like ping IP from a command line to make sure you can see the module on your network from your computer. Once you have your LED web service running it’s time to make it accessible from your Pebble watch. Building the Pebble watch app Building a Pebble watch app using a combination of the cloud pebble IDE and JavaScript makes things super easy. If you’ve done any sort of web development with JavaScript before you’ll find building apps this way really fast. Whilst the amount of access to the hardware is a bit reduced compared to C, you can make calls to external services, create interfaces and have access to the sensors so there’s plenty to play with. If you haven’t already, link your Pebble account to a developer account and login to CloudPebble.net - create a new project and select Pebble JS from the project type. Give your project a descriptive name. Once created you’ll be taken to a blank workspace. Click on the app.js file under source files. The app.js file is the main file you build your application in - you can add more but ours is really simple so we only need the one. Open up the app.js file in the repository folder on your computer and copy and paste the contents of it into the IDE. Before you do anything else, change the HOST IP address to the IP of the ESP8266 at the top of the file. The application code is broken down with further explanations below. var UI = require('ui'); var ajax = require('ajax'); Sets up the required UI library to draw things to the screen and gets an ajax library to make requests to web services. function colour_request(r, g, b) { var req = { url: HOST, method: 'post', data: {red:r, green:g, blue:b} }; ajax(req, function(data, status, request) { console.log(data); }, function(data, status, request) { console.log('The ajax request failed: ' + data + status + JSON.stringify(request)); } ); } This function makes the AJAX request to our ESP8266 service posting the data. This is set up so it can be called from any interaction point within the application as we need it, taking a value for red, green and blue. var colours = [ { title: "OFF", r: 0, g: 0, b: 0, }, { title: "RED", r: 255, g: 0, b: 0, }, { title: "GREEN", r: 0, g: 255, b: 0, }, { title: "BLUE", r: 0, g: 0, b: 255, }, { title: "YELLOW", r: 255, g: 255, b: 0, }, { title: "MAGENTA", r: 255, g: 0, b: 255, }, { title: "CYAN", r: 0, g: 255, b: 255, }, { title: "WHITE", r: 255, g: 255, b: 255, }, ]; Next we define an array of objects that represent the menu items in our application. The title’s will be used for the text on the menu items and then the r,g,b values will be used in order to provide the colour values for the colour_request function. var menu = new UI.Menu({ sections: [{ title: 'Choose LED colour', items: colours }] }); menu.on('select', function(e) { colour_request(e.item.r, e.item.g, e.item.b); }); menu.show(); The last part of the code creates the UI menu items and then defines an event handler for each menu item to call the colour_request function when it is selected and then finally menu.show() puts everything in the screen. Compile and deploy the application When it comes time to test or deploy your application you can either download a PBW file which can then be installed on the Pebble using command line tools or else you can use the emulator that is in the IDE. To use the emulator simple save and then hit the “Play” button to run your app within a watch emulator in the IDE. The great thing about the emulator is that you can test things like UI interactions and ensure configuration happens properly in a nice tight development loop. Unfortunately, because the emulator runs in Pebble’s network, it more than likely won’t have access to your ESP8266 which is sitting on your LAN. One way to fix this is simply create a route through your firewall and map it to your ESP8266. I won’t get into how to do this for your particular router. Simply change the HOST IP address in the app.js file to your public Internet Address and then configure a route to your internal LAN address. If you do this you can test directly from the emulator quite happily. The other option, and one you’ll need to do at some point anyway, is to install the application on your watch directly. Ensure your HOST IP address is updated in app.js and your phone is on the same network as the ESP8266. Now, from the CloudPebble IDE, click “Compilation” and then run a build. This will sit as pending for a little while and then if everything works it will go into the build log with the status “succeeded”. Download the PBW file by clicking on the “PBW” button and make note of the name of the file as it saves it to your downloads. Open up your phone and ensure developer mode is selected as instructed in the build tools set up at the top of this post. Get the IP address from the developer connection screen, as shown below. Open up a terminal and make sure you’ve activated the Pebble development environment. Export your phone’s IP address as shown from the developer connection tool in the Pebble app. export PEBBLE_PHONE=<<IP ADDRESS>> Now you should be able to do things like this: pebble ping And you’ll receive a notification on your watch. If that’s the case then simply install the application with: pebble install ~/Downloads/Watch_LED.pbw The Pebble will vibrate when it is working. Make it wireless Once you have everything working, the last step is to get onto battery power. This is easy - simply remove the USB-Serial adapter then plug in the Ground and VCC of your battery pack into the ground and VCC of your setup. At that point the module will power up - wait approximately 15 seconds and you should be able to ping it (and then control the LEDs from your watch). Going further The Pebble Watch and the ESP8266 are both extremely interesting devices. With a small amount of JS, the watch can be connected to any service that talks standard web protocols and with only $5 and a little bit of Lua it’s possible to make a wireless hardware service that the watch can interact with. Using this basic model you can make all sorts of internet connected devices. Here are some other things you could try now you’ve got this working: Extend the watch app to be able to talk to multiple ESP8266s scattered around your workspace. Change the event model so that the LEDs update on different notifications from your watch such as blue for a new tweet, red for a new email etc. Flip the service around - instead of controlling hardware, attach a sensor to the ESP8266 and then pass the data back to the watch periodically where it can display what’s happening. About the Author Andrew Fisher is a creator and destroyer of things that combine mobile web, ubicomp and lots of data. He is a sometime programmer, interaction researcher and CTO at JBA, a data consultancy in Melbourne, Australia. He can be found on Twitter @ajfisher.

0
0
10099

How-To Tutorials

article-image-getting-started-selenium-webdriver-and-python

Packt

23 Dec 2014

19 min read

GETTING STARTED WITH SELENIUM WEBDRIVER AND PYTHON

Packt

23 Dec 2014

19 min read

0
0
10098

Packt

25 Nov 2010

7 min read

Materials with Ogre 3D

Packt

25 Nov 2010

7 min read

OGRE 3D 1.7 Beginner's Guide Create real time 3D applications using OGRE 3D from scratch Easy-to-follow introduction to OGRE 3D Create exciting 3D applications using OGRE 3D Create your own scenes and monsters, play with the lights and shadows, and learn to use plugins Get challenged to be creative and make fun and addictive games on your own A hands-on do-it-yourself approach with over 100 examples Creating a white quad We will use this to create a sample quad that we can experiment with. Time for action – creating the quad We will start with an empty application and insert the code for our quad into the createScene() function: Begin with creating the manual object: Ogre::ManualObject* manual = mSceneMgr- >createManualObject("Quad"); manual->begin("BaseWhiteNoLighting", RenderOperation::OT_TRIANGLE_ LIST); Create four points for our quad: manual->position(5.0, 0.0, 0.0); manual->textureCoord(0,1); manual->position(-5.0, 10.0, 0.0); manual->textureCoord(1,0); manual->position(-5.0, 0.0, 0.0); manual->textureCoord(1,1); manual->position(5.0, 10.0, 0.0);manual->textureCoord(0,0); Use indices to describe the quad: manual->index(0); manual->index(1); manual->index(2); manual->index(0); manual->index(3); manual->index(1); Finish the manual object and convert it to a mesh: manual->end(); manual->convertToMesh("Quad"); Create an instance of the entity and attach it to the scene using a scene node: Ogre::Entity * ent = mSceneMgr->createEntity("Quad"); Ogre::SceneNode* node = mSceneMgr->getRootSceneNode()- >createChildSceneNode("Node1"); node->attachObject(ent); Compile and run the application. You should see a white quad. What just happened? We used our knowledge to create a quad and attach to it a material that simply renders everything in white. The next step is to create our own material. Creating our own material Always rendering everything in white isn't exactly exciting, so let's create our first material. Time for action – creating a material Now, we are going to create our own material using the white quad we created. Change the material name in the application from BaseWhiteNoLighting to MyMaterial1: manual->begin("MyMaterial1", RenderOperation::OT_TRIANGLE_LIST); Create a new file named Ogre3DBeginnersGuide.material in the mediamaterialsscripts folder of our Ogre3D SDK. Write the following code into the material file: material MyMaterial1 { technique { pass { texture_unit { texture leaf.png } } } } Compile and run the application. You should see a white quad with a plant drawn onto it. What just happened? We created our first material file. In Ogre 3D, materials can be defined in material files. To be able to find our material files, we need to put them in a directory listed in the resources.cfg, like the one we used. We also could give the path to the file directly in code using the ResourceManager. To use our material defined in the material file, we just had to use the name during the begin call of the manual object. The interesting part is the material file itself. Materials Each material starts with the keyword material, the name of the material, and then an open curly bracket. To end the material, use a closed curly bracket—this technique should be very familiar to you by now. Each material consists of one or more techniques; a technique describes a way to achieve the desired effect. Because there are a lot of different graphic cards with different capabilities, we can define several techniques and Ogre 3D goes from top to bottom and selects the first technique that is supported by the user's graphic cards. Inside a technique, we can have several passes. A pass is a single rendering of your geometry. For most of the materials we are going to create, we only need one pass. However, some more complex materials might need two or three passes, so Ogre 3D enables us to define several passes per technique. In this pass, we only define a texture unit. A texture unit defines one texture and its properties. This time the only property we define is the texture to be used. We use leaf.png as the image used for our texture. This texture comes with the SDK and is in a folder that gets indexed by resources.cfg, so we can use it without any work from our side. Have a go hero – creating another material Create a new material called MyMaterial2 that uses Water02.jpg as an image. Texture coordinates take two There are different strategies used when texture coordinates are outside the 0 to 1 range. Now, let's create some materials to see them in action. Time for action – preparing our quad We are going to use the quad from the previous example with the leaf texture material: Change the texture coordinates of the quad from range 0 to 1 to 0 to 2. The quad code should then look like this: manual->position(5.0, 0.0, 0.0); manual->textureCoord(0,2); manual->position(-5.0, 10.0, 0.0); manual->textureCoord(2,0); manual->position(-5.0, 0.0, 0.0); manual->textureCoord(2,2); manual->position(5.0, 10.0, 0.0); manual->textureCoord(0,0); Now compile and run the application. Just as before, we will see a quad with a leaf texture, but this time we will see the texture four times. What just happened? We simply changed our quad to have texture coordinates that range from zero to two. This means that Ogre 3D needs to use one of its strategies to render texture coordinates that are larger than 1. The default mode is wrap. This means each value over 1 is wrapped to be between zero and one. The following is a diagram showing this effect and how the texture coordinates are wrapped. Outside the corners, we see the original texture coordinates and inside the corners, we see the value after the wrapping. Also for better understanding, we see the four texture repetitions with their implicit texture coordinates. We have seen how our texture gets wrapped using the default texture wrapping mode. Our plant texture shows the effect pretty well, but it doesn't show the usefulness of this technique. Let's use another texture to see the benefits of the wrapping mode. Using the wrapping mode with another texture Time for action – adding a rock texture For this example, we are going to use another texture. Otherwise, we wouldn't see the effect of this texture mode: Create a new material similar to the previous one, except change the used texture to: terr_rock6.jpg: material MyMaterial3 { technique { pass { texture_unit { texture terr_rock6.jpg } } } } Change the used material from MyMaterial1 to MyMaterial3: manual->begin("MyMaterial3", RenderOperation::OT_TRIANGLE_LIST) Compile and run the application. You should see a quad covered in a rock texture. What just happened? This time, the quad seems like it's covered in one single texture. We don't see any obvious repetitions like we did with the plant texture. The reason for this is that, like we already know, the texture wrapping mode repeats. The texture was created in such a way that at the left end of the texture, the texture is started again with its right side and the same is true for the lower end. This kind of texture is called seamless. The texture we used was prepared so that the left and right side fit perfectly together. The same goes for the upper and lower part of the texture. If this wasn't the case, we would see instances where the texture is repeated.

0
0
10092

Packt

22 Aug 2013

13 min read

Getting Ready for RubyMotion

Packt

22 Aug 2013

13 min read

(For more resources related to this topic, see here.) How can I develop an iOS application? To develop iOS applications, there are various third-party frameworks available, apart from Apple libraries. If we broadly categorize the ways in which we can create iOS applications, we can divide them into three ways. Native apps using Objective-C This is the most standard way to build your application, by interacting with Apple APIs and writing apps in Objective-C. Applications made using native Apple APIs can use all possible device capabilities, and are relatively more reliable and high performing (however, the topic of performance is debatable based on the quality of the developer's code). Mobile web applications Mobile web applications are simple web applications extended for mobile web browsers, which can be created using standard web technologies such as HTML5. For example, if we browse through http://www.twitter.com in a mobile browser, it will be redirected to http://mobile.twitter.com, which renders its corresponding views for mobile devices. These applications are easy to create but the downside is that they have limited access to user data (for example, phonebook) and hardware (for example, camera). Hybrid applications These applications are somewhere in between mobile web apps and native applications. They are created using common web technologies such as HTML5 and JavaScript and have the ability to use device capabilities via their homegrown APIs. Some of the popular hybrid frameworks include Rhomobile and Phonegap. If we compare the speed of development and user experience, it can be summed up with the following diagrams: From the preceding diagrams we see that mobile web apps can be created very quickly but we have to compromise on user experience. While native apps using Objective-C have good user experience, they have a very steep initial learning curve for web developers. RubyMotion is good news for both users and developers. Users get an amazing experience of a native application and developers are able to develop applications rapidly in comparison to applications developed using Objective-C. Let's now learn about RubyMotion. What is RubyMotion? RubyMotion is a toolchain that allows developers to develop native iOS applications using the Ruby programming language. RubyMotion acts as a compiler that interacts with the iOS SDK(Software Development Kit). This gives us enormous power to make use of Apple libraries; therefore, once the application has compiled and loaded, the device has no idea whether it's an application made using Objective-C or RubyMotion. RubyMotion is a product of HipByte, founded by Laurent Sansonetti. While developing applications with RubyMotion using Ruby, you always have access to the iOS SDK classes. This gives you the benefit of even mixing Objective-C and Ruby code, as RubyMotion implements Ruby on top of the Objective-C runtime and iOS Foundation classes. This is how a typical RubyMotion application works. The code written in RubyMotion is fully compiled into machine code, so the application created by RubyMotion is as fast as the one created using Objective-C. Why RubyMotion? So far we have learned what RubyMotion is, but the question that comes to mind is, why should we use RubyMotion? There are many reasons why RubyMotion is a good choice for building robust iOS apps. The following sections detail a few that we think matter the most. If you are not an Objective-C fan For a newbie developer, Objective-C is an arduous affair. It's complicated to code; even for doing a simple thing, we have to write many lines of code. Though it is a powerful language and one of the best object-oriented ones available, it is time consuming and the learning curve is very steep. On the other hand, Ruby is more expressive, simple, and productive in comparison to Objective-C. Because of its simplicity, developers can shift their focus onto problem solving rather than spending time on trivial stuff, which is taken care by Ruby itself. In short, we can say RubyMotion allows us to use the power of Objective-C with the simplicity of Ruby. Ruby classes used in RubyMotion are inherited from Objective-C classes. If you are familiar with the concept of object-oriented programming, you can understand its power. This means we can directly use Apple iOS SDK classes from your RubyMotion code. It is not a bridge RubyMotion apps get direct access to iOS SDK APIs, which means the size of application and performance created using RubyMotion is comparable to the one created using Objective-C. It implements Ruby on top of the Objective-C runtime and iOS Foundation classes. RubyMotion uses a state-of-the-art static compiler based on Low Level Virtual Machine (LLVM), which converts the Ruby source code into a blazing fast machine code. The original source code is never present in the application bundle. A typical application weighs less than 1 MB, but the size can increase depending on the use case. Managed memory One of the key features of RubyMotion is that it takes care of memory management. Just like ARC (Automatic Reference Counting) with Xcode 4.4 and above, we don't have to take the pain of releasing the memory once an object is no longer used. RubyMotion does the magic and we don't need to think about it. It handles it on its own. Terminal-based workflow RubyMotion has a terminal-based workflow; from creation of the application to deployment, everything can be done through terminals. If you are used to working on terminals, you know it adds to speedier development. Easy debugging with REPL The terminal window where you run Rake also gives you the option to debug with REPL (Read Evaluate Print Loop), which lets you use Ruby expressions that are evaluated on the spot, and the results are reflected on the simulator while the application is still running. The ability to make live changes to the user interface and internal application data structures at runtime is extremely useful for testing and troubleshooting issues with the application, as this saves a lot of time and is much faster than a traditional edit-compile-run loop. It is extendable We can use RubyMotion salted gems easily by just adding them in the Rakefile. What are RubyMotion salted gems? We can't use all the gems that are available for Ruby right now, but there are a lot of gems specifically developed for RubyMotion. As the RubyMotion developer community expands, so will its gem bouquet, and this will make our application development rapid. Third-party Objective-C libraries can be easily used in a RubyMotion project. It supports CocoaPods, which is a dependency manager for Objective-C libraries, making this process a bit easier. Debugging and testing RubyMotion has a console-based inbuilt interactive debugger for troubleshooting the issues both on a simulator and on a device using GDB (GNU Debugger). GDB is extremely powerful on its own, and RubyMotion uses it for debugging the compiled Ruby code. Also, RubyMotion projects are fit for Test Driven Development (TDD). We can write a unit test for our code from the beginning. We can use Behavior Driven Development (BDD) with RubyMotion, which is integrated into every project. Pop quiz Q.1.How can we distinguish between the iOS application created by RubyMotion and the iOS application created by Objective-C? You can distinguish based on the user experience of the application. You can distinguish based on the performance of the application. You can't distinguish based on the user experience and performance of the application. Solution: If your answer was option 3, you were right. We can't distinguish between applications created by RubyMotion or Objective-C as the user experience and performance are similar. Q.2. How can we extend RubyMotion? We can use Objective-C libraries. We can use all Ruby gems. We can use RubyMotion-flavored gems. We can't use any other libraries. Solution: If your answer was option 1 and 3, you were right. Yes, we can use Objective-C libraries and also RubyMotion-flavored gems. RubyMotion installation – furnish your environment Now that we have got a good introduction to RubyMotion, let's set up our development environment; but before that let's run through some of the prerequisites. Prerequisites for RubyMotion You need a Mac OS: we can't develop iOS applications with RubyMotion on any other operating system; so we definitely need a Mac OS. OSX 10.6 or higher: RubyMotion requires a Mac running OSX 10.6 or higher. OSX 10.7 Lion is highly recommended. Ruby: the Ruby framework comes preinstalled with Mac OS X. If you have multiple versions of Ruby, we recommend that you use Ruby Version Manager (RVM). For more details, visit https://rvm.io/. Xcode: next we need to install Xcode, which includes the iOS SDK, developed by Apple and essential for developing iOS applications. It can be downloaded from the App Store for free. It also includes the iPhone/iPad simulator, which will be used for testing our application. Command Line Tools: after installing the Xcode toolchain, we need to install the command-line tools package, which is necessary for RubyMotion. To confirm that command-line tools is installed with your Xcode, open Xcode in your Applications folder, go to the Preferences window, and click on the Downloads tab. You should see the Command Line Tools package in this list. If it is not yet installed, make sure to click on the Install button. If you have an old version of Xcode, run the following command on the terminal: sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer This command will set up the default Xcode path. Installing RubyMotion RubyMotion installation is really simple and takes no time at all. RubyMotion is a commercial product that you need to purchase from www.rubymotion.com. Once purchased, you will receive your unique license key and installer. RubyMotion installation is a five-step procedure and is given here: Once you have received the package, run the RubyMotion installer as follows: Read and accept the EULA (End User License Agreement). Enter the license number you have received as shown in the following screenshot: Time for a short break—it will take a few minutes for RubyMotion to get downloaded and installed on your system. You can relax for some time. Yippee!! There is no step 5. And that's how quick it is to start working with RubyMotion. Update RubyMotion RubyMotion is a fast-moving framework and we need to upgrade it once there is a new release available. Upgrading RubyMotion is again really simple—with one command, you can easily upgrade it to the latest version. sudo motion update You need to be connected to the Internet for an upgrade to happen. If you want to work on an old version, you can downgrade using the following command: sudo motion update –force-version=1.2 But we recommend using the latest version. How do we check we've done everything correctly? Now that we have installed our RubyMotion copy, it's good practice to confirm which version we have installed; to do this, go to the terminal and run the following: motion –v This command outputs the RubyMotion version installed on your machine. If you get an error, you need to reinstall. Pick your own editor – you are not forced to use Xcode With RubyMotion, you are not forced to use Xcode. As every developer is more comfortable with a specific editor, you are open to choose what you like. However, we recommend the following editors for Ruby development: RubyMine Vim TextMate Sublime Emacs RubyMine now provides full support to a RubyMotion project. How to get help If you are facing some issues, the preferred way to get a solution is to discuss it at the RubyMotion Google group, (https://groups.google.com/forum/?fromgroups#!forum/rubymotion), where you can interact with fellow developers from the community and get a speedy resolution. Sometimes you might not get a precise response from the RubyMotion group. Not to worry, RubyMotion support is there to rescue you. If you have a feature request, an issue, or simply want to ask a question, you can log a support ticket—that too from the command line using the following command: $ motion support This will open up a new window in your browser. You can fill and submit the form with your query. Your RubyMotion license key, email address, and environment details will be added automatically. The RubyMotion community is growing at a very fast pace. In a short span of time, a lot of popular RubyMotion gems have been created by developers. FAQs We believe no question is silly. By now you will have many questions in your mind regarding RubyMotion. We have tried to answer a few of the most frequently asked questions (FAQs) related to topics covered so far in this section. Here are a few of them: Q1. Are the applications created by RubyMotion in keeping with Apple guidelines? Answer. Yes, RubyMotion strongly follows the review guidelines provided by Apple. Many applications created using RubyMotion are already available at the App Store. Q2. Will my RubyMotion application work on a Blackberry, Android, or Windows phone? Answer. No, applications created using RubyMotion are only for iOS devices; it is an alternative to programming in Objective-C. For a single-source multi-device application, we would recommend hybrid frameworks such as Rhomobile, Phonegap, and Titanium. For android development using Ruby, you can try Rubuto. Q3. Can I share an application with someone? Answer. Yes and no. With the Apple Developer Program membership, you can share your application only for testing purposes with a maximum of 100 devices, where each device has to be registered individually with Apple. Also, you cannot distribute your application on the App Store for testing. Once you have finished developing your application and are ready to ship, you can submit it to Apple for an App Store review. Q4. Can I use Ruby gems? Answer. Yes and no. No because we can't use normal Ruby gems, which you generally use in your Ruby on Rails projects; and yes because you can use gems that are specifically developed for RubyMotion, and there are already many such gems. Q5. Will my application work on iPad and iPod Touch? Answer. Absolutely, your application will work on any iOS devices, namely iPhone, iPad, and iPod Touch. Q6 Is Ruby allowed on the App Store? Answer. The App Store can't distinguish between applications made using Objective-C and those made using RubyMotion. So, no worries, our RubyMotion applications are fit for the App Store. Q7. Can I use third-party Objective-C libraries? Answer. Certainly. Third-party Objective-C libraries can be used in your project. RubyMotion provides integration with the CocoaPods dependency manager, which helps in reducing the hassle. You also can use C/C++ code provided that you wrap it into the Objective-C classes and methods. Q8. Is RubyMotion open source? Answer. RubyMotion as a toolchain is open source (available at GitHub). The closed source part is the Ruby runtime, which is, however, very similar to MacRuby runtime (which is open source). Summary Let's review all that we have learned so far. We first discussed the different ways to create iOS applications. Then we started with RubyMotion and discussed why to use it. And in the last section, we learned how to get started with RubyMotion and which editor fits with it. Now that we have our RubyMotion framework up and running, the next obvious task is to create our very first application, the most rudimentary Hello World application. Resources for Article : Further resources on this subject: Introducing RubyMotion and the Hello World app [Article] iPhone Applications Tune-Up: Design for Performance [Article] Introducing Xcode Tools for iPhone Development [Article]

0
0
10090

article-image-content-switching-using-citrix-security

Packt

10 Apr 2013

8 min read

Content Switching using Citrix Security

Packt

10 Apr 2013

8 min read

(For more resources related to this topic, see here.) Getting ready We will start with the packet flow of NetScaler and where content switching comes into play. The following diagram is self-explanatory (it is not the entire packet flow to the receiver's endpoint; the focus here is only to CS and LB): The content switching vserver can be used for HTTP/HTTPS/TCP and UDP protocols, and it can direct it only to another vserver, not to the backend service directly. The content switching vserver doesn't need an LB vserver to be bound to it for its status to be UP. Even with nothing bound to the CS vserver, the status would show UP (this comes in handy when you want to blackhole unwanted traffic).Hence, it is always recommended to check whether the load balancing vservers that are bound to the content switching vserver are up and running. If you want to avoid the preceding condition, the following CLI command will help you achieve it (by default, the value is disabled): root@ NetScaler> add cs vserver <name> <serviceType> (<IPAddress>) [-stateupdate ( ENABLED | DISABLED )] Content switching can be done based on the following client attributes: Mobile user/PC Images/videos Dynamic/static content Client with/without cookies Geographical locations. Per VLAN Similarly, server-side differentiations can also be made based on the following attributes: Server speed and capacity Source/destination port Source/destination IP SSL/HTTP Citrix also has an additional feature (starting from NetScaler version 9.3) that dynamically selects the load balancing feature based on any criteria or condition provided in the CS action/policy: >add cs action <name> -targetLBVserver <string-expression> >add cs policy <policyName> -rule <RULEValue> -action <actionName> The policy is then bound to the CS vserver CS vservers can be configured to process URLs in a case-sensitive manner. By default, this option is ON: >set cs vserver CSVserver -caseSensitive ON The load balancing vserver bound to the CS vserver need not have any IP address configured unless it is used in a different access as well. How to do it... We shall focus on a few case studies that we commonly come across, and that can be solved with the help of content switching: Case 1: Customer ABC accesses an online shopping portal and gets redirected to a secure connection at the payment gateway. For this scenario, an HTTP LB vserver is used and is bound to the CS vserver, which is on HTTPS: The configuration in the preceding screenshot shows that a CS policy as well as a responder policy is bound to the CS vserver named testVserver. The CS policy works on directing the traffic to the target LB vserver (if there are no CS policies bound at all, it goes to the default LB vserver; this default LB vserver should be configured on the CS Vserver). The responder policy, if bound to the CS vserver works on HTTP requests before matching any CS policy. The configuration is verified by using show cs vserver <vserver name>. A packet capture taken on NetScaler will clearly show the redirect from HTTP to HTTPS as <HTTP 302>. If there is any traffic that doesn't match any specific CS policies that are bound, then it uses the default policy. If there is no default policy, the user will get an error – HTTP 1.1 Service Unavailable error message. Case 2: The customer Star Networks has a single web application that contains two domains, namely www.starnetworks.com and www.starnetworks.com.edu and has a content switching setup, which works fine when accessing www.starnetworks.com, but throws an error when accessing www.starnetworks.com.edu. This happens because the peceding domains are not the same; they are different and the certificate that is bound to the CS vserver would be of type www.starnetworks.com only. To resolve this issue, we can bind multiple certificates to the CS vserver with the Server Name Indication (SNI) option enabled. The SNI option can be enabled in the SSL Parameters tab (this would pop up only if the SSL protocol is chosen while creating the vserver). The CLI command to enable SNI is as follows: >bind sslvserver star_cs_vserver -certkeyname -SNICert > bind sslvserver star_cs_vserver -certkeyname -SNICert For each domain added, NetScaler will establish a secure channel between itself and the client. With this solution, you can avoid configuring multiple CS vservers. Case 3: A Customer has a large pool of IP subnets that needs categorizing, and it would be a next to impossible task to configure that number of content switching policies; how does he go about deploying this scenario? The solution is as follows: A database file should be created that includes the IP address range and the domain: >shell #cd /var/ NetScaler/locdb # vi test.db Run the following command to apply the changes made to the database file: > add locationfile aol.db Bind the CS policy with an expression stating, for example, as follows: "CLIENT.IP.SRC.MATCHES_LOCATION ("star.*.*.*.*.*")"" How it works... The working of NetScaler in all three preceding scenarios is that it analyzes the incoming traffic directed to the CS VIP and parses through the bound CS policies, if any. If a match is found, it goes to the target LB vserver. If there are any other policies that are bound (for example, a responder policy or a rewrite policy), then the responder policy gets executed even before the CS policy is executed (since responder policies are usually applied to the HTTP requests).However, rewrite policies can be bound either at the CS or LB level, depending on whether the request or response needs to be modified. To recap what we have seen in the case studies mentioned before, the first case helps us to do a simple redirect from HTTP to HTTPS using a responder policy bound at the CS level. The second case shows us how multiple certificates with the SNI option are used to solve domain differences that would otherwise cause issues. The final case study shows us the basic but handy setting to map IP address ranges to target load balancing vservers. An important thing to note – there are scenarios where the vserver and the services that are bound to them may be different ports altogether (for example, HTTP LB VIP would be listening on port 80, but the services would be on port 8080). In such cases, the redirectPortRewrite feature should be enabled. There's more... This section concentrates on tidbits and troubleshooting techniques: Tips and troubleshooting We can start with checking the output of show cs and show lb vservers, to see if the services bound to them are up and running: root@ns > show cs vserver cs_star_vserver 1) cs_star_vserver (IP_ADDRESS_HERE:80) - HTTP Type: CONTENT State: UP Client Idle Timeout: 180 sec Down state flush: ENABLED Port Rewrite : DISABLED Default: lb_vserver Content Precedence: RULE Vserver IP and Port insertion: OFF Case Sensitivity: OFF If there are responder and rewrite policies, then we can check whether the number of hits on that policy are incrementing or not. Packet captures (using Wireshark) on the server and NetScaler. In some cases, the client would show us the packet flow in depth. The Down state flush feature of the NetScaler is useful for admins planning their downtimes in advance. This feature is enabled, by default, on the vserver and service level. When the feature is enabled, the connections that are already open and established will be terminated and the users will have to retry their connections again. The requests that are already being processed alone would be honored. When the feature is disabled, the open and established connections are honored, and no new connections will be accepted at this time. If enabled at the vserver level, and if the state of the vserver is DOWN, then the vserver will flush the client and server connections that are linked. Otherwise, it would terminate only the client facing connections. At the server level, if the service is marked as DOWN, then only the server facing connections would be flushed. There is another option on the Advanced tab of the CS/LB vserver to direct the excess traffic to a backup vserver. In cases where the backup server also overflows, there is an option to use the redirect URL, which is also found in the Advanced tab of the CS/LB vserver. Summary This article has explained the implementation of content switching using Citrix Security. Resources for Article : Further resources on this subject: Managing Citrix Policies [Article] Getting Started with XenApp 6 [Article] Getting Started with the Citrix Access Gateway Product Family [Article]

0
1
10088

article-image-index-item-sharding-and-projection-dynamodb

Packt

17 Sep 2014

13 min read

Index, Item Sharding, and Projection in DynamoDB

Packt

17 Sep 2014

13 min read

Understanding the secondary index and projections should go hand in hand because of the fact that a secondary index cannot be used efficiently without specifying projection. In this article by Uchit Vyas and Prabhakaran Kuppusamy, authors of DynamoDB Applied Design Patterns, we will take a look at local and global secondary indexes, and projection and its usage with indexes. (For more resources related to this topic, see here.) The use of projection in DynamoDB is pretty much similar to that of traditional databases. However, here are a few things to watch out for: Whenever a DynamoDB table is created, it is mandatory to create a primary key, which can be of a simple type (hash type), or it can be of a complex type (hash and range key). For the specified primary key, an index will be created (we call this index the primary index). Along with this primary key index, the user is allowed to create up to five secondary indexes per table. There are two kinds of secondary index. The first is a local secondary index (in which the hash key of the index must be the same as that of the table) and the second is the global secondary index (in which the hash key can be any field). In both of these secondary index types, the range key can be a field that the user needs to create an index for. Secondary indexes A quick question: while writing a query in any database, keeping the primary key field as part of the query (especially in the where condition) will return results much faster compared to the other way. Why? This is because of the fact that an index will be created automatically in most of the databases for the primary key field. This the case with DynamoDB also. This index is called the primary index of the table. There is no customization possible using the primary index, so the primary index is seldom discussed. In order to make retrieval faster, the frequently-retrieved attributes need to be made as part of the index. However, a DynamoDB table can have only one primary index and the index can have a maximum of two attributes (hash and range key). So for faster retrieval, the user should be given privileges to create user-defined indexes. This index, which is created by the user, is called the secondary index. Similar to the table key schema, the secondary index also has a key schema. Based on the key schema attributes, the secondary index can be either a local or global secondary index. Whenever a secondary index is created, during every item insertion, the items in the index will be rearranged. This rearrangement will happen for each item insertion into the table, provided the item contains both the index's hash and range key attribute. Projection Once we have an understanding of the secondary index, we are all set to learn about projection. While creating the secondary index, it is mandatory to specify the hash and range attributes based on which the index is created. Apart from these two attributes, if the query wants one or more attribute (assuming that none of these attributes are projected into the index), then DynamoDB will scan the entire table. This will consume a lot of throughput capacity and will have comparatively higher latency. The following is the table (with some data) that is used to store book information: Here are few more details about the table: The BookTitle attribute is the hash key of the table and local secondary index The Edition attribute is the range key of the table The PubDate attribute is the range key of the index (let's call this index IDX_PubDate) Local secondary index While creating the secondary index, the hash and range key of the table and index will be inserted into the index; optionally, the user can specify what other attributes need to be added. There are three kinds of projection possible in DynamoDB: KEYS_ONLY: Using this, the index consists of the hash and range key values of the table and index INCLUDE: Using this, the index consists of attributes in KEYS_ONLY plus other non-key attributes that we specify ALL: Using this, the index consists of all of the attributes from the source table The following code shows the creation of a local secondary index named Idx_PubDate with BookTitle as the hash key (which is a must in the case of a local secondary index), PubDate as the range key, and using the KEYS_ONLY projection: private static LocalSecondaryIndex getLocalSecondaryIndex() { ArrayList<KeySchemaElement> indexKeySchema = newArrayList<KeySchemaElement>(); indexKeySchema.add(new KeySchemaElement() .withAttributeName("BookTitle") .withKeyType(KeyType.HASH)); indexKeySchema.add(new KeySchemaElement() .withAttributeName("PubDate") .withKeyType(KeyType.RANGE)); LocalSecondaryIndex lsi = new LocalSecondaryIndex() .withIndexName("Idx_PubDate") .withKeySchema(indexKeySchema) .withProjection(new Projection() .withProjectionType("KEYS_ONLY")); return lsi; } The usage of the KEYS_ONLY index type will create the smallest possible index and the usage of ALL will create the biggest possible index. We will discuss the trade-offs between these index types a little later. Going back to our example, let us assume that we are using the KEYS_ONLY index type, so none of the attributes (other than the previous three key attributes) are projected into the index. So the index will look as follows: You may notice that the row order of the index is almost the same as that of the table order (except the second and third rows). Here, you can observe one point: the table records will be grouped primarily based on the hash key, and then the records that have the same hash key will be ordered based on the range key of the index. In the case of the index, even though the table's range key is part of the index attribute, it will not play any role in the ordering (only the index's hash and range keys will take part in the ordering). There is a negative in this approach. If the user is writing a query using this index to fetch BookTitle and Publisher with PubDate as 28-Dec-2008, then what happens? Will DynamoDB complain that the Publisher attribute is not projected into the index? The answer is no. The reason is that even though Publisher is not projected into the index, we can still retrieve it using the secondary index. However, retrieving a nonprojected attribute will scan the entire table. So if we are sure that certain attributes need to be fetched frequently, then we must project it into the index; otherwise, it will consume a large number of capacity units and retrieval will be much slower as well. One more question: if the user is writing a query using the local secondary index to fetch BookTitle and Publisher with PubDate as 28-Dec-2008, then what happens? Will DynamoDB complain that the PubDate attribute is not part of the primary key and hence queries are not allowed on nonprimary key attributes? The answer is no. It is a rule of thumb that we can write queries on the secondary index attributes. It is possible to include nonprimary key attributes as part of the query, but these attributes must at least be key attributes of the index. The following code shows how to add non-key attributes to the secondary index's projection: private static Projection getProjectionWithNonKeyAttr() { Projection projection = new Projection() .withProjectionType(ProjectionType.INCLUDE); ArrayList<String> nonKeyAttributes = new ArrayList<String>(); nonKeyAttributes.add("Language"); nonKeyAttributes.add("Author2"); projection.setNonKeyAttributes(nonKeyAttributes); return projection; } There is a slight limitation with the local secondary index. If we write a query on a non-key (both table and index) attribute, then internally DynamoDB might need to scan the entire table; this is inefficient. For example, consider a situation in which we need to retrieve the number of editions of the books in each and every language. Since both of the attributes are non-key, even if we create a local secondary index with either of the attributes (Edition and Language), the query will still result in a scan operation on the entire table. Global secondary index A problem arises here: is there any way in which we can create a secondary index using both the index keys that are different from the table's primary keys? The answer is the global secondary index. The following code shows how to create the global secondary index for this scenario: private static GlobalSecondaryIndex getGlobalSecondaryIndex() { GlobalSecondaryIndex gsi = new GlobalSecondaryIndex() .withIndexName("Idx_Pub_Edtn") .withProvisionedThroughput(new ProvisionedThroughput() .withReadCapacityUnits((long) 1) .withWriteCapacityUnits((long) 1)) .withProjection(newProjection().withProjectionType ("KEYS_ONLY")); ArrayList<KeySchemaElement> indexKeySchema1 = newArrayList<KeySchemaElement>(); indexKeySchema1.add(new KeySchemaElement() .withAttributeName("Language") .withKeyType(KeyType.HASH)); indexKeySchema1.add(new KeySchemaElement() .withAttributeName("Edition") .withKeyType(KeyType.RANGE)); gsi.setKeySchema(indexKeySchema1); return gsi; } While deciding the attributes to be projected into a global secondary index, there are trade-offs we must consider between provisioned throughput and storage costs. A few of these are listed as follows: If our application doesn't need to query a table so often and it performs frequent writes or updates against the data in the table, then we must consider projecting the KEYS_ONLY attributes. The global secondary index will be minimum size, but it will still be available when required for the query activity. The smaller the index, the cheaper the cost to store it and our write costs will be cheaper too. If we need to access only those few attributes that have the lowest possible latency, then we must project only those (lesser) attributes into a global secondary index. If we need to access almost all of the non-key attributes of the DynamoDB table on a frequent basis, we can project these attributes (even the entire table) into the global secondary index. This will give us maximum flexibility with the trade-off that our storage cost would increase, or even double if we project the entire table's attributes into the index. The additional storage costs to store the global secondary index might equalize the cost of performing frequent table scans. If our application will frequently retrieve some non-key attributes, we must consider projecting these non-key attributes into the global secondary index. Item sharding Sharding, also called horizontal partitioning, is a technique in which rows are distributed among the database servers to perform queries faster. In the case of sharding, a hash operation will be performed on the table rows (mostly on one of the columns) and, based on the hash operation output, the rows will be grouped and sent to the proper database server. Take a look at the following diagram: As shown in the previous diagram, if all the table data (only four rows and one column are shown for illustration purpose) is stored in a single database server, the read and write operations will become slower and the server that has the frequently accessed table data will work more compared to the server storing the table data that is not accessed frequently. The following diagram shows the advantage of sharding over a multitable, multiserver database environment: In the previous diagram, two tables (Tbl_Places and Tbl_Sports) are shown on the left-hand side with four sample rows of data (Austria.. means only the first column of the first item is illustrated and all other fields are represented by ..).We are going to perform a hash operation on the first column only. In DynamoDB, this hashing will be performed automatically. Once the hashing is done, similar hash rows will be saved automatically in different servers (if necessary) to satisfy the specified provisioned throughput capacity. Have you ever wondered about the importance of the hash type key while creating a table (which is mandatory)? Of course we all know the importance of the range key and what it does. It simply sorts items based on the range key value. So far, we might have been thinking that the range key is more important than the hash key. If you think that way, then you may be correct, provided we neither need our table to be provisioned faster nor do we need to create any partitions for our table. As long as the table data is smaller, the importance of the hash key will be realized only while writing a query operation. However, once the table grows, in order to satisfy the same provision throughput, DynamoDB needs to partition our table data based on this hash key (as shown in the previous diagram). This partitioning of table items based on the hash key attribute is called sharding. It means the partitions are created by splitting items and not attributes. This is the reason why a query that has the hash key (of table and index) retrieves items much faster. Since the number of partitions is managed automatically by DynamoDB, we cannot just hope for things to work fine. We also need to keep certain things in mind, for example, the hash key attribute should have more distinct values. To simplify, it is not advisable to put binary values (such as Yes or No, Present or Past or Future, and so on) into the hash key attributes, thereby restricting the number of partitions. If our hash key attribute has either Yes or No values in all the items, then DynamoDB can create only a maximum of two partitions; therefore, the specified provisioned throughput cannot be achieved. Just consider that we have created a table called Tbl_Sports with a provisioned throughput capacity of 10, and then we put 10 items into the table. Assuming that only a single partition is created, we are able to retrieve 10 items per second. After a point of time, we put 10 more items into the table. DynamoDB will create another partition (by hashing over the hash key), thereby satisfying the provisioned throughput capacity. There is a formula taken from the AWS site: Total provisioned throughput/partitions = throughput per partition OR No. of partitions = Total provisioned throughput/throughput per partition In order to satisfy throughput capacity, the other parameters will be automatically managed by DynamoDB. Summary In this article, we saw what the local and global secondary indexes are. We walked through projection and its usage with indexes. Resources for Article: Further resources on this subject: Comparative Study of NoSQL Products [Article] Ruby with MongoDB for Web Development [Article] Amazon DynamoDB - Modelling relationships, Error handling [Article]

0
0
10079

article-image-chain-responsibility-pattern

Packt

05 Feb 2015

12 min read

The Chain of Responsibility Pattern

Packt

05 Feb 2015

12 min read

In this article by Sakis Kasampalis, author of the book Mastering Python Design Patterns, we will see a detailed description of the Chain of Responsibility design pattern with the help of a real-life example as well as a software example. Also, its use cases and implementation are discussed. (For more resources related to this topic, see here.) When developing an application, most of the time we know which method should satisfy a particular request in advance. However, this is not always the case. For example, we can think of any broadcast computer network, such as the original Ethernet implementation [j.mp/wikishared]. In broadcast computer networks, all requests are sent to all nodes (broadcast domains are excluded for simplicity), but only the nodes that are interested in a sent request process it. All computers that participate in a broadcast network are connected to each other using a common medium such as the cable that connects the three nodes in the following figure: If a node is not interested or does not know how to handle a request, it can perform the following actions: Ignore the request and do nothing Forward the request to the next node The way in which the node reacts to a request is an implementation detail. However, we can use the analogy of a broadcast computer network to understand what the chain of responsibility pattern is all about. The Chain of Responsibility pattern is used when we want to give a chance to multiple objects to satisfy a single request, or when we don't know which object (from a chain of objects) should process a specific request in advance. The principle is the same as the following: There is a chain (linked list, tree, or any other convenient data structure) of objects. We start by sending a request to the first object in the chain. The object decides whether it should satisfy the request or not. The object forwards the request to the next object. This procedure is repeated until we reach the end of the chain. At the application level, instead of talking about cables and network nodes, we can focus on objects and the flow of a request. The following figure, courtesy of a title="Scala for Machine Learning" www.sourcemaking.com [j.mp/smchain], shows how the client code sends a request to all processing elements (also known as nodes or handlers) of an application: Note that the client code only knows about the first processing element, instead of having references to all of them, and each processing element only knows about its immediate next neighbor (called the successor), not about every other processing element. This is usually a one-way relationship, which in programming terms means a singly linked list in contrast to a doubly linked list; a singly linked list does not allow navigation in both ways, while a doubly linked list allows that. This chain organization is used for a good reason. It achieves decoupling between the sender (client) and the receivers (processing elements) [GOF95, page 254]. A real-life example ATMs and, in general, any kind of machine that accepts/returns banknotes or coins (for example, a snack vending machine) use the chain of responsibility pattern. There is always a single slot for all banknotes, as shown in the following figure, courtesy of www.sourcemaking.com: When a banknote is dropped, it is routed to the appropriate receptacle. When it is returned, it is taken from the appropriate receptacle [j.mp/smchain], [j.mp/c2chain]. We can think of the single slot as the shared communication medium and the different receptacles as the processing elements. The result contains cash from one or more receptacles. For example, in the preceding figure, we see what happens when we request $175 from the ATM. A software example I tried to find some good examples of Python applications that use the Chain of Responsibility pattern but I couldn't, most likely because Python programmers don't use this name. So, my apologies, but I will use other programming languages as a reference. The servlet filters of Java are pieces of code that are executed before an HTTP request arrives at a target. When using servlet filters, there is a chain of filters. Each filter performs a different action (user authentication, logging, data compression, and so forth), and either forwards the request to the next filter until the chain is exhausted, or it breaks the flow if there is an error (for example, the authentication failed three consecutive times) [j.mp/soservl]. Apple's Cocoa and Cocoa Touch frameworks use Chain of Responsibility to handle events. When a view receives an event that it doesn't know how to handle, it forwards the event to its superview. This goes on until a view is capable of handling the event or the chain of views is exhausted [j.mp/chaincocoa]. Use cases By using the Chain of Responsibility pattern, we give a chance to a number of different objects to satisfy a specific request. This is useful when we don't know which object should satisfy a request in advance. An example is a purchase system. In purchase systems, there are many approval authorities. One approval authority might be able to approve orders up to a certain value, let's say $100. If the order is more than $100, the order is sent to the next approval authority in the chain that can approve orders up to $200, and so forth. Another case where Chain of Responsibility is useful is when we know that more than one object might need to process a single request. This is what happens in an event-based programming. A single event such as a left mouse click can be caught by more than one listener. It is important to note that the Chain of Responsibility pattern is not very useful if all the requests can be taken care of by a single processing element, unless we really don't know which element that is. The value of this pattern is the decoupling that it offers. Instead of having a many-to-many relationship between a client and all processing elements (and the same is true regarding the relationship between a processing element and all other processing elements), a client only needs to know how to communicate with the start (head) of the chain. The following figure demonstrates the difference between tight and loose coupling. The idea behind loosely coupled systems is to simplify maintenance and make it easier for us to understand how they function [j.mp/loosecoup]: Implementation There are many ways to implement Chain of Responsibility in Python, but my favorite implementation is the one by Vespe Savikko [j.mp/savviko]. Vespe's implementation uses dynamic dispatching in a Pythonic style to handle requests [j.mp/ddispatch]. Let's implement a simple event-based system using Vespe's implementation as a guide. The following is the UML class diagram of the system: The Event class describes an event. We'll keep it simple, so in our case an event has only name: class Event: def __init__(self, name): self.name = name def __str__(self): return self.name The Widget class is the core class of the application. The parent aggregation shown in the UML diagram indicates that each widget can have a reference to a parent object, which by convention, we assume is a Widget instance. Note, however, that according to the rules of inheritance, an instance of any of the subclasses of Widget (for example, an instance of MsgText) is also an instance of Widget. The default value of parent is None: class Widget: def __init__(self, parent=None): self.parent = parent The handle() method uses dynamic dispatching through hasattr() and getattr() to decide who is the handler of a specific request (event). If the widget that is asked to handle an event does not support it, there are two fallback mechanisms. If the widget has parent, then the handle() method of parent is executed. If the widget has no parent but a handle_default() method, handle_default() is executed: def handle(self, event): handler = 'handle_{}'.format(event) if hasattr(self, handler): method = getattr(self, handler) method(event) elif self.parent: self.parent.handle(event) elif hasattr(self, 'handle_default'): self.handle_default(event) At this point, you might have realized why the Widget and Event classes are only associated (no aggregation or composition relationships) in the UML class diagram. The association is used to show that the Widget class "knows" about the Event class but does not have any strict references to it, since an event needs to be passed only as a parameter to handle(). MainWIndow, MsgText, and SendDialog are all widgets with different behaviors. Not all these three widgets are expected to be able to handle the same events, and even if they can handle the same event, they might behave differently. MainWIndow can handle only the close and default events: class MainWindow(Widget): def handle_close(self, event): print('MainWindow: {}'.format(event)) def handle_default(self, event): print('MainWindow Default: {}'.format(event)) SendDialog can handle only the paint event: class SendDialog(Widget): def handle_paint(self, event): print('SendDialog: {}'.format(event)) Finally, MsgText can handle only the down event: class MsgText(Widget): def handle_down(self, event): print('MsgText: {}'.format(event)) The main() function shows how we can create a few widgets and events, and how the widgets react to those events. All events are sent to all the widgets. Note the parent relationship of each widget. The sd object (an instance of SendDialog) has as its parent the mw object (an instance of MainWindow). However, not all objects need to have a parent that is an instance of MainWindow. For example, the msg object (an instance of MsgText) has the sd object as a parent: def main(): mw = MainWindow() sd = SendDialog(mw) msg = MsgText(sd) for e in ('down', 'paint', 'unhandled', 'close'): evt = Event(e) print('nSending event -{}- to MainWindow'.format(evt)) mw.handle(evt) print('Sending event -{}- to SendDialog'.format(evt)) sd.handle(evt) print('Sending event -{}- to MsgText'.format(evt)) msg.handle(evt) The following is the full code of the example (chain.py): class Event: def __init__(self, name): self.name = name def __str__(self): return self.name class Widget: def __init__(self, parent=None): self.parent = parent def handle(self, event): handler = 'handle_{}'.format(event) if hasattr(self, handler): method = getattr(self, handler) method(event) elif self.parent: self.parent.handle(event) elif hasattr(self, 'handle_default'): self.handle_default(event) class MainWindow(Widget): def handle_close(self, event): print('MainWindow: {}'.format(event)) def handle_default(self, event): print('MainWindow Default: {}'.format(event)) class SendDialog(Widget): def handle_paint(self, event): print('SendDialog: {}'.format(event)) class MsgText(Widget): def handle_down(self, event): print('MsgText: {}'.format(event)) def main(): mw = MainWindow() sd = SendDialog(mw) msg = MsgText(sd) for e in ('down', 'paint', 'unhandled', 'close'): evt = Event(e) print('nSending event -{}- to MainWindow'.format(evt)) mw.handle(evt) print('Sending event -{}- to SendDialog'.format(evt)) sd.handle(evt) print('Sending event -{}- to MsgText'.format(evt)) msg.handle(evt) if __name__ == '__main__': main() Executing chain.py gives us the following results: >>> python3 chain.py Sending event -down- to MainWindow MainWindow Default: down Sending event -down- to SendDialog MainWindow Default: down Sending event -down- to MsgText MsgText: down Sending event -paint- to MainWindow MainWindow Default: paint Sending event -paint- to SendDialog SendDialog: paint Sending event -paint- to MsgText SendDialog: paint Sending event -unhandled- to MainWindow MainWindow Default: unhandled Sending event -unhandled- to SendDialog MainWindow Default: unhandled Sending event -unhandled- to MsgText MainWindow Default: unhandled Sending event -close- to MainWindow MainWindow: close Sending event -close- to SendDialog MainWindow: close Sending event -close- to MsgText MainWindow: close There are some interesting things that we can see in the output. For instance, sending a down event to MainWindow ends up being handled by the default MainWindow handler. Another nice case is that although a close event cannot be handled directly by SendDialog and MsgText, all the close events end up being handled properly by MainWindow. That's the beauty of using the parent relationship as a fallback mechanism. If you want to spend some more creative time on the event example, you can replace the dumb print statements and add some actual behavior to the listed events. Of course, you are not limited to the listed events. Just add your favorite event and make it do something useful! Another exercise is to add a MsgText instance during runtime that has MainWindow as the parent. Is this hard? Do the same for an event (add a new event to an existing widget). Which is harder? Summary In this article, we covered the Chain of Responsibility design pattern. This pattern is useful to model requests / handle events when the number and type of handlers isn't known in advance. Examples of systems that fit well with Chain of Responsibility are event-based systems, purchase systems, and shipping systems. In the Chain Of Responsibility pattern, the sender has direct access to the first node of a chain. If the request cannot be satisfied by the first node, it forwards to the next node. This continues until either the request is satisfied by a node or the whole chain is traversed. This design is used to achieve loose coupling between the sender and the receiver(s). ATMs are an example of Chain Of Responsibility. The single slot that is used for all banknotes can be considered the head of the chain. From here, depending on the transaction, one or more receptacles is used to process the transaction. The receptacles can be considered the processing elements of the chain. Java's servlet filters use the Chain of Responsibility pattern to perform different actions (for example, compression and authentication) on an HTTP request. Apple's Cocoa frameworks use the same pattern to handle events such as button presses and finger gestures. Resources for Article: Further resources on this subject: Exploring Model View Controller [Article] Analyzing a Complex Dataset [Article] Automating Your System Administration and Deployment Tasks Over SSH [Article]

0
0
10075

article-image-making-entity-multiplayer-ready

Packt

07 Apr 2014

8 min read

Making an entity multiplayer-ready

Packt

07 Apr 2014

8 min read

(For more resources related to this topic, see here.) Understanding the dataflow of Lua entities in a multiplayer environment When using your own Lua entities in a multiplayer environment, you need to make sure everything your entity does on one of the clients is also triggered on all other clients. Let's take a light switch as an example. If one of the players turned on the light switch, the switch should also be flipped on all other clients. Each client connected to the game has an instance of that light switch in their level. The CryENGINE network implementation already handles all the work involved in linking these individual instances together using network entity IDs. Each light switch can contact its own instances on all connected clients and call its functions over the network. All you need to do is use the functionality that is already there. One way of implementing the light switch functionality is to turn on the switch in the entity as soon as the OnUsed() event is triggered and then send a message to all other clients in the network to also turn on their lights. This might work for something as simple as a switch, but can soon get messy when the entity becomes more complex. Ping times and message orders can lead to inconsistencies if two players try to flip the light switch at the same time. The representation of the process would look like the following diagram: Not so good – the light switch entity could trigger its own switch on all network instances of itself Doing it this way, with the clients notifying each other, can cause many problems. In a more stable solution, these kinds of events are usually run through the server. The server entity—let's call it the master entity—determines the state of the entities across the network at all times and distributes the entities throughout the network. This could be visualized as shown in the following diagram: Better – the light switch entity calls the server that will distribute the event to all clients In the light switch scenario mentioned earlier, the light switch entity would send an event to the server light switch entity first. Then, the server entity would call each light switch entity, including the original sender, to turn on their lights. It is important to understand that the entity that received the event originally does nothing else but inform the server about the event. The actual light is not turned on until the server calls back to all entities with the request to do so. The aforementioned dataflow works in single player as well, as CryENGINE will just pretend that the local machine is both the client and the server. This way, you will not have to make adjustments or add extra code to your entity to check whether it is single player or multiplayer. In a multiplayer environment with a server and multiple clients, it is important to set the script up so that it acts properly and the correct functions are called on either the client or the server. The first step to achieve this is to add a client and server table to the entity script using the following code: Client = {}, Server = {}, With this addition, our script table looks like the following code snippet: Testy = {Properties={ fileModel = "",Physics = { bRigidBody=1, = 1, Density = -1, Mass = -1, }, Client = {}, Server = {}, Editor={ Icon="User.bmp", }, } Now, we can go ahead and modify the functions so that they work properly in multiplayer. We do this by adding the Client and Server subtables to our script. This way, the network system will be able to identify the Client/Server functions on the entity. The Client/Server functions The Client/Server functions are defined within your entity script by using the respective subtables that we previously defined in the entity table. Let's update our script and add a simple function that outputs a debug text into the console on each client. In order for everything to work properly, we first need to update our OnInit() function and make sure it gets called on the server properly. Simply add a server subtable to the function so that it looks like the following code snippet: functionTesty.Server:OnInit() self:OnReset(); end; This way, our OnReset() function will still be called properly. Now, we can add a new function that outputs a debug text for us. Let's keep it simple and just make it output a console log using the CryENGINE Log function, as shown in the following code snippet: functionTesty.Client:PrintLogOutput(text) Log(text); end; This function will simply print some text into the CryENGINE console. Of course, you can add more sophisticated code at this point to be executed on the client. Please also note the Client subtable in the function definition that tells the engine that this is a client function. In the next step, we have to add a way to trigger this function so that we can test the behavior properly. There are many ways of doing this, but to keep things simple, we will simply use the OnHit() callback function that will be automatically triggered when the entity is hit by something; for example, a bullet. This way, we can test our script easily by just shooting at our entity. The OnHit() callback function is quite simple. All it needs to do in our case is to call our PrintLogOutput function, or rather request the server to call it. For this purpose, we add another function to be called on the server that calls our PrintLogOutput() function. Again, please note that we are using the Client subtable of the entity to catch the hit that happens on the client. Our two new functions should look as shown in the following code snippet: functionTesty.Client:OnHit(user) self.server:SvRequestLogOutput("My Text!"); end functionTesty.Server:SvRequestLogOutput(text) self.allClients:PrintLogOutput(text); end We now have two new functions: one is a client function calling a server function and the other one is a server function calling the actual function on all the clients. The Remote Method Invocation definitions As a last step, before we are finished, we need to expose our entity and its functions to the network. We can do this by adding a table within the root of our entity script that defines the necessary Remote Method Invocation (RMI). The Net.Expose table will expose our entity and its functions to the network so that they can be called remotely, as shown in the following code snippet: Net.Expose { Class = Testy, ClientMethods = { PrintLogOutput = { RELIABLE_UNORDERED, POST_ATTACH, STRING }, }, ServerMethods = { SvRequestLogOutput = { RELIABLE_UNORDERED, POST_ATTACH, STRING}, }, ServerProperties = { }, }; Each RMI is defined by providing a function name, a set of RMI flags, and additional parameters. The first RMI flag is an order flag and defines the order of the network packets. You can choose between the following options: UNRELIABLE_ORDERED RELIABLE_ORDERED RELIABLE_UNORDERED These flags tell the engine whether the order of the packets is important or not. The attachment flag will define at what time the RMI is attached during the serialization process of the network. This parameter can be either of the following flags: PREATTACH: This flag attaches the RMI before game data serialization. POSTATTACH: This flag attaches the RMI after game data serialization. NOATTACH: This flag is used when it is not important if the RMI is attached before or after the game data serialization. FAST: This flag performs an immediate transfer of the RMI without waiting for a frame update. This flag is very CPU intensive and should be avoided if possible. The Net.Expose table we just added defines which functions will be exposed on the client and the server and will give us access to the following three subtables: allClients otherClients server With these functions, we can now call functions either on the server or the clients. You can use the allClients subtable to call a function on all clients or the otherClients subtable to call it on all clients except the own client. At this point, the entity table of our script should look as follows: Testy = { Properties={ fileModel = "", Physics = { bRigidBody=1, bRigidBodyActive = 1, Density = -1, Mass = -1, }, Client = {}, Server = {}, Editor={ Icon="User.bmp", ShowBounds = 1, }, } Net.Expose { Class = Testy, ClientMethods = { PrintLogOutput = { RELIABLE_UNORDERED, POST_ATTACH, STRING }, }, ServerMethods = { SvRequestLogOutput = { RELIABLE_UNORDERED, POST_ATTACH, STRING}, }, ServerProperties = { }, }; This defines our entity and its network exposure. With our latest updates, the rest of our script with all its functions should look as follows: functionTesty.Server:OnInit() self:OnReset(); end; functionTesty:OnReset() local props=self.Properties; if(not EmptyString(props.fileModel))then self:LoadObject(0,props.fileModel); end; EntityCommon.PhysicalizeRigid(self,0,props.Physics,0); self:DrawSlot(0, 1); end; functionTesty:OnPropertyChange() self:OnReset(); end; functionTesty.Client:PrintLogOutput(text) Log(text); end; functionTesty.Client:OnHit(user) self.server:SvRequestLogOutput("My Text!"); end functionTesty.Server:SvRequestLogOutput(text) self.allClients:PrintLogOutput(text); end With these functions added to our entity, everything should be ready to go and you can test the behavior in game mode. When the entity is being shot at, the OnHit() function will request the log output to be printed from the server. The server calls the actual function on all clients. Summary In this article we learned about making our entity ready for a multiplayer environment by understanding the dataflow of Lua entities, understanding the Client/Server functions, and by exposing our entities to the network using the Remote Method Invocation definitions. Resources for Article: Further resources on this subject: CryENGINE 3: Terrain Sculpting [Article] CryENGINE 3: Breaking Ground with Sandbox [Article] CryENGINE 3: Fun Physics [Article]

0
0
10074

article-image-developing-javafx-application-ios

Packt

08 Jul 2015

10 min read

Developing a JavaFX Application for iOS

Packt

08 Jul 2015

10 min read

In this article by Mohamed Taman, authors of the book JavaFX Essentials, we will learn how to develop a JavaFX, Apple has a great market share in the mobile and PC/Laptop world, with many different devices, from mobile phones such as the iPhone to musical devices such as the iPod and tablets such as the iPad. (For more resources related to this topic, see here.) It has a rapidly growing application market, called the Apple Store, serving its community, where the number of available apps increases daily. Mobile application developers should be ready for such a market. Mobile application developers targeting both iOS and Android face many challenges. By just comparing the native development environments of these two platforms, you will find that they differ substantially. iOS development, according to Apple, is based on the Xcode IDE (https://developer.apple.com/xcode/) and its programming languages. Traditionally, it was Objetive-C and, in June 2014, Apple introduced Swift (https://developer.apple.com/swift/); on the other hand, Android development, as defined by Google, is based on the Intellij IDEA IDE and the Java programming language. Not many developers are proficient in both environments. In addition, these differences rule out any code reuse between the platforms. JavaFX 8 is filling the gap for reusable code between the platforms, as we will see in this article, by sharing the same application in both platforms. Here are some skills that you will have gained by the end of this article: Installing and configuring iOS environment tools and software Creating iOS JavaFX 8 applications Simulating and debugging JavaFX mobile applications Packaging and deploying applications on iOS mobile devices Using RoboVM to run JavaFX on iOS RoboVM is the bridge from Java to Objetive-C. Using this, it becomes easy to develop JavaFX 8 applications that are to be run on iOS-based devices, as the ultimate goal of the RoboVM project is to solve this problem without compromising on developer experience or app user experience. As we saw in the article about Android, using JavaFXPorts to generate APKs was a relatively easy task due to the fact that Android is based on Java and the Dalvik VM. On the contrary, iOS doesn't have a VM for Java, and it doesn't allow dynamic loading of native libraries. Another approach is required. The RoboVM open source project tries to close the gap for Java developers by creating a bridge between Java and Objective-C using an ahead-of-time compiler that translates Java bytecode into native ARM or x86 machine code. Features Let's go through the RoboVM features: Brings Java and other JVM languages, such as Scala, Clojure, and Groovy, to iOS-based devices Translates Java bytecode into machine code ahead of time for fast execution directly on the CPU without any overhead The main target is iOS and the ARM processor (32- and 64-bit), but there is also support for Mac OS X and Linux running on x86 CPUs (both 32- and 64-bit) Does not impose any restrictions on the Java platform features accessible to the developer, such as reflection or file I/O Supports standard JAR files that let the developer reuse the vast ecosystem of third-party Java libraries Provides access to the full native iOS APIs through a Java-to-Objective-C bridge, enabling the development of apps with truly native UIs and with full hardware access Integrates with the most popular tools such as NetBeans, Eclipse, Intellij IDEA, Maven, and Gradle App Store ready, with hundreds of apps already in the store Limitations Mainly due to the restrictions of the iOS platform, there are a few limitations when using RoboVM: Loading custom bytecode at runtime is not supported. All class files comprising the app have to be available at compile time on the developer machine. The Java Native Interface technology as used on the desktop or on servers usually loads native code from dynamic libraries, but Apple does not permit custom dynamic libraries to be shipped with an iOS app. RoboVM supports a variant of JNI based on static libraries. Another big limitation is that RoboVM is an alpha-state project under development and not yet recommended for production usage. RoboVM has full support for reflection. How it works Since February 2015 there has been an agreement between the companies behind RoboVM and JavaFXPorts, and now a single plugin called jfxmobile-plugin allows us to build applications for three platforms—desktop, Android, and iOS—from the same codebase. The JavaFXMobile plugin adds a number of tasks to your Java application that allow you to create .ipa packages that can be submitted to the Apple Store. Android mostly uses Java as the main development language, so it is easy to merge your JavaFX 8 code with it. On iOS, the situation is internally totally different—but with similar Gradle commands. The plugin will download and install the RoboVM compiler, and it will use RoboVM compiler commands to create an iOS application in build/javafxports/ios. Getting started In this section, you will learn how to install the RoboVM compiler using the JavaFXMobile plugin, and make sure the tool chain works correctly by reusing the same application, Phone Dial version 1.0. Prerequisites In order to use the RoboVM compiler to build iOS apps, the following tools are required: Gradle 2.4 or higher is required to build applications with the jfxmobile plugin. A Mac running Mac OS X 10.9 or later. Xcode 6.x from the Mac App Store (https://itunes.apple.com/us/app/xcode/id497799835?mt=12). The first time you install Xcode, and every time you update to a new version, you have to open it once to agree to the Xcode terms. Preparing a project for iOS We will reuse the project we developed before, for the Android platform, since there is no difference in code, project structure, or Gradle build script when targeting iOS. They share the same properties and features, but with different Gradle commands that serve iOS development, and a minor change in the Gradle build script for the RoboVM compiler. Therefore, we will see the power of WORA Write Once, Run Everywhere with the same application. Project structure Based on the same project structure from the Android, the project structure for our iOS app should be as shown in the following figure: The application We are going to reuse the same application from the Phone DialPad version 2.0 JavaFX 8 application: As you can see, reusing the same codebase is a very powerful and useful feature, especially when you are developing to target many mobile platforms such as iOS and Android at the same time. Interoperability with low-level iOS APIs To have the same functionality of natively calling the default iOS phone dialer from our application as we did with Android, we have to provide the native solution for iOS as the following IosPlatform implementation: import org.robovm.apple.foundation.NSURL; import org.robovm.apple.uikit.UIApplication; import packt.taman.jfx8.ch4.Platform; public class IosPlatform implements Platform { @Override public void callNumber(String number) { if (!number.equals("")) { NSURL nsURL = new NSURL("telprompt://" + number); UIApplication.getSharedApplication().openURL(nsURL); } } } Gradle build files We will use the Gradle build script file, but with a minor change by adding the following lines to the end of the script: jfxmobile { ios { forceLinkClasses = [ 'packt.taman.jfx8.ch4.**.*' ] } android { manifest = 'lib/android/AndroidManifest.xml' } } All the work involved in installing and using robovm compilers is done by the jfxmobile plugin. The purpose of those lines is to give the RoboVM compiler the location of the main application class that has to be loaded at runtime is, as it is not visible by default to the compiler. The forceLinkClasses property ensures that those classes are linked in during RoboVM compilation. Building the application After we have added the necessary configuration set to build the script for iOS, its time to build the application in order to deploy it to different iOS target devices. To do so, we have to run the following command: $ gradle build We should have the following output: BUILD SUCCESSFUL Total time: 44.74 secs We have built our application successfully; next, we need to generate the .ipa and, in the case of production, you have to test it by deploying it to as many iOS versions as you can. Generating the iOS .ipa package file In order to generate the final .ipa iOS package for our JavaFX 8 application, which is necessary for the final distribution to any device or the AppStore, you have to run the following gradle command: gradle ios This will generate the .ipa file in the directory build/javafxports/ios. Deploying the application During development, we need to check our application GUI and final application prototype on iOS simulators and measure the application performance and functionality on different devices. These procedures are very useful, especially for testers. Let's see how it is a very easy task to run our application on either simulators or on real devices. Deploying to a simulator On a simulator, you can simply run the following command to check if your application is running: $ gradle launchIPhoneSimulator This command will package and launch the application in an iPhone simulator as shown in the following screenshot: DialPad2 JavaFX 8 application running on the iOS 8.3/iPhone 4s simulator This command will launch the application in an iPad simulator: $ gradle launchIPadSimulator Deploying to an Apple device In order to package a JavaFX 8 application and deploy it to an Apple device, simply run the following command: $ gradle launchIOSDevice This command will launch the JavaFX 8 application in the device that is connected to your desktop/laptop. Then, once the application is launched on your device, type in any number and then tap Call. The iPhone will ask for permission to dial using the default mobile dialer; tap on Ok. The default mobile dialer will be launched and will the number as shown in the following figure: To be able to test and deploy your apps on your devices, you will need an active subscription with the Apple Developer Program. Visit the Apple Developer Portal, https://developer.apple.com/register/index.action, to sign up. You will also need to provision your device for development. You can find information on device provisioning in the Apple Developer Portal, or follow this guide: http://www.bignerdranch.com/we-teach/how-to-prepare/ios-device-provisioning/. Summary This article gave us a very good understanding of how JavaFX-based applications can be developed and customized using RoboVM for iOS to make it possible to run your applications on Apple platforms. You learned about RoboVM features and limitations, and how it works; you also gained skills that you can use for developing. You then learned how to install the required software and tools for iOS development and how to enable Xcode along with the RoboVM compiler, to package and install the Phone Dial JavaFX-8-based application on OS simulators. Finally, we provided tips on how to run and deploy your application on real devices. Resources for Article: Further resources on this subject: Function passing [article] Creating Java EE Applications [article] Contexts and Dependency Injection in NetBeans [article]

0
0
10071

Packt

14 Oct 2013

8 min read

What is Oracle Public Cloud?

Packt

14 Oct 2013

8 min read

0
0
10055

Loading data, creating an app, and adding dashboards and reports in Splunk

Getting started with Amazon Machine Learning workflow [Tutorial]

Mission Running in EVE Online

2018 new year resolutions to thrive in an Algorithmic World - Part 1 of 3

Effective Text Generation, Editing and Translation with ChatGPT

Using your smart watch to control networked LEDs

GETTING STARTED WITH SELENIUM WEBDRIVER AND PYTHON

Materials with Ogre 3D

Getting Ready for RubyMotion

Content Switching using Citrix Security

Trending Topics

Index, Item Sharding, and Projection in DynamoDB

The Chain of Responsibility Pattern

Making an entity multiplayer-ready

Developing a JavaFX Application for iOS

What is Oracle Public Cloud?

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access