Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials - Programming

1081 Articles
article-image-learning-r-geospatial-analysis
Packt
08 May 2015
3 min read
Save for later

Learning R for Geospatial Analysis

Packt
08 May 2015
3 min read
The defining feature of spatial data analysis is the reference, within the data being analyzed, to locations on the surface of the earth. This is a very broad subject encompassing distinct areas of expertise such as spatial statistics, geometric computation, and image processing. In practice, spatial data is commonly stored, viewed, and analyzed in Geographic Information System (GIS) software, of which the most well-known example is ArcGIS. However, most often, menu-based interfaces of GIS software are too narrow in scope to meet with specialized demands or too inflexible to feasibly accomplish customized, repetitive tasks. Writing scripts rather than using menus or working in combination with external software are two commonly used paths to solve such problems. However, what if we can use a single environment, combining the advantages of programming and spatial data analysis capabilities with a comprehensive ecosystem of computational tools that are readily implementable in customized procedures? This book will demonstrate that the R programming language is indeed such an environment and teach you how to use it in order to perform various spatial data analysis tasks. (For more resources related to this topic, see here.) What you will learn This book covers the basic concepts related to writing R code. You will also learn how to work with vectors, time series, tables, rasters, points, lines, and polygons. The book also covers several advanced themes associated with raster data analysis in R. Demonstrations on how rasters and vector layers can be combined in a single analysis are shown. Transformation between raster and vector data structures as well as data extraction from a raster based on vector layers are covered in this book. Moreover, we will also learn how spatial interpolation can be carried out in R through examples of interpolating meteorological point measurements to create annual temperature maps of Spain. You will also explore some of the most useful methods for advanced visualization of spatial data in R, using the ggplot2, ggmap, and lattice packages. How the book differs Most currently available books on this subject are focused on advanced applications such as spatial statistics, assuming you have prior knowledge of R and the respective scientific domains. Yet, introductory material on R from the point of view of a spatial data analyst, which is focused on introductory topics such as spatial data handling, computation, and visualization, is scarce. This book aims to fill the gap. Thus, this book is intended for anyone who wants to learn how to efficiently analyze geospatial data with R. No prior experience with R and/or programming is required; only you need to be familiar with basic geographic information concepts (such as spatial coordinates). Required skills To follow through the examples in this book, all you need to do is install R (which is available for free) and download the example datasets from the book's website. Some of the examples also require you to have an Internet connection to download additional datasets and R packages from the R environment. Summary This book is composed of step-by-step tutorials, starting with the language basics before proceeding to cover the main GIS operations and data types. Visualization of spatial data is vital either during the various analysis steps and/or as the final product, and this book shows you how to get the most out of R's visualization capabilities. The book culminates with examples of cutting-edge applications utilizing R's strengths as a statistical and graphical tool. Resources for Article:  Further resources on this subject: Data visualization [article] Machine Learning in Bioinformatics [article] Specialized Machine Learning Topics [article]
Read more
  • 0
  • 0
  • 18379

article-image-understanding-python-regex-engine
Packt
18 Feb 2014
8 min read
Save for later

Understanding the Python regex engine

Packt
18 Feb 2014
8 min read
(For more resources related to this topic, see here.) These are the most common characteristics of the algorithm: It supports "lazy quantifiers" such as *?, +?, and ??. It matches the first coincidence, even though there are longer ones in the string. >>>re.search("engineer | engineering", "engineering").group()'engineer' This also means that order is important. The algorithm tracks only one transition at one step, which means that the engine checks one character at a time. Backreferences and capturing parentheses are supported. Backtracking is the ability to remember the last successful position so that it can go back and retry if needed In the worst case, complexity is exponential O(Cn). We'll see this later in Backtracking. Backtracking Backtracking allows going back and repeating the different paths of the regular expression. It does so by remembering the last successful position, this applies to alternation and quantifiers, let’s see an example: Backtracking As we see in the image the regex engine tries to match one character at a time until it fails and starts again with the following path it can retry. The regex used in the image is the perfect example of the importance of how the regex is built, in this case the expression can be rebuild as spa (in | niard), so that the regex engine doesn’t have to go back up to the start of the string in order to retry the second alternative. This leads us to what is called catastrophic backtracking a well-known problem with backtracking that can give you several problems, ranging from slow regex to a crash with a stack overflow. In the previous example, you can see that the behavior grows not only with the input but also with the different paths in the regex, that’s why the algorithm is exponential O(Cn), with this in mind it’s easy to understand why we can end up with a stack overflow. The problem arises when the regex fails to match the String. Let’s benchmark a regex with technique we’ve seen previously, so we can understand the problem better. First let’s try a simple regex: >>> def catastrophic(n): print "Testing with %d characters" %n pat = re.compile('(a+)+c') text = "%s" %('a' * n) pat.search(text) As you can see the text we’re trying to match it’s always going to fail due to there is no c at the end. Let’s test it with different inputs. >>> for n in range(20, 30): test(catastrophic, n) Testing with 20 characters The function catastrophic lasted: 0.130457 Testing with 21 characters The function catastrophic lasted: 0.245125 …… The function catastrophic lasted: 14.828221 Testing with 28 characters The function catastrophic lasted: 29.830929 Testing with 29 characters The function catastrophic lasted: 61.110949 The behavior of this regex looks like quadratic. But why? what’s happening here? The problem is that (a+) starts greedy, so it tries to get as many a’s as possible and after that it fails to match the (a+)+, that is, it backtracks to the second a, and continue consuming a’s until it fails to match c, when it tries again (backtrack) the whole process starting with the second a. Let’s see another example, in this case with an exponential behavior: >>> def catastrophic(n): print "Testing with %d characters" %n pat = re.compile('(x+)+(b+)+c') text = 'x' * n text += 'b' * n pat.search(text) >>> for n in range(12, 18): test(catastrophic, n) Testing with 12 characters The function catastrophic lasted: 1.035162 Testing with 13 characters The function catastrophic lasted: 4.084714 Testing with 14 characters The function catastrophic lasted: 16.319145 Testing with 15 characters The function catastrophic lasted: 65.855182 Testing with 16 characters The function catastrophic lasted: 276.941307 As you can see the behavior is exponential, which can lead to a catastrophic scenarios. And finally let’s see what happen when regex has a match. >>> def non_catastrophic(n): print "Testing with %d characters" %n pat = re.compile('(x+)+(b+)+c') text = 'x' * n text += 'b' * n text += 'c' pat.search(text) >>> for n in range(12, 18): test(non_catastrophic, n) Testing with 10 characters The function catastrophic lasted: 0.000029 …… Testing with 19 characters The function catastrophic lasted: 0.000012 Optimization recommendations In the following sections we will find a number of recommendations that could be used to apply to improve regular expressions. The best tool will always be the common sense, and even following these recommendations common sense will need to be used. It has to be understood when the recommendation is applicable and when not. For instance the recommendation don’t be greedy cannot be used in the 100% of the cases. Reuse compiled patterns To use a regular expression we have to convert it from the string representation to a compiled form as RegexObject. This compilation takes some time. If instead of using the compile function we are using the rest of the methods to avoid the creation of the RegexObject, we should understand that the compilation is executed anyway and a number of compiled RegexObject are cached automatically. However, when we are compiling that cache won’t back us. Every single compile execution will consume an amount of time that perhaps could be negligible for a single execution, but it’s definitely relevant if many executions are performed. Extract common parts in alternation Alternation is always a performance risk point in regular expressions. When using them in Python, and therefore in a sort of NFA implementation, we should extract any common part outside of the alternation. For instance if we have /(Hello⇢World|Hello⇢Continent|Hello⇢Country,)/, we could easily extract Hello⇢ having the following expression /Hello⇢(World|Continent|Country)/. This will make our engine to just check Hello⇢ once, and not going back and recheck for each possibility. Shortcut the alternation Ordering in alternation is relevant; each of the different options present in the alternation will be checked one by one, from the left to the right. This can be used in favor of performance. If we place the more likely options at the beginning of the alternation, more checks will mark the alternation as matched sooner. For instance, we know that the more common colors of cars are white and black. If we are writing a regular expression accepting some colors, we should put white and black first, as those are the more likely to appear. This is: /(white|black|red|blue|green)/. For the rest of the elements, if they have the very same odds of appearing, if could be favorable to put the shortest ones before the longer ones. Use non capturing groups when appropriate Capturing groups will consume some time per each group defined in an expression. This time is not very important but is still relevant if we are executing a regular expression many times. Sometimes we are using groups but we might not be interested in the result. For instance when using alternation. If that is the case we can save some execution time to the engine by marking that group as non-capturing. This is: (?:person|company).. Be specific When the patterns we define are very specific, the engine can help us performing quick integrity checks before the actual pattern matching is executed. For instance, if we pass to the engine the expression /w{15}/ to be matched against the text hello, the engine could decide to check if the input string is actually at least 15 characters long instead of matching the expression. Don’t be greedy The quantifiers and we learnt the difference between greedy and reluctant quantifiers. We also found that the quantifiers are greedy by default. What does this mean to performance? It means that the engine will always try to catch as many characters as possible and then reducing the scope step by step until it's done. This could potentially make the regular expression slow if the match is typically short. Keep in mind, however, this is only applicable if the match is usually short. Summary In this article, we understood how to see the engine working behind the scenes. We learned some theory of the engine design and how it's easy to fall in a common pitfall—the catastrophic backtracking. Finally, we reviewed different general recommendations to improve the performance of our regular expressions. Resources for Article: Further resources on this subject: Python LDAP Applications: Part 1 - Installing and Configuring the Python-LDAP Library and Binding to an LDAP Directory [Article] Python Data Persistence using MySQL Part III: Building Python Data Structures Upon the Underlying Database Data [Article] Python Testing: Installing the Robot Framework [Article]
Read more
  • 0
  • 0
  • 18289

article-image-web-framework-behavior-tuning
Packt
12 Jan 2017
8 min read
Save for later

Web Framework Behavior Tuning

Packt
12 Jan 2017
8 min read
In this article by Alex Antonov, the author of the book Spring Boot Cookbook – Second Edition, learn to use and configure spring resources and build your own Spring-based application using Spring Boot. In this article, you will learn about the following topics: Configuring route matching patterns Configuring custom static path mappings Adding custom connectors (For more resources related to this topic, see here.) Introduction We will look into enhancing our web application by doing behavior tuning, configuring the custom routing rules and patterns, adding additional static asset paths, and adding and modifying servlet container connectors and other properties, such as enabling SSL. Configuring route matching patterns When we build web applications, it is not always the case that a default, out-of-the-box, mapping configuration is applicable. At times, we want to create our RESTful URLs that contain characters such as . (dot), which Spring treats as a delimiter defining format, like path.xml, or we might not want to recognize a trailing slash, and so on. Conveniently, Spring provides us with a way to get this accomplished with ease. Let's imagine that the ISBN format does allow the use of dots to separate the book number from the revision with a pattern looking like [isbn-number].[revision]. How to do it… We will configure our application to not use the suffix pattern match of .* and not to strip the values after the dot when parsing the parameters. Let's perform the following steps: Let's add the necessary configuration to our WebConfiguration class with the following content: @Override public void configurePathMatch(PathMatchConfigurer configurer) { configurer.setUseSuffixPatternMatch(false). setUseTrailingSlashMatch(true); } Start the application by running ./gradlew clean bootRun. Let's open http://localhost:8080/books/978-1-78528-415-1.1 in the browser to see the following results: If we enter the correct ISBN, we will see a different result, as shown in the following screenshot: How it works… Let's look at what we did in detail. The configurePathMatch(PathMatchConfigurer configurer) method gives us an ability to set our own behavior in how we want Spring to match the request URL path to the controller parameters: configurer.setUseSuffixPatternMatch(false): This method indicates that we don't want to use the .* suffix so as to strip the trailing characters after the last dot. This translates into Spring parsing out 978-1-78528-415-1.1 as an {isbn} parameter for BookController. So, http://localhost:8080/books/978-1-78528-415-1.1 and http://localhost:8080/books/978-1-78528-415-1 will become different URLs. configurer.setUseTrailingSlashMatch(true): This method indicates that we want to use the trailing / in the URL as a match, as if it were not there. This effectively makes http://localhost:8080/books/978-1-78528-415-1 the same as http://localhost:8080/books/978-1-78528-415-1/. If you want to do further configuration on how the path matching takes place, you can provide your own implementation of PathMatcher and UrlPathHelper, but these will be required in the most extreme and custom-tailored situations and are not generally recommended. Configuring custom static path mappings It is possible to control how our web application deals with static assets and the files that exist on the filesystem or are bundled in the deployable archive. Let's say that we want to expose our internal application.properties file via the static web URL of http://localhost:8080/internal/application.properties from our application. To get started with this, proceed with the steps in the next section. How to do it… Let's add a new method, addResourceHandlers, to the WebConfiguration class with the following content: @Override public void addResourceHandlers(ResourceHandlerRegistry registry) { registry.addResourceHandler("/internal/**").addResourceLocations("classpath:/"); } Start the application by running ./gradlew clean bootRun. Let's open http://localhost:8080/internal/application.properties in the browser to see the following results: How it works… The method that we overrode, addResourceHandlers(ResourceHandlerRegistry registry), is another configuration method from WebMvcConfigurer, which gives us an ability to define custom mappings for static resource URLs and connect them with the resources on the filesystem or application classpath. In our case, we defined a mapping of anything that is being accessed via the / internal URL to be looked for in classpath:/ of our application. (For production environment, you probably don't want to expose the entire classpath as a static resource!) So, let's take a detailed look at what we did, as follows: registry.addResourceHandler("/internal/**"): This method adds a resource handler to the registry to handle our static resources, and it returns ResourceHandlerRegistration to us, which can be used to further configure the mapping in a chained fashion. /internal/** is a path pattern that will be used to match against the request URL using PathMatcher. We have seen how PathMatcher can be configured in the previous example but, by default, an AntPathMatcher implementation is used. We can configure more than one URL pattern to be matched to a particular resource location. addResourceLocations("classpath:/"):This method is called on the newly created instance of ResourceHandlerRegistration, and it defines the directories where the resources should be loaded from. These should be valid filesystems or classpath directories, and there can be more than one entered. If multiple locations are provided, they will be checked in the order in which they were entered. setCachePeriod (Integer cachePeriod): Using this method, we can also configure a caching interval for the given resource by adding custom connectors. Another very common scenario in the enterprise application development and deployment is to run the application with two separate HTTP port connectors: one for HTTP and the other for HTTPS. Adding custom connectors Another very common scenario in the enterprise application development and deployment is to run the application with two separate HTTP port connectors: one for HTTP and the other for HTTPS. Getting ready For this recipe, we will undo the changes that we implemented in the previous example. In order to create an HTTPS connector, we will need a few things; but, most importantly, we will need to generate a certificate keystore that is used to encrypt and decrypt the SSL communication with the browser. If you are using Unix or Mac, you can do it by running the following command: $JAVA_HOME/bin/keytool -genkey -alias tomcat -keyalg RSA On Windows, this can be achieved via the following code: "%JAVA_HOME%binkeytool" -genkey -alias tomcat -keyalg RSA During the creation of the keystore, you should enter the information that is appropriate to you, including passwords, name, and so on. For the purpose of this book, we will use the default password: changeit. Once the execution is complete, a newly generated keystore file will appear in your home directory under the name .keystore. You can find more information about preparing the certificate keystore at https://tomcat.apache.org/tomcat-8.0-doc/ssl-howto.html#Prepare_the_Certificate_Keystore. How to do it… With the keystore creation complete, we will need to create a separate properties file in order to store our configuration for the HTTPS connector, such as port and others. After that, we will create a configuration property binding object and use it to configure our new connector. Perform the following steps: First, we will create a new properties file named tomcat.https.properties in the src/main/resources directory from the root of our project with the following content: custom.tomcat.https.port=8443 custom.tomcat.https.secure=true custom.tomcat.https.scheme=https custom.tomcat.https.ssl=true custom.tomcat.https.keystore=${user.home}/.keystore custom.tomcat.https.keystore-password=changeit Next, we will create a nested static class named TomcatSslConnectorProperties in our WebConfiguration, with the following content: @ConfigurationProperties(prefix = "custom.tomcat.https") public static class TomcatSslConnectorProperties { private Integer port; private Boolean ssl= true; private Boolean secure = true; private String scheme = "https"; private File keystore; private String keystorePassword; //Skipping getters and setters to save space, but we do need them public void configureConnector(Connector connector) { if (port != null) connector.setPort(port); if (secure != null) connector.setSecure(secure); if (scheme != null) connector.setScheme(scheme); if (ssl!= null) connector.setProperty("SSLEnabled", ssl.toString()); if (keystore!= null &&keystore.exists()) { connector.setProperty("keystoreFile", keystore.getAbsolutePath()); connector.setProperty("keystorePassword", keystorePassword); } } } Now, we will need to add our newly created tomcat.http.properties file as a Spring Boot property source and enable TomcatSslConnectorProperties to be bound. This can be done by adding the following code right above the class declaration of the WebConfiguration class: @Configuration @PropertySource("classpath:/tomcat.https.properties") @EnableConfigurationProperties(WebConfiguration.TomcatSslConnectorProperties.class) public class WebConfiguration extends WebMvcConfigurerAdapter {...} Finally, we will need to create an EmbeddedServletContainerFactory Spring bean where we will add our HTTPS connector. We will do that by adding the following code to the WebConfiguration class: @Bean public EmbeddedServletContainerFactory servletContainer(TomcatSslConnectorProperties properties) { TomcatEmbeddedServletContainerFactory tomcat = new TomcatEmbeddedServletContainerFactory(); tomcat.addAdditionalTomcatConnectors( createSslConnector(properties)); return tomcat; } private Connector createSslConnector(TomcatSslConnectorProperties properties) { Connector connector = new Connector(); properties.configureConnector(connector); return connector; } Start the application by running ./gradlew clean bootRun. Let's open https://localhost:8443/internal/tomcat.https.properties in the browser to see the following results: Summary In this article, you learned how to fine-tune the behavior of a web application. This article has given a small gist about custom routes, asset paths, and amending routing patterns. You also learned how to add more connectors to the servlet container. Resources for Article: Further resources on this subject: Introduction to Spring Framework [article] Setting up Microsoft Bot Framework Dev Environment [article] Creating our first bot, WebBot [article]
Read more
  • 0
  • 0
  • 18275

article-image-working-user-defined-values-sap-business-one
Packt
03 Jun 2011
8 min read
Save for later

Working with User Defined Values in SAP Business One

Packt
03 Jun 2011
8 min read
  Mastering SQL Queries for SAP Business One Utilize the power of SQL queries to bring Business Intelligence to your small to medium-sized business         Read more about this book       The User-Defined Values function enables SAP Business One users to enter values, originated by a predefined search process, for any field in the system (including user-defined fields). This function enables the user to enter data more efficiently and – perhaps most importantly – more accurately. In fact, the concept is sort of a "Workflow Light" implementation. It can both save user time and reduce data double entries. In this article by Gordon Du, author of Mastering SQL Queries for SAP Business One, we will see how to work with User-Defined Values. (For more resources on Microsoft, see here.) How to work with User-Defined Values To access the User-Defined Values, you can choose menu item Tools | User-Defined Values. You can also use the shortcut key Shift+Alt+F2 instead. Another option is to access it directly from a non-assigned field by using Shift+F2. This will be discussed later. You must notice that the option will not be available until you brought up at least one form. This is because the UDV has to be associated with a form. It can't stand alone. The following screenshots are taken from A/R Down Payment Invoice. It is one of the standard marketing documents. From the UDV point of view, there is no big difference between this and the other types of documents, namely, Sales Order, Purchase Order, Invoice, and so on. After a form is opened, a UDV can be defined. We will start from an empty screen to show you the first step: bringing up a form. When a form is opened, you can define or change any UDV. In this case, we stop our cursor on the Due Date field and then enter Shift+F2. A system message will pop up as shown in the following screenshot: If you click on Yes, it will bring up the same window in the manner you select the menu item mentioned earlier from the Tools menu or press Shift+Alt+F2. When you get the User-Define Values-Setup screen, you have three options. Apart from the default option: Without Search User-Define Values, you actually have only two choices: Search in Existing User-Define Values Search in Existing User-Define Values according to Saved Query Let's go through the last option first: Search in Existing User-Define Values according to Saved Query. The topic related to query will always be assigned with the top priority. There are quite a few screenshots that will help you understand the entire process. Search in existing User-Defined Values according to the saved queries The goal for this example is to input the due date as the current date automatically. The first thing to do for this option is to click on the bottom radio button among three options. The screenshot is shown next: After you have clicked the Search in Existing User-Defined Values according to Saved Query radio button, you will find a long empty textbox in a grey color and a checkbox for Auto Refresh When Field Changes underneath. Don't get confused by the color. Even though in other functions throughout SAP Business One, a gray colored field normally means that you cannot input or enter information into the field. That is not the case here. You can double-click it to get the User-Defined Values. When you double-click on the empty across-window text box, you can bring up the query manager window to select a query. You can then browse the query category that relates to Formatted Searches and find the query you need. The query called Auto Date Today in the showcase is very simple. The query script is as simple as this: SELECT GetDate() This query returns the current date as the result. You need to double-click to select the query and then go back to the previous screen but with the query name, as shown in the following screenshot: It may not be good enough to select only query because if you stop here you have to always manually trigger the FMS query run by entering Shift+F2. To automate the FMS query process, you can click on the checkbox under the selected query. After you check this box, another long text box will be displayed with a drop-down list button. Under the text box, there are two radio buttons for Auto Refresh When Field Changes: Refresh Regularly Display Saved User-Defined Value Display Saved User-Defined Values will be the default selection, if you do not change it. When you click on the drop-down list arrow button, you will get a list of fields that are associated with the current form. You can see in the following screenshot that Customer/Vendor Code field has been selected. For header document UDV, this field is often the most useful field to auto refresh the UDV. In theory, you can select any fields from the list. However, in reality only a few fields are good candidates for the task. These include Customer/Vendor Code, Document Currency, Document Number, and Document Total for document header; Item Code and Quantity for document lines. Choosing the correct data field from this drop-down list is always the most difficult step in Formatted Search, and you should test your data field selection fully. Now, the text box is filled with Customer/Vendor Code for automatically refreshing the UDV. Between two options, this query can only select the default option of Display Saved User-Defined Value. Otherwise, the date will always change to the date you have updated the document on. That will invalidate the usage of this UDV. The Refresh Regularly option is only suitable to the value that is closely related to the changed field that you have selected. In general, Display Saved User-Defined Value is always a better option than Refresh Regularly. At least it gives the system less burden. If you have selected Refresh Regularly, it means you want to get the UDV changed whenever the base field changes. The last step to set up this UDV is by clicking Update. As soon as you click the button, the User-Defined Values–Setup window will be closed. You can find a green message on the bottom-left of the screen saying Operation Completed Successfully. You can find a small "magnifying glass" added to the right corner of the Due Date field. This means the Formatted Search is successfully set up. You can try it for yourself. Sometimes this "magnifying glass" disappears for no reason. Actually, there are reasons but not easy to be understood. The main reason is that you may have assigned some different values to the same field on different forms. Other reasons may be related to add-on, and so on. In order to test this FMS, the first thing to try is to use the menu function or key combination Shift+F2. The other option is to just click on the "magnifying glass". Both functions have the same result. It will force the query to run. You can find that the date is filled by the same date as posting date and document date. You may find some interesting date definitions in SAP Business One, such as Posting Date is held by the field DocDate. Document Date however, is saved under TaxDate. Be careful in dealing with dates. You must follow the system's definition in using those terms, so that you get the correct result. A better way to use this FMS query is by entering the customer code directly without forcing FMS query to run first. The following screenshot shows that the customer code OneTime has been entered. Please note that the DueDate field is still empty. Is there anything wrong? No. That is the system's expected behavior. Only if your cursor leaves the Customer Code field, can the FMS query be triggered. That is a perfect example of When Field Value Changes. The system can only know that the field value is changed when you tab out of the field. When you are working with the field, the field is not changed yet. Be careful to follow system requirements while entering data. Never press Enter in most of the forms unless you are ready for the last step to add or update data. If you do, you may add the wrong documents to the system and they are irrevocable. The previous screenshot shows the complete process of setting up search in Existing User-Define Values according to Saved Query. Now it is time to discuss the $ sign field.
Read more
  • 0
  • 0
  • 18254

article-image-exploring-scala-performance
Packt
19 May 2016
19 min read
Save for later

Exploring Scala Performance

Packt
19 May 2016
19 min read
In this article by Michael Diamant and Vincent Theron, author of the book Scala High Performance Programming, we look at how Scala features get compiled with bytecode. (For more resources related to this topic, see here.) Value classes The domain model of the order book application included two classes, Price and OrderId. We pointed out that we created domain classes for Price and OrderId to provide contextual meanings to the wrapped BigDecimal and Long. While providing us with readable code and compilation time safety, this practice also increases the amount of instances that are created by our application. Allocating memory and generating class instances create more work for the garbage collector by increasing the frequency of collections and by potentially introducing additional long-lived objects. The garbage collector will have to work harder to collect them, and this process may severely impact our latency. Luckily, as of Scala 2.10, the AnyVal abstract class is available for developers to define their own value classes to solve this problem. The AnyVal class is defined in the Scala doc (http://www.scala-lang.org/api/current/#scala.AnyVal) as, "the root class of all value types, which describe values not implemented as objects in the underlying host system." The AnyVal class can be used to define a value class, which receives special treatment from the compiler. Value classes are optimized at compile time to avoid the allocation of an instance, and instead, they use the wrapped type. Bytecode representation As an example, to improve the performance of our order book, we can define Price and OrderId as value classes: case class Price(value: BigDecimal) extends AnyVal case class OrderId(value: Long) extends AnyVal To illustrate the special treatment of value classes, we define a dummy function taking a Price value class and an OrderId value class as arguments: def printInfo(p: Price, oId: OrderId): Unit = println(s"Price: ${p.value}, ID: ${oId.value}") From this definition, the compiler produces the following method signature: public void printInfo(scala.math.BigDecimal, long); We see that the generated signature takes a BigDecimal object and a long object, even though the Scala code allows us to take advantage of the types defined in our model. This means that we cannot use an instance of BigDecimal or Long when calling printInfo because the compiler will throw an error. An interesting thing to notice is that the second parameter of printInfo is not compiled as Long (an object), but long (a primitive type, note the lower case 'l'). Long and other objects matching to primitive types, such as Int,Float or Short, are specially handled by the compiler to be represented by their primitive type at runtime. Value classes can also define methods. Let's enrich our Price class, as follows: case class Price(value: BigDecimal) extends AnyVal { def lowerThan(p: Price): Boolean = this.value < p.value } // Example usage val p1 = Price(BigDecimal(1.23)) val p2 = Price(BigDecimal(2.03)) p1.lowerThan(p2) // returns true Our new method allows us to compare two instances of Price. At compile time, a companion object is created for Price. This companion object defines a lowerThan method that takes two BigDecimal objects as parameters. In reality, when we call lowerThan on an instance of Price, the code is transformed by the compiler from an instance method call to a static method call that is defined in the companion object: public final boolean lowerThan$extension(scala.math.BigDecimal, scala.math.BigDecimal); Code: 0: aload_1 1: aload_2 2: invokevirtual #56 // Method scala/math/BigDecimal.$less:(Lscala/math/BigDecimal;)Z 5: ireturn If we were to write the pseudo-code equivalent to the preceding Scala code, it would look something like the following: val p1 = BigDecimal(1.23) val p2 = BigDecimal(2.03) Price.lowerThan(p1, p2) // returns true   Performance considerations Value classes are a great addition to our developer toolbox. They help us reduce the count of instances and spare some work for the garbage collector, while allowing us to rely on meaningful types that reflect our business abstractions. However, extending AnyVal comes with a certain set of conditions that the class must fulfill. For example, a value class may only have one primary constructor that takes one public val as a single parameter. Furthermore, this parameter cannot be a value class. We saw that value classes can define methods via def. Neither val nor var are allowed inside a value class. A nested class or object definitions are also impossible. Another limitation prevents value classes from extending anything other than a universal trait, that is, a trait that extends Any, only has defs as members, and performs no initialization. If any of these conditions is not fulfilled, the compiler generates an error. In addition to the preceding constraints that are listed, there are special cases in which a value class has to be instantiated by the JVM. Such cases include performing a pattern matching or runtime type test, or assigning a value class to an array. An example of the latter looks like the following snippet: def newPriceArray(count: Int): Array[Price] = { val a = new Array[Price](count) for(i <- 0 until count){ a(i) = Price(BigDecimal(Random.nextInt())) } a } The generated bytecode is as follows: public highperfscala.anyval.ValueClasses$$anonfun$newPriceArray$1(highperfscala.anyval.ValueClasses$Price[]); Code: 0: aload_0 1: aload_1 2: putfield #29 // Field a$1:[Lhighperfscala/anyval/ValueClasses$Price; 5: aload_0 6: invokespecial #80 // Method scala/runtime/AbstractFunction1$mcVI$sp."<init>":()V 9: return public void apply$mcVI$sp(int); Code: 0: aload_0 1: getfield #29 // Field a$1:[Lhighperfscala/anyval/ValueClasses$Price; 4: iload_1 5: new #31 // class highperfscala/anyval/ValueClasses$Price // omitted for brevity 21: invokevirtual #55 // Method scala/math/BigDecimal$.apply:(I)Lscala/math/BigDecimal; 24: invokespecial #59 // Method highperfscala/anyval/ValueClasses$Price."<init>":(Lscala/math/BigDecimal;)V 27: aastore 28: return Notice how mcVI$sp is invoked from newPriceArray, and this creates a new instance of ValueClasses$Price at the 5 instruction. As turning a single field case class into a value class is as trivial as extending the AnyVal trait, we recommend that you always use AnyVal wherever possible. The overhead is quite low, and it generate high benefits in terms of garbage collection's performance. To learn more about value classes, their limitations and use cases, you can find detailed descriptions at http://docs.scala-lang.org/overviews/core/value-classes.html. Tagged types – an alternative to value classes Value classes are an easy to use tool, and they can yield great improvements in terms of performance. However, they come with a constraining set of conditions, which can make them impossible to use in certain cases. We will conclude this section with a glance at an interesting alternative to leveraging the tagged type feature that is implemented by the Scalaz library. The Scalaz implementation of tagged types is inspired by another Scala library, named shapeless. The shapeless library provides tools to write type-safe, generic code with minimal boilerplate. While we will not explore shapeless, we encourage you to learn more about the project at https://github.com/milessabin/shapeless. Tagged types are another way to enforce compile-type checking without incurring the cost of instance instantiation. They rely on the Tagged structural type and the @@ type alias that is defined in the Scalaz library, as follows: type Tagged[U] = { type Tag = U } type @@[T, U] = T with Tagged[U] Let's rewrite part of our code to leverage tagged types with our Price object: object TaggedTypes { sealed trait PriceTag type Price = BigDecimal @@ PriceTag object Price { def newPrice(p: BigDecimal): Price = Tag[BigDecimal, PriceTag](p) def lowerThan(a: Price, b: Price): Boolean = Tag.unwrap(a) < Tag.unwrap(b) } } Let's perform a short walkthrough of the code snippet. We will define a PriceTag sealed trait that we will use to tag our instances, a Price type alias is created and defined as a BigDecimal object tagged with PriceTag. The Price object defines useful functions, including the newPrice factory function that is used to tag a given BigDecimal object and return a Price object (that is, a tagged BigDecimal object). We will also implement an equivalent to the lowerThan method. This function takes two Price objects (that is two tagged BigDecimal objects), extracts the content of the tag that are two BigDecimal objects, and compares them. Using our new Price type, we rewrite the same newPriceArray function that we previously looked at (the code is omitted for brevity, but you can refer to it in the attached source code), and print the following generated bytecode: public void apply$mcVI$sp(int); Code: 0: aload_0 1: getfield #29 // Field a$1:[Ljava/lang/Object; 4: iload_1 5: getstatic #35 // Field highperfscala/anyval/TaggedTypes$Price$.MODULE$:Lhighperfscala/anyval/TaggedTypes$Price$; 8: getstatic #40 // Field scala/package$.MODULE$:Lscala/package$; 11: invokevirtual #44 // Method scala/package$.BigDecimal:()Lscala/math/BigDecimal$; 14: getstatic #49 // Field scala/util/Random$.MODULE$:Lscala/util/Random$; 17: invokevirtual #53 // Method scala/util/Random$.nextInt:()I 20: invokevirtual #58 // Method scala/math/BigDecimal$.apply:(I)Lscala/math/BigDecimal; 23: invokevirtual #62 // Method highperfscala/anyval/TaggedTypes$Price$.newPrice:(Lscala/math/BigDecimal;)Ljava/lang/Object; 26: aastore 27: return In this version, we no longer see an instantiation of Price, even though we are assigning them to an array. The tagged Price implementation involves a runtime cast, but we anticipate that the cost of this cast will be less than the instance allocations (and garbage collection) that was observed in the previous value class Price strategy. Specialization To understand the significance of specialization, it is important to first grasp the concept of object boxing. The JVM defines primitive types (boolean, byte, char, float, int, long, short, and double) that are stack allocated rather than heap allocated. When a generic type is introduced, for example, scala.collection.immutable.List, the JVM references an object equivalent, instead of a primitive type. In this example, an instantiated list of integers would be heap allocated objects rather than integer primitives. The process of converting a primitive to its object equivalent is called boxing, and the reverse process is called unboxing. Boxing is a relevant concern for performance-sensitive programming because boxing involves heap allocation. In performance-sensitive code that performs numerical computations, the cost of boxing and unboxing can create an order of magnitude or larger performance slowdowns. Consider the following example to illustrate boxing overhead: List.fill(10000)(2).map(_* 2) Creating the list via fill yields 10,000 heap allocations of the integer object. Performing the multiplication in map requires 10,000 unboxings to perform multiplication and then 10,000 boxings to add the multiplication result into the new list. From this simple example, you can imagine how critical section arithmetic will be slowed down due to boxing or unboxing operations. As shown in Oracle's tutorial on boxing at https://docs.oracle.com/javase/tutorial/java/data/autoboxing.html, boxing in Java and also in Scala happens transparently. This means that without careful profiling or bytecode analysis, it is difficult to discern where you are paying the cost for object boxing. To ameliorate this problem, Scala provides a feature named specialization. Specialization refers to the compile-time process of generating duplicate versions of a generic trait or class that refer directly to a primitive type instead of the associated object wrapper. At runtime, the compiler-generated version of the generic class, or as it is commonly referred to, the specialized version of the class, is instantiated. This process eliminates the runtime cost of boxing primitives, which means that you can define generic abstractions while retaining the performance of a handwritten, specialized implementation. Bytecode representation Let's look at a concrete example to better understand how the specialization process works. Consider a naive, generic representation of the number of shares purchased, as follows: case class ShareCount[T](value: T) For this example, let's assume that the intended usage is to swap between an integer or long representation of ShareCount. With this definition, instantiating a long-based ShareCount instance incurs the cost of boxing, as follows: def newShareCount(l: Long): ShareCount[Long] = ShareCount(l) This definition translates to the following bytecode: public highperfscala.specialization.Specialization$ShareCount<java.lang.Object> newShareCount(long); Code: 0: new #21 // class orderbook/Specialization$ShareCount 3: dup 4: lload_1 5: invokestatic #27 // Method scala/runtime/BoxesRunTime.boxToLong:(J)Ljava/lang/Long; 8: invokespecial #30 // Method orderbook/Specialization$ShareCount."<init>":(Ljava/lang/Object;)V 11: areturn In the preceding bytecode, it is clear in the 5 instruction that the primitive long value is boxed before instantiating the ShareCount instance. By introducing the @specialized annotation, we are able to eliminate the boxing by having the compiler provide an implementation of ShareCount that works with primitive long values. It is possible to specify which types you wish to specialize by supplying a set of types. As defined in the Specializables trait (http://www.scala-lang.org/api/current/index.html#scala.Specializable), you are able to specialize for all JVM primitives, such as Unit and AnyRef. For our example, let's specialize ShareCount for integers and longs, as follows: case class ShareCount[@specialized(Long, Int) T](value: T) With this definition, the bytecode now becomes the following: public highperfscala.specialization.Specialization$ShareCount<java.lang.Object> newShareCount(long); Code: 0: new #21 // class highperfscala.specialization/Specialization$ShareCount$mcJ$sp 3: dup 4: lload_1 5: invokespecial #24 // Method highperfscala.specialization/Specialization$ShareCount$mcJ$sp."<init>":(J)V 8: areturn The boxing disappears and is curiously replaced with a different class name, ShareCount $mcJ$sp. This is because we are invoking the compiler-generated version of ShareCount that is specialized for long values. By inspecting the output of javap, we see that the specialized class generated by the compiler is a subclass of ShareCount: public class highperfscala.specialization.Specialization$ShareCount$mcI$sp extends highperfscala.specialization.Specialization$ShareCount<java .lang.Object> Bear this specialization implementation detail in mind as we turn to the Performance considerations section. The use of inheritance forces tradeoffs to be made in more complex use cases. Performance considerations At first glance, specialization appears to be a simple panacea for JVM boxing. However, there are several caveats to consider when using specialization. A liberal use of specialization leads to significant increases in compile time and resulting code size. Consider specializing Function3, which accepts three arguments as input and produces one result. To specialize four arguments across all types (that is, Byte, Short, Int, Long, Char, Float, Double, Boolean, Unit, and AnyRef) yields 10^4 or 10,000 possible permutations. For this reason, the standard library conserves application of specialization. In your own use cases, consider carefully which types you wish to specialize. If we specialize Function3 only for Int and Long, the number of generated classes shrinks to 2^4 or 16. Specialization involving inheritance requires extra attention because it is trivial to lose specialization when extending a generic class. Consider the following example: class ParentFoo[@specialized T](t: T) class ChildFoo[T](t: T) extends ParentFoo[T](t) def newChildFoo(i: Int): ChildFoo[Int] = new ChildFoo[Int](i) In this scenario, you likely expect that ChildFoo is defined with a primitive integer. However, as ChildFoo does not mark its type with the @specialized annotation, zero specialized classes are created. Here is the bytecode to prove it: public highperfscala.specialization.Inheritance$ChildFoo<java.lang.Object> newChildFoo(int); Code: 0: new #16 // class highperfscala/specialization/Inheritance$ChildFoo 3: dup 4: iload_1 5: invokestatic #22 // Method scala/runtime/BoxesRunTime.boxToInteger:(I)Ljava/lang/Integer; 8: invokespecial #25 // Method highperfscala/specialization/Inheritance$ChildFoo."<init>":(Ljava/lang/Object;)V 11: areturn The next logical step is to add the @specialized annotation to the definition of ChildFoo. In doing so, we stumble across a scenario where the compiler warns about use of specialization, as follows: class ParentFoo must be a trait. Specialized version of class ChildFoo will inherit generic highperfscala.specialization.Inheritance.ParentFoo[Boolean] class ChildFoo[@specialized T](t: T) extends ParentFoo[T](t) The compiler indicates that you have created a diamond inheritance problem, where the specialized versions of ChildFoo extend both ChildFoo and the associated specialized version of ParentFoo. This issue can be resolved by modeling the problem with a trait, as follows: trait ParentBar[@specialized T] { def t(): T } class ChildBar[@specialized T](val t: T) extends ParentBar[T] def newChildBar(i: Int): ChildBar[Int] = new ChildBar(i) This definition compiles using a specialized version of ChildBar, as we originally were hoping for, as see in the following code: public highperfscala.specialization.Inheritance$ChildBar<java.lang.Object> newChildBar(int); Code: 0: new #32 // class highperfscala/specialization/Inheritance$ChildBar$mcI$sp 3: dup 4: iload_1 5: invokespecial #35 // Method highperfscala/specialization/Inheritance$ChildBar$mcI$sp."<init>":(I)V 8: areturn An analogous and equally error-prone scenario is when a generic function is defined around a specialized type. Consider the following definition: class Foo[T](t: T) object Foo { def create[T](t: T): Foo[T] = new Foo(t) } def boxed: Foo[Int] = Foo.create(1) Here, the definition of create is analogous to the child class from the inheritance example. Instances of Foo wrapping a primitive that are instantiated from the create method will be boxed. The following bytecode demonstrates how boxed leads to heap allocations: public highperfscala.specialization.MethodReturnTypes$Foo<java.lang.Object> boxed(); Code: 0: getstatic #19 // Field highperfscala/specialization/MethodReturnTypes$Foo$.MODULE$:Lhighperfscala/specialization/MethodReturnTypes$Foo$; 3: iconst_1 4: invokestatic #25 // Method scala/runtime/BoxesRunTime.boxToInteger:(I)Ljava/lang/Integer; 7: invokevirtual #29 // Method highperfscala/specialization/MethodReturnTypes$Foo$.create:(Ljava/lang/Object;)Lhighperfscala/specialization/MethodReturnTypes$Foo; 10: areturn The solution is to apply the @specialized annotation at the call site, as follows: def createSpecialized[@specialized T](t: T): Foo[T] = new Foo(t) The solution is to apply the @specialized annotation at the call site, as follows: def createSpecialized[@specialized T](t: T): Foo[T] = new Foo(t) One final interesting scenario is when specialization is used with multiple types and one of the types extends AnyRef or is a value class. To illustrate this scenario, consider the following example: case class ShareCount(value: Int) extends AnyVal case class ExecutionCount(value: Int) class Container2[@specialized X, @specialized Y](x: X, y: Y) def shareCount = new Container2(ShareCount(1), 1) def executionCount = new Container2(ExecutionCount(1), 1) def ints = new Container2(1, 1) In this example, which methods do you expect to box the second argument to Container2? For brevity, we omit the bytecode, but you can easily inspect it yourself. As it turns out, shareCount and executionCount box the integer. The compiler does not generate a specialized version of Container2 that accepts a primitive integer and a value extending AnyVal (for example, ExecutionCount). The shareCount variable also causes boxing due to the order in which the compiler removes the value class type information from the source code. In both scenarios, the workaround is to define a case class that is specific to a set of types (for example, ShareCount and Int). Removing the generics allows the compiler to select the primitive types. The conclusion to draw from these examples is that specialization requires extra focus to be used throughout an application without boxing. As the compiler is unable to infer scenarios where you accidentally forgot to apply the @specialized annotation, it fails to raise a warning. This places the onus on you to be vigilant about profiling and inspecting bytecode to detect scenarios where specialization is incidentally dropped. To combat some of the shortcomings that specialization brings, there is a compiler plugin under active development, named miniboxing, at http://scala-miniboxing.org/. This compiler plugin applies a different strategy that involves encoding all primitive types into a long value and carrying metadata to recall the original type. For example, boolean can be represented in long using a single bit to signal true or false. With this approach, performance is qualitatively similar to specialization while producing orders of magnitude for fewer classes for large permutations. Additionally, miniboxing is able to more robustly handle inheritance scenarios and can warn when boxing will occur. While the implementations of specialization and miniboxing differ, the end user usage is quite similar. Like specialization, you must add appropriate annotations to activate the miniboxing plugin. To learn more about the plugin, you can view the tutorials on the miniboxing project site. The extra focus to ensure specialization produces heap-allocation free code is worthwhile because of the performance wins in performance-sensitive code. To drive home the value of specialization, consider the following microbenchmark that computes the cost of a trade by multiplying share count with execution price. For simplicity, primitive types are used directly instead of value classes. Of course, in production code this would never happen: @BenchmarkMode(Array(Throughput)) @OutputTimeUnit(TimeUnit.SECONDS) @Warmup(iterations = 3, time = 5, timeUnit = TimeUnit.SECONDS) @Measurement(iterations = 30, time = 10, timeUnit = TimeUnit.SECONDS) @Fork(value = 1, warmups = 1, jvmArgs = Array("-Xms1G", "-Xmx1G")) class SpecializationBenchmark { @Benchmark def specialized(): Double = specializedExecution.shareCount.toDouble * specializedExecution.price @Benchmark def boxed(): Double = boxedExecution.shareCount.toDouble * boxedExecution.price } object SpecializationBenchmark { class SpecializedExecution[@specialized(Int) T1, @specialized(Double) T2]( val shareCount: Long, val price: Double) class BoxingExecution[T1, T2](val shareCount: T1, val price: T2) val specializedExecution: SpecializedExecution[Int, Double] = new SpecializedExecution(10l, 2d) val boxedExecution: BoxingExecution[Long, Double] = new BoxingExecution(10l, 2d) } In this benchmark, two versions of a generic execution class are defined. SpecializedExecution incurs zero boxing when computing the total cost because of specialization, while BoxingExecution requires object boxing and unboxing to perform the arithmetic. The microbenchmark is invoked with the following parameterization: sbt 'project chapter3' 'jmh:run SpecializationBenchmark -foe true' We configure this JMH benchmark via annotations that are placed at the class level in the code. Annotations have the advantage of setting proper defaults for your benchmark, and simplifying the command-line invocation. It is still possible to override the values in the annotation with command-line arguments. We use the -foe command-line argument to enable failure on error because there is no annotation to control this behavior. In the rest of this book, we will parameterize JMH with annotations and omit the annotations in the code samples because we always use the same values. The results are summarized in the following table: Benchmark Throughput (ops per second) Error as percentage of throughput boxed 251,534,293.11 ±2.23 specialized 302,371,879.84 ±0.87 This microbenchmark indicates that the specialized implementation yields approximately 17% higher throughput. By eliminating boxing in a critical section of the code, there is an order of magnitude performance improvement available through judicious usage of specialization. For performance-sensitive arithmetic, this benchmark provides justification for the extra effort that is required to ensure that specialization is applied properly. Summary This article talk about different Scala constructs and features. It also explained different features and how they get compiled with bytecode. Resources for Article: Further resources on this subject: Differences in style between Java and Scala code [article] Integrating Scala, Groovy, and Flex Development with Apache Maven [article] Cluster Computing Using Scala [article]
Read more
  • 0
  • 0
  • 18191

article-image-connecting-microsoft-sql-server-compact-35-visual-studio
Packt
13 Jul 2010
3 min read
Save for later

Connecting to Microsoft SQL Server Compact 3.5 with Visual Studio

Packt
13 Jul 2010
3 min read
SQL Server Compact Edition 3.5 can be used to create applications that are useful for a number of business uses such as: Portable applications; Occasionally connected clients and embedded applications and devices. SQL Server Compact differs from other SQL Servers in that there is just one file which can be password protected and features 128-bit file level encryption. It is referential integrity compliant; supports multiple connections; has transactions support with rich data types. In this tutorial by Jayaram Krishnaswamy, various scenarios where you may need to connect to SQL Server Compact using Visual Studio IDE (both 2008 and 2010) are described in detail. Connecting to SQL Server Compact 4.5 using Visual Studio 2010 Express (free version of Visual Studio) is also described. The connection is the starting point for any database related program and therefore mastering the connection task is crucial to work with the SQL Server Compact. (For more resources on Microsoft, see here.) If you are familiar with SQL Server you already know much of SQL Server Compact. It can be administered from SSMS and, using SQL Syntax and ADO.NET technology you can be immediately productive with SQL Server Compact.It is free to download (also free to deploy and redistribute) and comes in the form of just one code-free file. Its small foot print makes it easily deployable to a variety of device sizes and requires no administration. It also supports a subset of T-SQL and a rich set of data types. It can be used in creating desktop/web applications using Visual Studio 2008 and Visual Studio 2010. It also comes with a sample Northwind database. Download details Microsoft SQL Server Compact 3.5 may be downloaded from this site here. Make sure you download detailed features of this program from the same site. Also several bugs have been fixed in the program as detailed in the two SP's. Link to the latest service pack SP2 is here. By applying SP2 the installed version on the machine is upgraded to the latest version. Connecting to SQL Server Compact from Windows and Web projects You can use the Server Explorer in Visual Studio to drag and drop objects from SQL Server Compact provided you add a connection to the SQL Server Compact. In fact, in Visual Studio 2008 IDE you can configure a data connection without even starting a project from the View menu as shown here. When you click Add Connection... the following window will be displayed. This brings up the Add Connection dialog shown here. Click Change... to choose the correct data source for SQL Server Compact. The default is SQL Server client. The Change Data Source window is displayed as shown. Highlight Microsoft SQL Server Compact 3.5 and click OK. You are returned to Add Connection where you can browse or create a database or, choose also from a ActiveSync connected device such as a Smart phone which has a SQL Server Compact for devices installed. Presently connect to one on the computer (My Computer default option)-the sample database Northwind. Click Browse.... The Select SQL Server Compact 3.5 Database File dialog opens where your sample database Northwind is displayed as shown. Click Open. The database file is entered in the Add Connection dialogue. You may test the connection. You should get a Test connection succeeded message from Microsoft Visual Studio. Click OK. The Northwind.sdf file is displayed as a tree with Tables and View as shown in the next figure. Right click Northwind.sdf in the Server Explorer above and click Properties drop-down menu item. You will see the connection string for this conneciton as shown here.
Read more
  • 0
  • 1
  • 18109
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-enterprise-architecture-concepts
Packt
02 Jan 2017
8 min read
Save for later

Enterprise Architecture Concepts

Packt
02 Jan 2017
8 min read
In this article by Habib Ahmed Qureshi, Ganesan Senthilvel, and Ovais Mehboob Ahmed Khan, author of the book Enterprise Application Architecture with .NET Core, you will learn how to architect and design highly scalable, robust, clean, and highly performant applications in .NET Core 1.0. (For more resources related to this topic, see here.) In this article, we will cover the following topics: Why we need Enterprise Architecture? Knowing the role of an architect Why we need Enterprise Architecture? We will need to define, or at least provide, some basic fixed points to identify enterprise architecture specifically. Sketch Before playing an enterprise architect role, I used to get confused with so many architectural roles and terms, such as architect, solution architect, enterprise architect, data architect, blueprint, system diagram, and so on. In general, the industry perception is that the IT architect role is to draw few boxes with few suggestions; rest is with the development community. They feel that the architect role is quite easy just by drawing the diagram and not doing anything else. Like I said, it is completely a perception of few associates in the industry, and I used to be dragged by this category earlier: However, my enterprise architect job has cleared this perception and understands the true value of an enterprise architect. Definition of Enterprise Architecture In simple terms, enterprise is nothing but human endeavor. The objective of an enterprise is where people are collaborating for a particular purpose supported by a platform. Let me explain with an example of an online e-commerce company. Employees of that company are people who worked together to produce the profit of the firm using their various platforms, such as infrastructure, software, equipment, building, and so on. Enterprise has the structure/arrangements of all these pieces/components to build the complete organization. This is the exact place where enterprise architecture plays its key role. Every enterprise has an enterprise architect. EA is a process of architecting that applies the discipline to produce the prescribed output components. This process needs the experience, skill, discipline, and descriptions. Consider the following image where EA anticipates the system in two key states: Every enterprise needs an enterprise architect, not an optional. Let me give a simple example. When you need a car for business activities, you have two choices, either drive yourself or rent a driver. Still, you will need the driving capability to operate the car. EA is pretty similar to it. As depicted in the preceding diagram, EA anticipates the system in two key states, which are as follows: How it currently is How will it be in the future Basically, they work on options/alternatives to move from current to future state of an enterprise system. In this process, Enterprise Architecture does the following: Creates the frameworks to manage the architect Details the descriptions of the architect Roadmaps to lay the best way to change/improve the architecture Defines constraint/opportunity Anticipates the costs and benefits Evaluates the risks and values In this process of architecting, the system applies the discipline to produce the prescribed output components. Stakeholders of Enterprise Architecture Enterprise Architecture is so special because to its holistic view of management and evolution of an enterprise holistically. It has the unique combination of specialist technology, such as architecture frameworks and design pattern practices. Such a special EA has the following key stakeholders/users in its eco system: S.No. Stakeholders Organizational actions 1  Strategic planner Capability planning Set strategic direction Impact analysis 2  Decision makers Investment Divestment Approvals for the project Alignment with strategic direction 3  Analyst Quality assurance Compliance Alignment with business goals 4  Architects, project managers Solution development Investigate the opportunities Analysis of the existing options Business benefits Though many organizations intervened without EAs, every firm has the strong belief that it is better to architect before creating any system. It is integrated in coherent fashion with proactively designed system instead of random ad hoc and inconsistent mode. In terms of business benefits, cost is the key factor in the meaning of Return on Investment (RoI). That is how the industry business is driven in this highly competitive IT world. EA has the opportunity to prove its value for its own stakeholders with three major benefits, ranging from tactical to strategic positions. They are as follows: Cost reduction by technology standardization Business Process Improvement (BPI) Strategic differentiation Gartner's research paper on TCO: The First Justification for Enterprise IT Architecture by Colleen Young is one of the good references to justify the business benefits of an Enterprise Architecture. Check out https://www.gartner.com/doc/388268/enterprise-architecture-benefits-justification for more information. In the grand scheme of cost saving strategy, technology standardization adds a lot of efficiency to make the indirect benefits. Let me share my experience in this space. In one of my earlier legacy organization, it was noticed that the variety of technologies and products were built to server the business purpose due to the historical acquisitions and mergers. The best solution was platform standardization. All businesses have processes; few life examples are credit card processing, employee on-boarding, student enrollment, and so on. In this methodology, there are people involved with few steps for the particular system to get things done. In means of the business growth, the processes become chaotic, which leads to the duplicate efforts across the departments. Here, we miss the cross learning of the mistakes and corrections. BPI is an industry approach that is designed to support the enterprise for the realignment of the existing business operational process into the significant improved process. It helps the enterprise to identify and adopt in a better way using the industry tools and techniques. BPI is originally designed to induce a drastic game changing effect in the enterprise performance instead of bringing the changes in the incremental steps. In the current highly competitive market, strategic differentiation efforts make the firm create the perception in customers minds of receiving something of greater value than offered by the competition. An effective differentiation strategy is the best tool to highlight a business's unique features and make it stand out from the crowd. As the outcome of strategic differentiation, the business should realize the benefits on Enterprise Architecture investment. Also, it makes the business to institute the new ways of thinking to add the new customer segments along with new major competitive strategies. Knowing the role of an architect When I planned to switch my career to architecture track, I had too many questions in mind. People were referring to so many titles in the industry, such as architect, solution architect, enterprise architect, data architect, infra architect, and so on that I didn't know where exactly do I need to start and end. Industry had so many confusions to opt for. To understand it better, let me give my own work experiences as the best use cases. In the IT industry, two higher-level architects are named as follows: Solution architect (SA) Enterprise architect (EA) In my view, Enterprise Architecture is a much broader discipline than Solution Architecture with the sum of Business Architecture, Application Architecture, Data Architecture, and Technology Architecture. It will be covered in detail in the subsequent section: SA is focused on a specific solution and addresses the technological details that are compiled to the standards, roadmaps, and strategic objectives of the business. On comparing with SA, EA is at senior level. In general, EA takes a strategic, inclusive, and long term view at goals, opportunities, and challenges facing the company. However, SA is assigned to a particular project/program in an enterprise to ensure technical integrity and consistency of the solution at every stage of its life cycle. Role comparison between EA and SA Let me explain the working experiences of two different roles—EA and SA. When I played the SA role for Internet based telephony system, my role needs to build the tools, such as code generation, automation, and so on around the existing telephony system. It needs the skill set of the Microsoft platform technology and telephony domain to understand the existing system in a better way and then provide the better solution to improve the productivity and performance of the existing ecosystem. I was not really involved in the enterprise-level decision making process. Basically, it was pretty much like an individual contributor to build effective and efficient solutions to improvise the current system. As the second work, let me share my experience on the EA role for a leading financial company. The job was to build the enterprise data hub using the emerging big data technology. Degree of comparisons If we plot EA versus SA graphically, EA needs the higher degree of strategy focus and technology breath, as depicted in the following image: In terms of roles and responsibilities, EA and SA differ in their scope. Basically, the SA scope is limited within a project team and the expected delivery is to make the system quality of the solution to the business. In the same time, the EA scope is beyond SA by identifying or envisioning the future state of an organization. Summary In this article, you understood the fundamental concepts of enterprise architecture, and its related business need and benefits. Resources for Article: Further resources on this subject: Getting Started with ASP.NET Core and Bootstrap 4 [Article] Setting Up the Environment for ASP.NET MVC 6 [Article] How to Set Up CoreOS Environment [Article]
Read more
  • 0
  • 0
  • 17848

article-image-decoding-why-good-php-developerisnt-oxymoron
Packt
14 Sep 2016
20 min read
Save for later

Decoding Why "Good PHP Developer"Isn't an Oxymoron

Packt
14 Sep 2016
20 min read
In this article by Junade Ali, author of the book Mastering PHP Design Patterns, we will be revisiting object-oriented programming. Back in 2010 MailChimp published a post on their blog, it was entitled Ewww, You Use PHP? In this blog post they described the horror when they explained their choice of PHP to developers who consider the phrase good PHP programmer an oxymoron. In their rebuttal they argued that their PHP wasn't your grandfathers PHP and they use a sophisticated framework. I tend to judge the quality of PHP on the basis of, not only how it functions, but how secure it is and how it is architected. This book focuses on ideas of how you should architect your code. The design of software allows for developers to ease the extension of the code beyond its original purpose, in a bug free and elegant fashion. (For more resources related to this topic, see here.) As Martin Fowler put it: Any fool can write code that a computer can understand. Good programmers write code that humans can understand. This isn't just limited to code style, but how developers architect and structure their code. I've encountered many developers with their noses constantly stuck in documentation, copying and pasting bits of code until it works; hacking snippets together until it works. Moreover, I far too often see the software development process rapidly deteriorate as developers ever more tightly couple their classes with functions of ever increasing length. Software engineers mustn't just code software; they must know how to design it. Indeed often a good software engineer, when interviewing other software engineers will ask questions surrounding the design of the code itself. It is trivial to get a piece of code that will execute, and it is also benign to question a developer as to whether strtolower or str2lower is the correct name of a function (for the record, it's strtolower). Knowing the difference between a class and an object doesn't make you a competent developer; a better interview question would, for example, be how one could apply subtype polymorphism to a real software development challenge. Failure to assess software design skills dumbs down an interview and results in there being no way to differentiate between those who are good at it, and those who aren't. These advanced topics will be discussed throughout this book, by learning these tactics you will better understand what the right questions to ask are when discussing software architecture. Moxie Marlinspike once tweeted: As a software developer, I envy writers, musicians, and filmmakers. Unlike software, when they create something it is really done, forever. When developing software we mustn't forget we are authors, not just of instructions for a machine, but we are also authoring something that we later expect others to extend upon. Therefore, our code mustn't just be targeted at machines, but humans also. Code isn't just poetry for a machine, it should be poetry for humans also. This is, of course, better said than done. In PHP this may be found especially difficult given the freedom PHP offers developers on how they may architect and structure their code. By the very nature of freedom, it may be both used and abused, so it is true with the freedom offered in PHP. PHP offers freedom to developers to decide how to architect this code. By the very nature of freedom it can be both used and abused, so it is true with the freedom offered in PHP. Therefore, it is increasingly important that developers understand proper software design practices to ensure their code maintains long term maintainability. Indeed, another key skill lies in refactoringcode, improving design of existing code to make it easier to extend in the longer term Technical debt, the eventual consequence of poor system design, is something that I've found comes with the career of a PHP developer. This has been true for me whether it has been dealing with systems that provide advanced functionality or simple websites. It usually arises because a developer elects to implement bad design for a variety of reasons; this is when adding functionality to an existing codebase or taking poor design decisions during the initial construction of software. Refactoring can help us address these issues. SensioLabs (the creators of the Symfonyframework) have a tool called Insight that allows developers to calculate the technical debt in their own code. In 2011 they did an evaluation of technical debt in various projects using this tool; rather unsurprisingly they found that WordPress 4.1 topped the chart of all platforms they evaluated with them claiming it would take 20.1 years to resolve the technical debt that the project contains. Those familiar with the WordPress core may not be surprised by this, but this issue of course is not only associated to WordPress. In my career of working with PHP, from working with security critical cryptography systems to working with systems that work with mission critical embedded systems, dealing with technical debt comes with the job. Dealing with technical debt is not something to be ashamed of for a PHP Developer, indeed some may consider it courageous. Dealing with technical debt is no easy task, especially in the face of an ever more demanding user base, client, or project manager; constantly demanding more functionality without being familiar with the technical debt the project has associated to it. I recently emailed the PHP Internals group as to whether they should consider deprecating the error suppression operator @. When any PHP function is prepended by an @ symbol, the function will suppress an error returned by it. This can be brutal; especially where that function renders a fatal error that stops the execution of the script, making debugging a tough task. If the error is suppressed, the script may fail to execute without providing developers a reason as to why this is. Despite the fact that no one objected to the fact that there were better ways of handling errors (try/catch, proper validation) than abusing the error suppression operator and that deprecation should be an eventual aim of PHP, it is the case that some functions return needless warnings even though they already have a success/failure value. This means that due to technical debt in the PHP core itself, this operator cannot be deprecated until a lot of other prerequisite work is done. In the meantime, it is down to developers to decide the best methodologies of handling errors. Until the inherent problem of unnecessary error reporting is addressed, this operator cannot be deprecated. Therefore, it is down to developers to be educated as to the proper methodologies that should be used to address error handling and not to constantly resort to using an @ symbol. Fundamentally, technical debt slows down development of a project and often leads to code being deployed that is broken as developers try and work on a fragile project. When starting a new project, never be afraid to discus architecture as architecture meetings are vital to developer collaboration; as one scrum master I've worked with said in the face of criticism that "meetings are a great alternative to work", he said "meetings are work…how much work would you be doing without meetings?". Coding style - thePSR standards When it comes to coding style, I would like to introduce you to the PSR standards created by the PHP Framework Interop Group. Namely, the two standards that apply to coding standards are PSR-1 (Basic Coding Style) and PSR-2 (Coding Style Guide). In addition to this there are PSR standards that cover additional areas, for example, as of today; the PSR-4 standard is the most up-to-date autoloading standard published by the group. You can find out more about the standards at http://www.php-fig.org/. Coding style being used to enforce consistency throughout a codebase is something I strongly believe in, it does make a difference to your code readability throughout a project. It is especially important when you are starting a project (chances are you may be reading this book to find out how to do that right) as your coding style determines the style the developers following you in working on this project will adopt. Using a global standard such as PSR-1 or PSR-2 means that developers can easily switch between projects without having to reconfigure their code style in their IDE. Good code style can make formatting errors easier to spot. Needless to say that coding styles will develop as time progresses, to date I elect to work with the PSR standards. I am a strong believer in the phrase: Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. It isn't known who wrote this phrase originally; but it's widely thought that it could have been John Woods or potentially Martin Golding. I would strongly recommend familiarizingyourself with these standards before proceeding in this book. Revising object-oriented programming Object-oriented programming is more than just classes and objects, it's a whole programming paradigm based around objects(data structures) that contain data fields and methods. It is essential to understand this; using classes to organize a bunch of unrelated methods together is not object orientation. Assuming you're aware of classes (and how to instantiate them), allow me to remind you of a few different bits and pieces. Polymorphism Polymorphism is a fairly long word for a fairly simple concept. Essentially, polymorphism means the same interfaceis used with a different underlying code. So multiple classes could have a draw function, each accepting the same arguments, but at an underlying level the code is implemented differently. In this article, I would also like to talk about Subtype Polymorphism in particular (also known as Subtyping or Inclusion Polymorphism). Let's say we have animals as our supertype;our subtypes may well be cats, dogs, and sheep. In PHP, interfaces allow you to define a set of functionality that a class that implements it must contain, as of PHP 7 you can also use scalar type hints to define the return types we expect. So for example, suppose we defined the following interface: interface Animal { public function eat(string $food) : bool; public function talk(bool $shout) : string; } We could then implement this interface in our own class, as follows: class Cat implements Animal { } If we were to run this code without defining the classes we would get an error message as follows: Class Cat contains 2 abstract methods and must therefore be declared abstract or implement the remaining methods (Animal::eat, Animal::talk) Essentially, we are required to implement the methods we defined in our interface, so now let's go ahead and create a class that implements these methods: class Cat implements Animal { public function eat(string $food): bool { if ($food === "tuna") { return true; } else { return false; } } public function talk(bool $shout): string { if ($shout === true) { return "MEOW!"; } else { return "Meow."; } } } Now that we've implemented these methods we can then just instantiate the class we are after and use the functions contained in it: $felix = new Cat(); echo $felix->talk(false); So where does polymorphism come into this? Suppose we had another class for a dog: class Dog implements Animal { public function eat(string $food): bool { if (($food === "dog food") || ($food === "meat")) { return true; } else { return false; } } public function talk(bool $shout): string { if ($shout === true) { return "WOOF!"; } else { return "Woof woof."; } } } Now let's suppose we have multiple different types of animals in a pets array: $pets = array( 'felix' => new Cat(), 'oscar' => new Dog(), 'snowflake' => new Cat() ); We can now actually go ahead and loop through all these pets individually in order to run the talk function.We don't care about the type of pet because the talkmethod that is implemented in every class we getis by virtue of us having extended the Animals interface. So let's suppose we wanted to have all our animals run the talk method, we could just use the following code: foreach ($pets as $pet) { echo $pet->talk(false); } No need for unnecessary switch/case blocks in order to wrap around our classes, we just use software design to make things easier for us in the long-term. Abstract classes work in a similar way, except for the fact that abstract classes can contain functionality where interfaces cannot. It is important to note that any class that defines one or more abstract classes must also be defined as abstract. You cannot have a normal class defining abstract methods, but you can have normal methods in abstract classes. Let's start off by refactoring our interface to be an abstract class: abstract class Animal { abstract public function eat(string $food) : bool; abstract public function talk(bool $shout) : string; public function walk(int $speed): bool { if ($speed > 0) { return true; } else { return false; } } } You might have noticed that I have also added a walk method as an ordinary, non-abstract method; this is a standard method that can be used or extended by any classes that inherit the parent abstract class. They already have implementation. Note that it is impossible to instantiate an abstract class (much like it's not possible to instantiate an interface). Instead we must extend it. So, in our Cat class let's substitute: class Cat implements Animal With the following code: class Cat extends Animal That's all we need to refactor in order to get classes to extend the Animal abstract class. We must implement the abstract functions in the classes as we outlined for the interfaces, plus we can use the ordinary functions without needing to implement them: $whiskers = new Cat(); $whiskers->walk(1); As of PHP 5.4 it has also become possible to instantiate a class and access a property of it in one system. PHP.net advertised it as: Class member access on instantiation has been added, e.g. (new Foo)->bar(). You can also do it with individual properties, for example,(new Cat)->legs. In our example, we can use it as follows: (new IcyAprilChapterOneCat())->walk(1); Just to recap a few other points about how PHP implemented OOP, the final keyword before a class declaration or indeed a function declaration means that you cannot override such classes or functions after they've been defined. So, if we were to try extending a class we have named as final: final class Animal { public function walk() { return "walking..."; } } class Cat extends Animal { } This results in the following output: Fatal error: Class Cat may not inherit from final class (Animal) Similarly, if we were to do the same except at a function level: class Animal { final public function walk() { return "walking..."; } } class Cat extends Animal { public function walk () { return "walking with tail wagging..."; } } This results in the following output: Fatal error: Cannot override final method Animal::walk() Traits (multiple inheritance) Traits were introduced into PHP as a mechanism for introducing Horizontal Reuse. PHP conventionally acts as a single inheritance language, namely because of the fact that you can't inherit more than one class into a script. Traditional multiple inheritance is a controversial process that is often looked down upon by software engineers. Let me give you an example of using Traits first hand; let's define an abstract Animal class which we want to extend into another class: class Animal { public function walk() { return "walking..."; } } class Cat extends Animal { public function walk () { return "walking with tail wagging..."; } } So now let's suppose we have a function to name our class, but we don't want it to apply to all our classes that extend the Animal class, we want it to apply to certain classes irrespective of whether they inherit the properties of the abstract Animal class or not. So we've defined our functions like so: function setFirstName(string $name): bool { $this->firstName = $name; return true; } function setLastName(string $name): bool { $this->lastName = $name; return true; } The problem now is that there is no place we can put them without using Horizontal Reuse, apart from copying and pasting different bits of code or resorting to using conditional inheritance. This is where Traits come to the rescue; let's start off by wrapping these methods in a Trait called Name: trait Name { function setFirstName(string $name): bool { $this->firstName = $name; return true; } function setLastName(string $name): bool { $this->lastName = $name; return true; } } So now that we've defined our Trait, we can just tell PHP to use it in our Cat class: class Cat extends Animal { use Name; public function walk() { return "walking with tail wagging..."; } } Notice the use of theName statement? That's where the magic happens. Now you can call the functions in that Trait without any problems: $whiskers = new Cat(); $whiskers->setFirstName('Paul'); echo $whiskers->firstName; All put together, the new code block looks as follows: trait Name { function setFirstName(string $name): bool { $this->firstName = $name; return true; } function setLastName(string $name): bool { $this->lastName = $name; return true; } } class Animal { public function walk() { return "walking..."; } } class Cat extends Animal { use Name; public function walk() { return "walking with tail wagging..."; } } $whiskers = new Cat(); $whiskers->setFirstName('Paul'); echo $whiskers->firstName; Scalar type hints Let me take this opportunity to introduce you to a PHP7 concept known as scalar type hinting; it allows you to define the return types (yes, I know this isn't strictly under the scope of OOP; deal with it). Let's define a function, as follows: function addNumbers (int $a, int $b): int { return $a + $b; } Let's take a look at this function; firstly you will notice that before each of the arguments we define the type of variable we want to receive, in this case,int or integer. Next up you'll notice there's a bit of code after the function definition : int, which defines our return type so our function can only receive an integer. If you don't provide the right type of variable as a function argument or don't return the right type of variable from the function; you will get a TypeError exception. In strict mode, PHP will also throw a TypeError exception in the event that strict mode is enabled and you also provide the incorrect number of arguments. It is also possible in PHP to define strict_types; let me explain why you might want to do this. Without strict_types, PHP will attempt to automatically convert a variable to the defined type in very limited circumstances. For example, if you pass a string containing solely numbers it will be converted to an integer, a string that's non-numeric, however, will result in a TypeError exception. Once you enable strict_typesthis all changes, you can no longer have this automatic casting behavior. Taking our previous example, without strict_types, you could do the following: echo addNumbers(5, "5.0"); Trying it again after enablingstrict_types, you will find that PHP throws a TypeError exception. This configuration only applies on an individual file basis, putting it before you include other files will not result in this configuration being inherited to those files. There are multiple benefits of why PHP chose to go down this route; they are listed very clearly in Version: 0.5.3 of the RFC that implemented scalar type hints called PHP RFC: Scalar Type Declarations. You can read about it by going to http://www.wiki.php.net (the wiki, not the main PHP website) and searching for scalar_type_hints_v5. In order to enable it, make sure you put this as the very first statement in your PHP script: declare(strict_types=1); This will not work unless you define strict_typesas the very first statement in a PHP script; no other usages of this definition are permitted. Indeed if you try to define it later on, your script PHP will throw a fatal error. Of course, in the interests of the rage induced PHP core fanatic reading this book in its coffee stained form, I should mention that there are other valid types that can be used in type hinting. For example, PHP 5.1.0 introduced this with arrays and PHP 5.0.0 introduced the ability for a developer to do this with their own classes. Let me give you a quick example of how this would work in practice, suppose we had an Address class: class Address { public $firstLine; public $postcode; public $country; public function __construct(string $firstLine, string $postcode, string $country) { $this->firstLine = $firstLine; $this->postcode = $postcode; $this->country = $country; } } We can then type the hint of the Address class that we inject into a Customer class: class Customer { public $name; public $address; public function __construct($name, Address $address) { $this->name = $name; $this->address = $address; } } And just to show how it all can come together: $address = new Address('10 Downing Street', 'SW1A2AA', 'UK'); $customer = new Customer('Davey Cameron', $address); var_dump($customer); Limiting debug access to private/protected properties If you define a class which contains private or protected variables, you will notice an odd behavior if you were to var_dumpthe object of that class. You will notice that when you wrap the object in a var_dumpit reveals all variables; be they protected, private, or public. PHP treats var_dump as an internal debugging function, meaning all data becomes visible. Fortunately, there is a workaround for this. PHP 5.6 introduced the __debugInfo magic method. Functions in classes preceded by a double underscore represent magic methods and have special functionality associated to them. Every time you try to var_dump an object that has the __debugInfo magic method set, the var_dump will be overridden with the result of that function call instead. Let me show you how this works in practice, let's start by defining a class: class Bear { private $hasPaws = true; } Let's instantiate this class: $richard = new Bear(); Now if we were to try and access the private variable that ishasPaws, we would get a fatal error; so this call: echo $richard->hasPaws; Would result in the following fatal error being thrown: Fatal error: Cannot access private property Bear::$hasPaws That is the expected output, we don't want a private property visible outside its object. That being said, if we wrap the object with a var_dump as follows: var_dump($richard); We would then get the following output: object(Bear)#1 (1) { ["hasPaws":"Bear":private]=> bool(true) } As you can see, our private property is marked as private, but nevertheless it is visible. So how would we go about preventing this? So, let's redefine our class as follows: class Bear { private $hasPaws = true; public function __debugInfo () { return call_user_func('get_object_vars', $this); } } Now, after we instantiate our class and var_dump the resulting object, we get the following output: object(Bear)#1 (0) { } The script all put together looks like this now, you will notice I've added an extra public property called growls, which I have set to true: <?php class Bear { private $hasPaws = true; public $growls = true; public function __debugInfo () { return call_user_func('get_object_vars', $this); } } $richard = new Bear(); var_dump($richard); If we were to var_dump this script (with both public and private property to play with), we would get the following output: object(Bear)#1 (1) { ["growls"]=> bool(true) } As you can see, only the public property is visible. So what is the moral of the story from this little experiment? Firstly, that var_dumps exposesprivate and protected properties inside objects, and secondly, that this behavior can be overridden. Summary In this article, we revised some PHP principles, including OOP principles. We also revised some PHP syntax basics. Resources for Article: Further resources on this subject: Running Simpletest and PHPUnit [article] Data Tables and DataTables Plugin in jQuery 1.3 with PHP [article] Understanding PHP basics [article]
Read more
  • 0
  • 0
  • 17843

article-image-examining-encodingjson-package-go
Packt
28 Dec 2016
13 min read
Save for later

Examining the encoding/json Package with Go

Packt
28 Dec 2016
13 min read
In this article by Nic Jackson, author of the book Building Microservices with Go, we will examine the encoding/json package to see just how easy Go makes it for us to use JSON objects for our requests and responses. (For more resources related to this topic, see here.) Reading and writing JSON Thanks to the encoding/json package, which is built into the standard library, encoding and decoding JSON to and from Go types is both fast and easy. It implements the simplistic Marshal and Unmarshal functions; however, if we need them, the package also provides Encoder and Decoder types, which allow us greater control when reading and writing streams of JSON data. In this section, we are going to examine both of these approaches, but first let's take a look at how simple it is to convert a standard Go struct into its corresponding JSON string. Marshalling Go structs to JSON To encode JSON data, the encoding/json package provides the Marshal function, which has the following signature: func Marshal(v interface{}) ([]byte, error) This function takes one parameter, which is of the interface type, so that's pretty much any object you can think of, since interface represents any type in Go. It returns a tuple of ([]byte, error). You will see this return style quite frequently in Go. Some languages implement a try...catch approach, which encourages an error to be thrown when an operation cannot be performed. Go suggests the (return type, error) pattern, where the error is nil when an operation succeeds. In Go, unhanded errors are a bad thing, and while the language does implement the panic and recover functions, which resemble exception handling in other languages, the situations in which you should use them are quite different (The Go Programming Language, Donovan and Kernighan). In Go, panic causes normal execution to stop, and all deferred function calls in the Go routine are executed; the program will then crash with a log message. It is generally used for unexpected errors that indicate a bug in the code, and good, robust Go code will attempt to handle these runtime exceptions and return a detailed error object back to the calling function. This pattern is exactly what is implemented with the Marshal function. In case Marshal cannot create a JSON-encoded byte array from the given object, which could be due to a runtime panic, then this is captured and an error object detailing the problem is returned to the caller. Let's try this out, expanding on our existing example. Instead of simply printing a string from our handler, let's create a simple struct for the response and return that: 10 type helloWorldResponse struct { 11 Message string 12 } In our handler, we will create an instance of this object, set the message, and then use the Marshal function to encode it to a string before returning. Let's see what that will look like: 23 func helloWorldHandler(w http.ResponseWriter, r *http.Request) { 24 response := helloWorldResponse{Message: "HelloWorld"} 25 data, err := json.Marshal(response) 26 if err != nil { 27 panic("Ooops") 28 } 29 30 fmt.Fprint(w, string(data)) 31 } Now when we rerun our program and refresh our browser, we'll see the following output rendered in valid JSON: {"Message":"Hello World"} This is awesome, but the default behavior of Marshal is to take the literal name of the field and use that as the field in the JSON output. What if I prefer to use camel case and would rather see message—could we just rename the field in our struct message? Unfortunately, we can't because in Go, lowercase properties are not exported. Marshal will ignore these and will not include them in the output. All is not lost: the encoding/json package implements struct field attributes, which allow us to change the output for the property to anything we choose. The example code is as follows: 10 type helloWorldResponse struct { 11 Message string `json:"message"` 12 } Using the struct field's tags, we can have greater control over how the output will look. In the preceding example, when we marshal this struct, the output from our server would be the following: {"message":"Hello World"} This is exactly what we want, but we can use field tags to control the output even further. We can convert object types and even ignore a field altogether if we need to: struct helloWorldResponse { // change the output field to be "message" Message string `json:"message"` // do not output this field Author string `json:"-"` // do not output the field if the value is empty Date string `json:",omitempty"` // convert output to a string and rename "id" Id int `json:"id, string"` } The channel, complex types, and functions cannot be encoded in JSON. Attempting to encode these types will result in an UnsupportedTypeError being returned by the Marshal function. It also can't represent cyclic data structures, so if your stuct contains a circular reference, then Marshal will result in an infinite recursion, which is never a good thing for a web request. If we want to export our JSON pretty formatted with indentation, we can use the MarshallIndent function, which allows you to pass an additional string parameter to specify what you would like the indent to be—two spaces, not a tab, right? func MarshalIndent(v interface{}, prefix, indent string) ([]byte, error) The astute reader might have noticed that we are decoding our struct into a byte array and then writing that to the response stream. This does not seem to be particularly efficient, and in fact, it is not. Go provides encoders and decoders, which can write directly to a stream. Since we already have a stream with the ResponseWriter interface, let's do just that. Before we do so, I think we need to look at the ResponseWriter interface a little to see what is going on there. ResponseWriter is an interface that defines three methods: // Returns the map of headers which will be sent by the // WriteHeader method. Header() // Writes the data to the connection. If WriteHeader has not // already been called then Write will call // WriteHeader(http.StatusOK). Write([]byte) (int, error) // Sends an HTTP response header with the status code. WriteHeader(int)   If we have a ResponseWriter, how can we use this with fmt.Fprint(w io.Writer, a ...interface{})? This method requires a Writer interface as a parameter, and we have a ResponseWriter. If we look at the signature for Writer, we can see that it is the following: Write(p []byte) (n int, err error) Because the ResponseWriter interface implements this method, it also satisfies the Writer interface; therefore, any object that implements ResponseWriter can be passed to any function that expects Writer. Amazing! Go rocks—but we don't have an answer to our question: is there any better way to send our data to the output stream without marshalling to a temporary string before we return it? The encoding/json package has a function called NewEncoder. This returns an Encoder object, which can be used to write JSON straight to an open writer, and guess what—we have one of those: func NewEncoder(w io.Writer) *Encoder So instead of storing the output of Marshal into a byte array, we can write it straight to the HTTP response, as shown in the following code: func helloWorldHandler(w http.ResponseWriter, r *http.Request) { response := HelloWorldResponse{Message: "HelloWorld"} encoder := json.NewEncoder(w) encoder.Encode(&response) } We will look at benchmarking in a later chapter, but to see why this is important, here's a simple benchmark to check the two methods against each other; have a look at the output: go test -v -run="none" -bench=. -benchtime="5s" -benchmem testing: warning: no tests to run PASS BenchmarkHelloHandlerVariable 10000000 1211 ns/op 248 B/op 5 allocs/op BenchmarkHelloHandlerEncoder 10000000 662 ns/op 8 B/op 1 allocs/op ok github.com/nicholasjackson/building-microservices-in-go/chapter1/bench 20.650s Using the Encoder rather than marshalling to a byte array is nearly 50% faster. We are dealing with nanoseconds here, so that time may seem irrelevant, but it isn't; this was two lines of code. If you have that level of inefficiency throughout the rest of your code, your application will run slower, you will need more hardware to satisfy the load, and that will cost you money. There is nothing clever in the differences between the two methods—all we have done is understood how the standard packages work and chosen the correct option for our requirements. That is not performance tuning, that is understanding the framework. Unmarshalling JSON to Go structs Now that we have learned how we can send JSON back to the client, what if we need to read input before returning the output? We could use URL parameters, and we will see what that is all about in the next chapter, but usually, you will need more complex data structures, which include the service to accept JSON as part of an HTTP POST request. If we apply techniques similar to those we learned in the previous section (to write JSON), reading JSON is just as easy. To decode JSON into a stuct, the encoding/json package provides us with the Unmarshal function: func Unmarshal(data []byte, v interface{}) error The Unmarshal function works in the opposite way to Marshal: it allocates maps, slices, and pointers as required. Incoming object keys are matched using either the struct field name or its tag and will work with a case-insensitive match; however, an exact match is preferred. Like Marshal, Unmarshal will only set exported struct fields: those that start with an upper case letter. We start by adding a new struct to represent the request, while Unmarshal can decode the JSON into an interface{} array, which would be of one of the following types: map[string]interface{} // for JSON objects []interface{} // for JSON arrays Which type it is depends on whether our JSON is an object or an array. In my opinion, it is much clearer to the readers of our code if we explicitly state what we are expecting as a request. We can also save ourselves work by not having to manually cast the data when we come to use it. Remember two things: You do not write code for the compiler; you write code for humans to understand You will spend more time reading code than you do writing it We are going to do ourselves a favor by taking into account these two points and creating a simple struct to represent our request, which will look like this: 14 type helloWorldRequest struct { 15 Name string `json:"name"` 16 } Again, we are going to use struct field tags because while we could let Unmarshal do case-insensitive matching so that {"name": "World} would correctly unmarshal into the struct the same as {"Name": "World"}, when we specify a tag, we are being explicit about the request form, and that is a good thing. In terms of speed and performance, it is also about 10% faster, and remember: performance matters. To access the JSON sent with the request, we need to take a look at the http.Request object passed to our handler. The following listing does not show all the methods in the request, just the ones we are going to be immediately dealing with. For the full documentation, I recommend checking out the docs at https://godoc.org/net/http#Request. type Requests struct { … // Method specifies the HTTP method (GET, POST, PUT, etc.). Method string // Header contains the request header fields received by the server. The type Header is a link to map[string] []string. Header Header // Body is the request's body. Body io.ReadCloser … } The JSON that has been sent with the request is accessible in the Body field. The Body field implements the io.ReadCloser interface as a stream and does not return []byte or string data. If we need the data contained in the body, we can simply read it into a byte array, like the following example: 30 body, err := ioutil.ReadAll(r.Body) 31 if err != nil { 32 http.Error(w, "Bad request", http.StatusBadRequest) 33 return 34 } Here is something we'll need to remember: we are not calling Body.Close(); if we were making a call with a client, we would need to do this as it is not automatically closed; however, when used in a ServeHTTP handler, the server automatically closes the request stream. To see how this all works inside our handler, we can look at the following handler: 28 func helloWorldHandler(w http.ResponseWriter, r *http.Request) { 29 30 body, err := ioutil.ReadAll(r.Body) 31 if err != nil { 32 http.Error(w, "Bad request", http.StatusBadRequest) 33 return 34 } 35 36 var request HelloWorldRequest 37 err = json.Unmarshal(body, &request) 38 if err != nil { 39 http.Error(w, "Bad request", http.StatusBadRequest) 40 return 41 } 42 43 response := HelloWorldResponse{Message: "Hello " + request.Name} 44 45 encoder := json.NewEncoder(w) 46 encoder.Encode(response) 47 } Let's run this example and see how it works; to test it, we can simply use curl to send a request to the running server. If you feel more comfortable using a GUI tool, then Postman, which is available for the Google Chrome browser, will work just fine. Otherwise, feel free to use your preferred tool. $ curl localhost:8080/helloworld -d '{"name":"Nic"}' You should see the following response: {"message":"Hello Nic"} What do you think will happen if you do not include a body with your request? $ curl localhost:8080/helloworld If you guessed correctly that you would get an "HTTP status 400 Bad Request" error, then you win a prize. The following error replies to the request with the given message and status code: func Error(w ResponseWriter, error string, code int) Once we have sent this, we need to return, stopping further execution of the function as this does not close the ResponseWriter and return flow to the calling function automatically. You might think you are done, but have a go and see whether you can improve the performance of the handler. Think about the things we were talking about when marshaling JSON. Got it? Well, if not, here is the answer: again, all we are doing is using Decoder, which is the opposite of the Encoder function we used when writing JSON, as shown in the following code example. This nets an instant 33% performance increase, and with less code, too. 27 func helloWorldHandler(w http.ResponseWriter, r *http.Request) { 28 29 var request HelloWorldRequest 30 decoder := json.NewDecoder(r.Body) 31 32 err := decoder.Decode(&request) 33 if err != nil { 34 http.Error(w, "Bad request", http.StatusBadRequest) 35 return 36 } 37 38 response := HelloWorldResponse{Message: "Hello " + request.Name} 39 40 encoder := json.NewEncoder(w) 41 encoder.Encode(response) 42 } Now that you can see just how easy it is to encode and decode JSON with Go, I would recommend taking 5 minutes to spend some time digging through the documentation for the encoding/json package as there is a whole lot more than you can do with it: https://golang.org/pkg/encoding/json/ Summary In this article, we looked at encoding and decoding data using the encoding/json package. Resources for Article: Further resources on this subject: Microservices – Brave New World [article] A capability model for microservices [article] Breaking into Microservices Architecture [article]
Read more
  • 0
  • 0
  • 17755

article-image-is-mozilla-the-most-progressive-tech-organization-on-the-planet-right-now
Richard Gall
16 Oct 2018
7 min read
Save for later

Is Mozilla the most progressive tech organization on the planet right now?

Richard Gall
16 Oct 2018
7 min read
2018, according to The Economist, has been the year of the techlash. scandals, protests, resignations, congressional testimonies - many of the largest companies in the world have been in the proverbial dock for a distinct lack of accountability. Together, these stories have created a narrative where many are starting to question the benefits of unbridled innovation. But Mozilla is one company that seems to have bucked that trend. In recent weeks there have been a series of news stories that suggest Mozilla is a company thinking differently about its place in the world, as well as the wider challenges technology poses society. All of these come together to present Mozilla in a new light. Cynics might suggest that much of this is little more than some smart PR work, but it's a little unfair to dismiss what some impressive work. So much has been happening across the industry that deserves scepticism at best and opprobrium at worst. To see a tech company stand out from the tiresome pattern of stories this year can only be a good thing. Mozilla on education: technology, ethical code, and the humanities Code ethics has become a big topic of conversation in 2018. And rightly so - with innovation happening at an alarming pace, it has become easy to make the mistake of viewing technology as a replacement for human agency, rather than something that emerges from it. When we talk about code ethics it reminds us that technology is something built from the decisions and actions of thousands of different people. It’s for this reason that last week’s news that Mozilla has teamed up with a number of organizations, including the Omidyar Network to announce a brand new prize for computer science students feels so important. At a time when the likes of Mark Zuckerberg dance around any notion of accountability, peddling a narrative where everything is just a little bit beyond Facebook’s orbit of control, the ‘Responsible Computer Science Challenge’ stands out. With $3.5 million up for grabs for smart computer science students, it’s evidence that Mozilla is putting its money where its mouth is and making ethical decision making something which, for once, actually pays. Mitchell Baker on the humanities and technology Mitchell Baker’s comments to the Guardian that accompanied the news also demonstrate a refreshingly honest perspective from a tech leader. “One thing that’s happened in 2018,” Baker said, “is that we’ve looked at the platforms, and the thinking behind the platforms, and the lack of focus on impact or result. It crystallised for me that if we have STEM education without the humanities, or without ethics, or without understanding human behaviour, then we are intentionally building the next generation of technologists who have not even the framework or the education or vocabulary to think about the relationship of STEM to society or humans or life.” Baker isn’t, however, a crypto-luddite or an elitist that wants full stack developer classicists. Instead she’s looking forward at the ways in which different disciplines can interact and inform one another. It’s arguably an intellectual form of DevOps. It is a way of bridging the gap between STEM skills and practices, and those rooted in the tradition of the humanities. The significance of this intervention shouldn’t be understated. It opens up a dialogue within society and the tech industry that might get us to a place where ethics is simply part and parcel of what it means to build and design software, not an optional extra. Mozilla’s approach to internal diversity: dropping meritocracy The respective cultures of organizations and communities across tech has been in the spotlight over the last few months. Witness the bitter furore over Linux change to its community guidelines to see just how important definitions and guidelines are to the people within them. That’s why Mozilla’s move to drop meritocracy from its guidelines of governance and leadership structures was a small yet significant move. It’s simply another statement of intent from a company eager to ensure it helps develop a culture more open and inclusive than the tech world has been over the last thirty decades. In a post published on the Mozilla blog at the start of October, Emma Irwin (D&I Strategy, Mozilla Contributors and Communities) and Larissa Shapiro (Head of Global Diversity & Inclusion at Mozilla) wrote that “Meritocracy does not consider the reality that tech does not operate on a level playing field.” The new governance proposal actually reflects Mozilla’s apparent progressiveness pretty well. In it, it states that “the project also seeks to debias this system of distributing authority through active interventions that engage and encourage participation from diverse communities.” While there has been some criticism of the change, it’s important to note that the words used by organizations of this size does have an impact on how we frame and focus problems. From this perspective, Mozilla’s decision could well be a vital small step in making tech more accessible and diverse. The tech world needs to engage with political decision makers Mozilla isn't just a 'progressive' tech company because of the content of its political beliefs. Instead, what's particularly important is how it appears to recognise that the problems that technology faces and engages with are, in fact, much bigger than technology itself. Just consider the actions of other tech leaders this year. Sundar Pichai didn't attend his congressional hearing, Jack Dorsey assured us that Twitter has safety at its heart while verifying neo-Nazis, while Mark Zuckerberg suggested that AI can fix the problems of election interference and fake news. The hubris has been staggering. Mozilla's leadership appears to be trying hard to avoid the same pitfalls. We shouldn’t be surprised that Mozilla actually embraced the idea of 2018’s ‘techlash.' The organization used the term in the title of a post directed at G20 leaders in August. Written alongside The Internet Society and the Web Foundation, it urged global leaders to “reinject hope back into technological innovation.” Implicit in the post is an acknowledgement that the aims and goals of much of the tech industry - improving people’s lives, making infrastructure more efficient - can’t be purely solved by the industry itself. It is a subtle stab at what might be considered hubris. Taking on government and regulation But this isn’t to say Mozilla is completely in thrall to government and regulation. Most recently (16 October), Mozilla voiced its concerns about current decryption laws being debated in Australian Parliament. The organization was clear, saying "this is at odds with the core principles of open source, user expectations, and potentially contractual license obligations.” At the beginning of September Mozilla also spoke out against EU copyright reform. The organization argued that “article 13 will be used to restrict the freedom of expression and creative potential of independent artists who depend upon online services to directly reach their audience and bypass the rigidities and limitations of the commercial content industry.”# While opposition to EU copyright reform came from a range of voices - including those huge corporations that have come under scrutiny during the ‘techlash’ - Mozilla is, at least, consistent. The key takeaway from Mozilla: let’s learn the lessons of 2018’s techlash The techlash has undoubtedly caused a lot of pain for many this year. But the worst thing that could happen is for the tech industry to fail to learn the lessons that are emerging. Mozilla deserve credit for trying hard to properly understand the implications of what’s been happening and develop a deliberate vision for how to move forward.
Read more
  • 0
  • 0
  • 17739
article-image-enabling-and-configuring-snmp-windows
Packt
14 Nov 2013
5 min read
Save for later

Enabling and configuring SNMP on Windows

Packt
14 Nov 2013
5 min read
This article by Justin M. Brant, the author of SolarWinds Server & Application Monitor: Deployment and Administration, covers enabling and configuring SNMP on Windows. (For more resources related to this topic, see here.) Procedures in this article are not required pre-deployment, as it is possible after deployment to populate SolarWinds SAM with nodes; however, it is recommended. Even after deployment, you should still enable and configure advanced monitoring services on your vital nodes. SolarWinds SAM uses three types of protocols to poll management data: Simple Network Management Protocol (SNMP): This is the most common network management service protocol. To utilize it, SNMP must be enabled and an SNMP community string must be assigned on the server, device, or application. The community string is essentially a password that is sent between a node and SolarWinds SAM. Once the community string is set and assigned, the node is permitted to expose management data to SolarWinds SAM, in the form of variables. Currently, there are three versions of SNMP: v1, v2c, and v3. SolarWinds SAM uses SNMPv2c by default. To poll using SNMPv1, you must disable SNMPv2c on the device. Similarly, to poll using SNMPv3, you must configure your devices and SolarWinds SAM accordingly. Windows Management Instrumentation (WMI): This has added functionality by incorporating Windows specifi c communications and security features. WMI comes preinstalled on Windows by default but is not automatically enabled and confi gured. WMI is not exclusive to Windows server platforms; it comes installed on all modern Microsoft operating systems, and can also be used to poll desktop operating systems, such as Windows 7. Internet Control Message Protocol (ICMP): This is the most basic of the three; it simply sends echo requests (pings) to a server or device for status, response time, and packet loss. SolarWinds SAM uses ICMP in conjunction with SNMP and WMI. Nodes can be confi gured to poll with ICMP exclusively, but you miss out on CPU, memory, and volume data. Some devices can only be polled with ICMP, although in most instances you will rarely use ICMP exclusively. Trying to decide between SNMP and WMI? SNMP is more standardized and provides data that you may not be able to poll with WMI, such as interface information. In addition, polling a single WMI-enabled node uses roughly five times the resources required to poll the same node with SNMP. This article will explain how to prepare for SolarWinds SAM deployment, by enabling and configuring network management services and protocols on: Windows servers. In this article we will reference service accounts. A service account is an account created to handoff credentials to SolarWinds SAM. Service accounts are a best practice primarily for security reasons, but also to ensure that user accounts do not become locked out. Enabling and configuring SNMP on Windows Procedures listed in this article will explain how to enable SNMP and then assign a community string, on Windows Server 2008 R2. All Windows server-related procedures in this book are performed on Windows Server 2008 R2. Procedures vary slightly in other supported versions. Installing an SNMP service on Windows This procedure explains how to install the SNMP service on Windows Server 2008 R2. Log in to a Windows server. Navigate to Start Menu | Control Panel | Administrative Tools | Server Manager. In order to see Administrative Tools in the Control Panel, you may need to select View by: Small Icons or Large Icons. Select Features and click on Add Features. Check SNMP Services, then click on Next and Install. Click on Close. Assigning an SNMP community string on Windows This procedure explains how to assign a community string on Windows 2008 R2, and ensure that the SNMP service is configured to run automatically on start up. Log in to a Windows server. Navigate to Start Menu | Control Panel | Administrative Tools | Services. Double-click on SNMP Service. On the General tab, select Automatic under Startup type. Select the Agent tab and ensure Physical, Applications, Internet, and End-to-end are all checked under the Service area. Optionally, enter a Contact person and system Location. Select the Security tab and click on the Add button under Accepted community names. Enter a Community Name and click on the Add button. For example, we used S4MS3rv3r. We recommend using something secure, as this is a password. Community String and Community Name mean the same thing.   READ ONLY community rights will normally suffice. A detailed explanation of community rights can be found on the author's blog: http://justinmbrant.blogspot.com/ Next, tick the Accept SNMP packets from these hosts radio button. Click on the Add button underneath the radio buttons and add the IP of the server you have designated as the SolarWinds SAM host. Once you complete these steps, the SNMP Service Properties Security tab should look something like the following screenshot. Notice that we used 192.168.1.3, as that is the IP of the server where we plan to deploy SolarWinds SAM. Summary In this article, we learned different types of protocols to poll management data. We also learned how to install SNMP as well as assign SNMP community string on Windows. Resources for Article: Further resources on this subject: The OpenFlow Controllers [Article] The GNS3 orchestra [Article] Network Monitoring Essentials [Article]
Read more
  • 0
  • 0
  • 17736

article-image-11-predictions-for-the-future-of-programming
Guest Contributor
19 Nov 2018
12 min read
Save for later

11 predictions for the future of programming

Guest Contributor
19 Nov 2018
12 min read
It’s been over five decades since programming pushed the boundaries of digital craftsmanship, and it is still doing so with no signs of stopping or slowing down. There is a new tool, framework, add-on, functionality, technology, or a programming language breaking the Internet every now and then. Any adept programmer not only needs to be good at coding but also has to stay abreast with the ongoing and upcoming happenings in the programming world. Just learning to code does not a give you a big edge over others. By having a good idea of what’s coming ahead,  present steps can be planned effectively. Obviously, no one can perfectly forecast the future of computer programming, but that won’t stop us from speculating, right! Here are 11  predictions for the future of programming that we think programmers should keep an eye on. #1 Cloud native as the new default Do you know that in order to cater to a single search query, Google Search uses more than 1000s of servers? All this is done in order to serve the right results. Cloud has been popular for past one decade but it’s destined to grow immensely in the future as more and more developers intend to use cloud for faster go to market. Tinkering in the cloud to build an app is so much easier as compared to managing your own servers as you don’t have to buy new servers, maintain them, upgrade them, or add new servers as and when the demand fluctuates. Web users are an impatient lot these days; so making web pages faster is the main goal for developers. 40% of people abandon a website that takes more than 3 seconds to load. More efficient algorithms save a few microseconds whereas additional impetus is provided by the rapidly developing enhanced servers. #2 IoT security concerns will escalate IoT is a growing technological concept these days. The promising piece of tech has already made it to the market, although in a limited form. Any smart device is just like a computer or machine that can be hacked by means of feeding some simple malicious lines of code. So, security of IoT devices is as important as their deployment. Or else, we will have to face dire consequences, as experienced recently in the form of a North Korean hacker charged for WannaCry ransomware and a 16 year old hacking into Apple’s servers to access customer data. Programmers need to develop suspicious-activity-proof algorithms for IoT devices. Failing to do so will not only make the devices vulnerable to unintended use but also put the entire system at risk. Hence, with the growth in the IoT market, concern about its safety will also mushroom. #3 Video Content will continue to dominate the Web In order to solve the dire glitches caused by plugins, the HTML standards committee started embedding video tags into HTML. Videos tags are programmable by virtue of the fact that basic video tags respond to JavaScript commands. Earlier video content was fixed. If you watch a video about dogs fighting cats, then you will be recommended just that. Nothing more, nothing less. However, this is not the case anymore. It is the time of seamless canvas design, in which web designers figure out clever ways to deploy different video content. Doing so allows the user to steer the way in which a narrative is unfolded and it opens up new ways of interacting with the video content. Now machine learning can deliver higher-quality streaming experiences that do not buffer as much as many existing systems. More efficient codecs and better video compression are also playing a role in making video a better digital consumption medium. Again, programming makes it feasible, as video tags and iframe are part of the programming code. #4 Consoles, consoles everywhere Thanks to the groundbreaking progress in video game console technology, PCs are continuously being rejected in favor of gaming consoles. Living room consoles are just the start. With the concept of intelligent devices, makers of other household items are also looking to make their offerings smarter. Our hairdryers and toasters are already boasting digital memory, allowing for remembering our preferences. However, the time when these, and other household units as well, will start communicating with each other i.e. exchanging information on their own is yet to come. All of these scenarios are only made possible by programming. As several programmers have already embarked on the journey for achieving results in the same direction, we might not be that far away from a time when the aforementioned scenario would be a day-to-day reality. #5 Data is important, data will be important Data is the backbone of the network of networks i.e. the Internet. What we see, read, and hear over the gigantic web is data, loads and loads of it. However, data collection is not something new for humanity. Since antiquity, humans have collected and stored large chunks of data for churning out important information at some later time. With the passage of time, enriching and protecting data have become important. While the former is achieved by presenting data in the form of videos, pictures, pie charts, etc., the latter is accomplished by adding SSL to the website and using better encryption techniques. Data processing has become equally important just like the digital ecosphere itself. In the enterprise community, data gathering will branch out more elaborately into storing, curating, and parsing. Simply said, data is and data will be the undisputed champion in the Digital World. #6 Machine Learning dominance Machine Learning is already flourishing and seeping into everyday enterprise and life. For example, machine learning algorithms are already finding a place in important automation code for big businesses. They are used for heaping big data projects. Languages like the R programming language and Python have enabled this proliferation of machine learning, so far. What’s amazing about machine learning is that it is slowly being integrated into modern life. It will soon become a common entity in a person’s life, just like smartphones and IoT. Again, machine learning also requires services of programming and code, of course. No code, no machine learning. At least for now. There is the rise of machine learning as a service trend which aims to remove or minimize programming. However, if we ever learning anything from the history of web development, even as drag and drop web design tools grow, professional web developers also grow in demand. We can expect to see a similar trend with machine learning as it continues down the path of democratization. #7 User Interface design will continue gaining popularity The time when an Internet user was expected to use a keyboard and mouse is long gone. With each passing day, using a PC is preferred less and less. Apart from offices and college laboratories, PCs are gradually being replaced by other smart devices. As smartphones, tablets, living room consoles, etc. take on the world, the emphasis on UI has heightened. A touch and a click on the screen is different. With the advancement in technology, the former is given preference. This is because it’s quick and convenient at the same time. Furthermore, face and fingerprint recognition are the new cool. Research on voice control is also advancing. Many brands have already incepted their very own virtual assistants, such as Amazon Alexa, Siri and Google Assistant, which can recognize the demands of their users with mere voice commands and interaction. For example, Android 9 Pie comes with a number of UI alterations to stay relevant with the present UI scenario, including a new position for the volume controls and Material Theming. The latter is a built-in Android toolset meant for customizing the Material Design supported by the Android. Again, designing a powerful user interface is dependent on great programming. A user interface needs not only to be robust only but also show signs of intuitiveness and interactivity. The stress on UI designing will continue growing in the future. Some of the upcoming UI trends forecasted for 2019 are the overlapping effect, functional animations, and contrast of fonts. #8 Open Source vs. Closed Development Nearly all laptops run on proprietary software but Smartphones with Android leading the race are mostly open source. iOS is still closed but it has a robust set of APIs on which developers can build their own empires. While open source software is something that anyone can tinker with, closed development environment restricts 3rd-party from accessing and toying with such a system. Among other differences between the two, a significant difference is in the quality of support. This is, obviously, better offered by closed source software. Open source is rocking the world with new developers entering into programming by tinkering with open source whereas closed environment is also growing tremendously because of personalization and security features. This is one hell of a competition. #9 Autonomous Transportation Another industry that requires services of programming is the autonomous vehicles. Just yesterday, Waymo announced that their first driverless cars will be on the road commercially next month. So far, we have only seen some of the many accomplishments that a driverless mode of transportation can achieve. Though we have only cars, for now, that is making use of autonomous transportation algorithms, soon other transportation means will also join the parade. There are already crowdfunding projects for autonomous skateboards. Known as XTND Board, it is a lightweight electric vehicle meant to redefine commuting. Autonomous aircrafts are  already being used in the military. However, pilotless airplane transportation may just be around the corner. All it requires is an excellent programming code to allow a vehicle to know that what route should it chose. So, maybe flights might become autonomous after rides. #10 The Law will redefine new limits Writing code is like fixing something, setting up protocols. What the program will do and what it won’t, depends entirely on the coding. However, there are several ways to manipulate harmful programming code. There’s a subtle analogy between programming code and law and both have their own jurisdictions. Though there is a bright, sunny side to the technological advancement, there’s also a darker side of the same that needs to be reviewed and regulated. As years will pass from this point in time, programmers will face real-world challenges to assist the Law & Order to sustain the malicious content of the society, both on the digital front and the real-world front. We have already seen how adding technology to law works. However, the other side is that it can also act as a tool to break the law(s). Cyberattacks, identity theft, and data laundering are some of the notable examples made possible by technology. This is a question which is also its own solution. In order to prevent such insincere acts, security personnel need to think like bypassers. This is where ethical hacking comes in. It is simply thinking and operating like a malicious hacker but doing so for the right cause. #11 Containers will continue to rule Theoretically, there isn’t a need for the so-called containers, which are heavily deployed in the modern-day programming. In theory, the executable files can run anywhere and various requisite permissions, such as using hardware, are given by the OS. Hence, there is, theoretically, no requirements for a container. However, because of being theoretical, all executables are considered the same. Obviously, this is not the general case. What happens is that executables are different and each one of them requires specific libraries to run. For instance, the WORA (Write Once, Run Anywhere) chant of Java fails owing to the virtue that there are several different versions of virtual machines (VMs). Though using a comprehensive VM might solve the issue, the solution lacks practicality. On the other hand, the sleek and lightweight containers win the preference. Containers are the solution to the issue of reliability caused by a software when it is to be migrated from one computing environment to another. A container is simply a complete package that contains an entire runtime environment with the application, its dependencies and libraries, other required binaries, configuration files, etc. So, when a container of a specific application has everything in it that it requires to operate, the container becomes independent of the platform. The containers will continue to rule in the future up ahead. If you are new to programming, you can check out programming terms for beginners to kickstart your coding journey. These were the future predictions that we can think of. Do want to add anything else? Please feel free to do so in the comments below. Author Bio Saurabh has worked globally for telecom and finance giants in various capacities. After working for a decade in Infosys and Sapient, he started his first startup, Leno, to solve a hyperlocal book-sharing problem. He is interested in product marketing, and analytics. His latest venture Hackr.io recommends the best online programming and design courses for every programming language. All the tutorials are submitted and voted by the programming community. What we learned from IBM Research’s ‘5 in 5’ predictions presented at Think 2018. “Deep learning is not an optimum solution for every problem faced”: An interview with Valentino Zocca. Why does the C programming language refuse to die?  
Read more
  • 0
  • 0
  • 17606

article-image-web-server-development
Packt
15 Apr 2016
24 min read
Save for later

Web Server Development

Packt
15 Apr 2016
24 min read
In this article by Holger Brunn, Alexandre Fayolle, and Daniel Eufémio Gago Reis, the authors of the book, Odoo Development Cookbook, have discussed how to deploy the web server in Odoo. In this article, we'll cover the following topics: Make a path accessible from the network Restrict access to web accessible paths Consume parameters passed to your handlers Modify an existing handler Using the RPC API (For more resources related to this topic, see here.) Introduction We'll introduce the basics of the web server part of Odoo in this article. Note that this article covers the fundamental pieces. All of Odoo's web request handling is driven by the Python library werkzeug (http://werkzeug.pocoo.org). While the complexity of werkzeug is mostly hidden by Odoo's convenient wrappers, it is an interesting read to see how things work under the hood. Make a path accessible from the network In this recipe, we'll see how to make an URL of the form http://yourserver/path1/path2 accessible to users. This can either be a web page or a path returning arbitrary data to be consumed by other programs. In the latter case, you would usually use the JSON format to consume parameters and to offer you data. Getting ready We'll make use of a ready-made library.book model. We want to allow any user to query the full list of books. Furthermore, we want to provide the same information to programs via a JSON request. How to do it… We'll need to add controllers, which go into a folder called controllers by convention. Add a controllers/main.py file with the HTML version of our page: from openerp import http from openerp.http import request class Main(http.Controller): @http.route('/my_module/books', type='http', auth='none') def books(self): records = request.env['library.book']. sudo().search([]) result = '<html><body><table><tr><td>' result += '</td></tr><tr><td>'.join( records.mapped('name')) result += '</td></tr></table></body></html>' return result Add a function to serve the same information in the JSON format @http.route('/my_module/books/json', type='json', auth='none') def books_json(self): records = request.env['library.book']. sudo().search([]) return records.read(['name']) Add the file controllers/__init__.py: from . import main Add controllers to your __init__.py addon: from . import controllers After restarting your server, you can visit /my_module/books in your browser and get presented with a flat list of book names. To test the JSON-RPC part, you'll have to craft a JSON request. A simple way to do that would be using the following command line to receive the output on the command line: curl -i -X POST -H "Content-Type: application/json" -d "{}" localhost:8069/my_module/books/json If you get 404 errors at this point, you probably have more than one database available on your instance. In this case, it's impossible for Odoo to determine which database is meant to serve the request. Use the --db-filter='^yourdatabasename$' parameter to force using exact database you installed the module in. Now the path should be accessible. How it works… The two crucial parts here are that our controller is derived from openerp.http.Controller and that the methods we use to serve content are decorated with openerp.http.route. Inheriting from openerp.http.Controller registers the controller with Odoo's routing system in a similar way as models are registered by inheriting from openerp.models.Model; also, Controller has a meta class that takes care of this. In general, paths handled by your addon should start with your addon's name to avoid name clashes. Of course, if you extend some addon's functionality, you'll use this addon's name. openerp.http.route The route decorator allows us to tell Odoo that a method is to be web accessible in the first place, and the first parameter determines on which path it is accessible. Instead of a string, you can also pass a list of strings in case you use the same function to serve multiple paths. The type argument defaults to http and determines what type of request is to be served. While strictly speaking JSON is HTTP, declaring the second function as type='json' makes life a lot easier, because Odoo then handles type conversions itself. Don't worry about the auth parameter for now, it will be addressed in recipe Restrict access to web accessible paths. Return values Odoo's treatment of the functions' return values is determined by the type argument of the route decorator. For type='http', we usually want to deliver some HTML, so the first function simply returns a string containing it. An alternative is to use request.make_response(), which gives you control over the headers to send in the response. So to indicate when our page was updated the last time, we might change the last line in books() to the following: return request.make_response( result, [ ('Last-modified', email.utils.formatdate( ( fields.Datetime.from_string( request.env['library.book'].sudo() .search([], order='write_date desc', limit=1) .write_date) - datetime.datetime(1970, 1, 1) ).total_seconds(), usegmt=True)), ]) This code sends a Last-modified header along with the HTML we generated, telling the browser when the list was modified for the last time. We extract this information from the write_date field of the library.book model. In order for the preceding snippet to work, you'll have to add some imports on the top of the file: import email import datetime from openerp import fields You can also create a Response object of werkzeug manually and return that, but there's little gain for the effort. Generating HTML manually is nice for demonstration purposes, but you should never do this in production code. Always use templates as appropriate and return them by calling request.render(). This will give you localization for free and makes your code better by separating business logic from the presentation layer. Also, templates provide you with functions to escape data before outputting HTML. The preceding code is vulnerable to cross-site-scripting attacks if a user manages to slip a script tag into the book name, for example. For a JSON request, simply return the data structure you want to hand over to the client, Odoo takes care of serialization. For this to work, you should restrict yourself to data types that are JSON serializable, which are roughly dictionaries, lists, strings, floats and integers. openerp.http.request The request object is a static object referring to the currently handled request, which contains everything you need to take useful action. Most important is the property request.env, which contains an Environment object which is just the same as in self.env for models. This environment is bound to the current user, which is none in the preceding example because we used auth='none'. Lack of a user is also why we have to sudo() all our calls to model methods in the example code. If you're used to web development, you'll expect session handling, which is perfectly correct. Use request.session for an OpenERPSession object (which is quite a thin wrapper around the Session object of werkzeug), and request.session.sid to access the session id. To store session values, just treat request.session as a dictionary: request.session['hello'] = 'world' request.session.get('hello') Note that storing data in the session is not different from using global variables. Use it only if you must - that is usually the case for multi request actions like a checkout in the website_sale module. And also in this case, handle all functionality concerning sessions in your controllers, never in your modules. There's more… The route decorator can have some extra parameters to customize its behavior further. By default, all HTTP methods are allowed, and Odoo intermingles with the parameters passed. Using the parameter methods, you can pass a list of methods to accept, which usually would be one of either ['GET'] or ['POST']. To allow cross origin requests (browsers block AJAX and some other types of requests to domains other than where the script was loaded from for security and privacy reasons), set the cors parameter to * to allow requests from all origins, or some URI to restrict requests to ones originating from this URI. If this parameter is unset, which is the default, the Access-Control-Allow-Origin header is not set, leaving you with the browser's standard behavior. In our example, we might want to set it on /my_module/books/json in order to allow scripts pulled from other websites accessing the list of books. By default, Odoo protects certain types of requests from an attack known as cross-site request forgery by passing a token along on every request. If you want to turn that off, set the parameter csrf to False, but note that this is a bad idea in general. See also If you host multiple Odoo databases on the same instance and each database has different web accessible paths on possibly multiple domain names per database, the standard regular expressions in the --db-filter parameter might not be enough to force the right database for every domain. In that case, use the community module dbfilter_from_header from https://github.com/OCA/server-tools in order to configure the database filters on proxy level. To see how using templates makes modularity possible, see recipe Modify an existing handler later in the article. Restrict access to web accessible paths We'll explore the three authentication mechanisms Odoo provides for routes in this recipe. We'll define routes with different authentication mechanisms in order to show their differences. Getting ready As we extend code from the previous recipe, we'll also depend on the library.book model, so you should get its code correct in order to proceed. How to do it… Define handlers in controllers/main.py: Add a path that shows all books: @http.route('/my_module/all-books', type='http', auth='none') def all_books(self): records = request.env['library.book'].sudo().search([]) result = '<html><body><table><tr><td>' result += '</td></tr><tr><td>'.join( records.mapped('name')) result += '</td></tr></table></body></html>' return result Add a path that shows all books and indicates which was written by the current user, if any: @http.route('/my_module/all-books/mark-mine', type='http', auth='public') def all_books_mark_mine(self): records = request.env['library.book'].sudo().search([]) result = '<html><body><table>' for record in records: result += '<tr>' if record.author_ids & request.env.user.partner_id: result += '<th>' else: result += '<td>' result += record.name if record.author_ids & request.env.user.partner_id: result += '</th>' else: result += '</td>' result += '</tr>' result += '</table></body></html>' return result Add a path that shows the current user's books: @http.route('/my_module/all-books/mine', type='http', auth='user') def all_books_mine(self): records = request.env['library.book'].search([ ('author_ids', 'in', request.env.user.partner_id.ids), ]) result = '<html><body><table><tr><td>' result += '</td></tr><tr><td>'.join( records.mapped('name')) result += '</td></tr></table></body></html>' return result With this code, the paths /my_module/all_books and /my_module/all_books/mark_mine look the same for unauthenticated users, while a logged in user sees her books in a bold font on the latter path. The path /my_module/all-books/mine is not accessible at all for unauthenticated users. If you try to access it without being authenticated, you'll be redirected to the login screen in order to do so. How it works… The difference between authentication methods is basically what you can expect from the content of request.env.user. For auth='none', the user record is always empty, even if an authenticated user is accessing the path. Use this if you want to serve content that has no dependencies on users, or if you want to provide database agnostic functionality in a server wide module. The value auth='public' sets the user record to a special user with XML ID, base.public_user, for unauthenticated users, and to the user's record for authenticated ones. This is the right choice if you want to offer functionality to both unauthenticated and authenticated users, while the authenticated ones get some extras, as demonstrated in the preceding code. Use auth='user' to be sure that only authenticated users have access to what you've got to offer. With this method, you can be sure request.env.user points to some existing user. There's more… The magic for authentication methods happens in the ir.http model from the base addon. For whatever value you pass to the auth parameter in your route, Odoo searches for a function called _auth_method_<yourvalue> on this model, so you can easily customize this by inheriting this model and declaring a method that takes care of your authentication method of choice. As an example, we provide an authentication method base_group_user which enforces a currently logged in user who is a member of the group with XML ID, base.group_user: from openerp import exceptions, http, models from openerp.http import request class IrHttp(models.Model): _inherit = 'ir.http' def _auth_method_base_group_user(self): self._auth_method_user() if not request.env.user.has_group('base.group_user'): raise exceptions.AccessDenied() Now you can say auth='base_group_user' in your decorator and be sure that users running this route's handler are members of this group. With a little trickery you can extend this to auth='groups(xmlid1,…)', the implementation of this is left as an exercise to the reader, but is included in the example code. Consume parameters passed to your handlers It's nice to be able to show content, but it's better to show content as a result of some user input. This recipe will demonstrate the different ways to receive this input and react to it. As the recipes before, we'll make use of the library.book model. How to do it… First, we'll add a route that expects a traditional parameter with a book's ID to show some details about it. Then, we'll do the same, but we'll incorporate our parameter into the path itself: Add a path that expects a book's ID as parameter: @http.route('/my_module/book_details', type='http', auth='none') def book_details(self, book_id): record = request.env['library.book'].sudo().browse( int(book_id)) return u'<html><body><h1>%s</h1>Authors: %s' % ( record.name, u', '.join(record.author_ids.mapped( 'name')) or 'none', ) Add a path where we can pass the book's ID in the path @http.route("/my_module/book_details/<model('library.book') :book>", type='http', auth='none') def book_details_in_path(self, book): return self.book_details(book.id) If you point your browser to /my_module/book_details?book_id=1, you should see a detail page of the book with ID 1. If this doesn't exist, you'll receive an error page. The second handler allows you to go to /my_module/book_details/1 and view the same page. How it works… By default, Odoo (actually werkzeug) intermingles with GET and POST parameters and passes them as keyword argument to your handler. So by simply declaring your function as expecting a parameter called book_id, you introduce this parameter as either GET (the parameter in the URL) or POST (usually passed by forms with your handler as action) parameter. Given that we didn't add a default value for this parameter, the runtime will raise an error if you try to access this path without setting the parameter. The second example makes use of the fact that in a werkzeug environment, most paths are virtual anyway. So we can simply define our path as containing some input. In this case, we say we expect the ID of a library.book as the last component of the path. The name after the colon is the name of a keyword argument. Our function will be called with this parameter passed as keyword argument. Here, Odoo takes care of looking up this ID and delivering a browse record, which of course only works if the user accessing this path has appropriate permissions. Given that book is a browse record, we can simply recycle the first example's function by passing book.id as parameter book_id to give out the same content. There's more… Defining parameters within the path is a functionality delivered by werkzeug, which is called converters. The model converter is added by Odoo, which also defines the converter, models, that accepts a comma separated list of IDs and passes a record set containing those IDs to your handler. The beauty of converters is that the runtime coerces the parameters to the expected type, while you're on your own with normal keyword parameters. These are delivered as strings and you have to take care of the necessary type conversions yourself, as seen in the first example. Built-in werkzeug converters include int, float, and string, but also more intricate ones such as path, any, or uuid. You can look up their semantics at http://werkzeug.pocoo.org/docs/0.11/routing/#builtin-converters. See also Odoo's custom converters are defined in ir_http.py in the base module and registered in the _get_converters method of ir.http. As an exercise, you can create your own converter that allows you to visit the /my_module/book_details/Odoo+cookbook page to receive the details of this book (if you added it to your library before). Modify an existing handler When you install the website module, the path /website/info displays some information about your Odoo instance. In this recipe, we override this in order to change this information page's layout, but also to change what is displayed. Getting ready Install the website module and inspect the path /website/info. Now craft a new module that depends on website and uses the following code. How to do it… We'll have to adapt the existing template and override the existing handler: Override the qweb template in a file called views/templates.xml: <?xml version="1.0" encoding="UTF-8"?> <odoo> <template id="show_website_info" inherit_id="website.show_website_info"> <xpath expr="//dl[@t-foreach='apps']" position="replace"> <table class="table"> <tr t-foreach="apps" t-as="app"> <th> <a t-att-href="app.website"> <t t-esc="app.name" /></a> </th> <td><t t-esc="app.summary" /></td> </tr> </table> </xpath> </template> </odoo> Override the handler in a file called controllers/main.py: from openerp import http from openerp.addons.website.controllers.main import Website class Website(Website): @http.route() def website_info(self): result = super(Website, self).website_info() result.qcontext['apps'] = result.qcontext[ 'apps'].filtered( lambda x: x.name != 'website') return result Now when visiting the info page, we'll only see a filtered list of installed applications, and in a table as opposed to the original definition list. How it works In the first step, we override an existing QWeb template. In order to find out which that is, you'll have to consult the code of the original handler. Usually, it will end with the following command line, which tells you that you need to override template.name: return request.render('template.name', values) In our case, the handler uses a template called website.info, but this one is extended immediately by another template called website.show_website_info, so it's more convenient to override this one. Here, we replace the definition list showing installed apps with a table. In order to override the handler method, we must identify the class that defines the handler, which is openerp.addons.website.controllers.main.Website in this case. We import the class to be able to inherit from it. Now we override the method and change the data passed to the response. Note that what the overridden handler returns is a Response object and not a string of HTML as the previous recipes did for the sake of brevity. This object contains a reference to the template to be used and the values accessible to the template, but is only evaluated at the very end of the request. In general, there are three ways to change an existing handler: If it uses a QWeb template, the simplest way of changing it is to override the template. This is the right choice for layout changes and small logic changes. QWeb templates get a context passed, which is available in the response as the field qcontext. This usually is a dictionary where you can add or remove values to suit your needs. In the preceding example, we filter the list of apps to only contain apps which have a website set. If the handler receives parameters, you could also preprocess those in order to have the overridden handler behave the way you want. There's more… As seen in the preceding section, inheritance with controllers works slightly differently than model inheritance: You actually need a reference to the base class and use Python inheritance on it. Don't forget to decorate your new handler with the @http.route decorator; Odoo uses it as a marker for which methods are exposed to the network layer. If you omit the decorator, you actually make the handler's path inaccessible. The @http.route decorator itself behaves similarly to field declarations: every value you don't set will be derived from the decorator of the function you're overriding, so we don't have to repeat values we don't want to change. After receiving a response object from the function you override, you can do a lot more than just changing the QWeb context: You can add or remove HTTP headers by manipulating response.headers. If you want to render an entirely different template, you can set response.template. To detect if a response is based on QWeb in the first place, query response.is_qweb. The resulting HTML code is available by calling response.render(). Using the RPC API One of Odoo's strengths is its interoperability, which is helped by the fact that basically any functionality is available via JSON-RPC 2.0 and XMLRPC. In this recipe, we'll explore how to use both of them from client code. This interface also enables you to integrate Odoo with any other application. Making functionality available via any of the two protocols on the server side is explained in the There's more section of this recipe. We'll query a list of installed modules from the Odoo instance, so that we could show a list as the one displayed in the previous recipe in our own application or website. How to do it… The following code is not meant to run within Odoo, but as simple scripts: First, we query the list of installed modules via XMLRPC: #!/usr/bin/env python2 import xmlrpclib db = 'odoo9' user = 'admin' password = 'admin' uid = xmlrpclib.ServerProxy( 'http://localhost:8069/xmlrpc/2/common') .authenticate(db, user, password, {}) odoo = xmlrpclib.ServerProxy( 'http://localhost:8069/xmlrpc/2/object') installed_modules = odoo.execute_kw( db, uid, password, 'ir.module.module', 'search_read', [[('state', '=', 'installed')], ['name']], {'context': {'lang': 'fr_FR'}}) for module in installed_modules: print module['name'] Then we do the same with JSONRPC: import json import urllib2 db = 'odoo9' user = 'admin' password = 'admin' request = urllib2.Request( 'http://localhost:8069/web/session/authenticate', json.dumps({ 'jsonrpc': '2.0', 'params': { 'db': db, 'login': user, 'password': password, }, }), {'Content-type': 'application/json'}) result = urllib2.urlopen(request).read() result = json.loads(result) session_id = result['result']['session_id'] request = urllib2.Request( 'http://localhost:8069/web/dataset/call_kw', json.dumps({ 'jsonrpc': '2.0', 'params': { 'model': 'ir.module.module', 'method': 'search_read', 'args': [ [('state', '=', 'installed')], ['name'], ], 'kwargs': {'context': {'lang': 'fr_FR'}}, }, }), { 'X-Openerp-Session-Id': session_id, 'Content-type': 'application/json', }) result = urllib2.urlopen(request).read() result = json.loads(result) for module in result['result']: print module['name'] Both code snippets will print a list of installed modules, and because they pass a context that sets the language to French, the list will be in French if there are no translations available. How it works… Both snippets call the function search_read, which is very convenient because you can specify a search domain on the model you call, pass a list of fields you want to be returned, and receive the result in one request. In older versions of Odoo, you had to call search first to receive a list of IDs and then call read to actually read the data. search_read returns a list of dictionaries, with the keys being the names of the fields requested and the values the record's data. The ID field will always be transmitted, no matter if you requested it or not. Now, we need to look at the specifics of the two protocols. XMLRPC The XMLRPC API expects a user ID and a password for every call, which is why we need to fetch this ID via the method authenticate on the path /xmlrpc/2/common. If you already know the user's ID, you can skip this step. As soon as you know the user's ID, you can call any model's method by calling execute_kw on the path /xmlrpc/2/object. This method expects the database you want to execute the function on, the user's ID and password for authentication, then the model you want to call your function on, and then the function's name. The next two mandatory parameters are a list of positional arguments to your function, and a dictionary of keyword arguments. JSONRPC Don't be distracted by the size of the code example, that's because Python doesn't have built in support for JSONRPC. As soon as you've wrapped the urllib calls in some helper functions, the example will be as concise as the XMLRPC one. As JSONRPC is stateful, the first thing we have to do is to request a session at /web/session/authenticate. This function takes the database, the user's name, and their password. The crucial part here is that we record the session ID Odoo created, which we pass in the header X-Openerp-Session-Id to /web/dataset/call_kw. Then the function behaves the same as execute_kw from; we need to pass a model name and a function to call on it, then positional and keyword arguments. There's more… Both protocols allow you to call basically any function of your models. In case you don't want a function to be available via either interface, prepend its name with an underscore – Odoo won't expose those functions as RPC calls. Furthermore, you need to take care that your parameters, as well as the return values, are serializable for the protocol. To be sure, restrict yourself to scalar values, dictionaries, and lists. As you can do roughly the same with both protocols, it's up to you which one to use. This decision should be mainly driven by what your platform supports best. In a web context, you're generally better off with JSON, because Odoo allows JSON handlers to pass a CORS header conveniently (see the Make a path accessible from the network recipe for details). This is rather difficult with XMLRPC. Summary In this article, we saw how to start about with the web server architecture. Later on, we covered the Routes and Controllers that will be used in the article and their authentication, how the handlers consumes parameters, and how to use an RPC API, namely, JSON-RPC and XML-RPC. Resources for Article: Further resources on this subject: Advanced React [article] Remote Authentication [article] ASP.Net Site Performance: Improving JavaScript Loading [article]
Read more
  • 0
  • 0
  • 17536
Packt
17 Mar 2016
9 min read
Save for later

Microservices – Brave New World

Packt
17 Mar 2016
9 min read
In this article by David Gonzalez, author of the book Developing Microservices with Node.js, we will cover the need for microservices, explain the monolithic approach, and study how to build and deploy microservices. (For more resources related to this topic, see here.) Need for microservices The world of software development has evolved quickly over the past 40 years. One of the key points of this evolution has been the size of these systems. From the days of MS-DOS, we taken a hundred-fold leap into our present systems. This growth in size creates a need for better ways of organizing the code and software components. Usually, when a company grows due to business needs, which is known as organic growth, the software gets organized on a monolithic architecture as it is the easiest and quickest way of building software. After few years (or even months), adding new features becomes harder due to the coupled nature of the created software. Monolithic software There are a few companies that have already started building their software using microservices, which is the ideal scenario. The problem is that not all the companies can plan their software upfront. Instead of planning, these companies build the software based on the organic growth experienced: few software components that group business flows by affinity. It is not rare to see companies having two big software components: the user facing website and the internal administration tools. This is usually known as a monolithic software architecture. Some of these companies face big problems when trying to scale the engineering teams. It is hard to coordinate the teams that build, deploy, and maintain a single software component. Clashes on releases and reintroduction of bugs are a common problem that drains a big chunk of energy from the teams. One of the solution to this problem (it also has other benefits) is to split the monolithic software into microservices so that the teams are able to specialize in few smaller modules and autonomous and isolated software components that can be versioned, updated, and deployed without interfering with the rest of the systems of the company. One of the most interesting solutions to this problem is splitting the monolithic architecture into microservices. This enables the engineering team to create isolated and autonomous units of work that are highly specialized in a given task (such as sending e-mails, processing card payment, and so on). Microservices in the real world Microservices are small software components that specialize in one task and work together to achieve a higher-level task. Forget about software for a second and think about how a company works. When someone applies for a job in a company, he applies for a given position: software engineer, systems administrator, or office manager The reason for it can be summarized in one word—specialization. If you are used to working as a software engineer, you will get better with the experience and add more value to the company. The fact that you don’t know how to deal with a customer, won’t affect your performance as it is not your area of expertise and will hardly add any value to your day-to-day work. A microservice is an autonomous unit of work that can execute one task without interfering with other parts of the system, similar to what a job position is to a company. This has a number of benefits that can be used in favor of the engineering team in order to help to scale the systems of a company. Nowadays, hundreds of systems are built using a microservices-oriented architectures, as follows: Netflix: They are one of the most popular streaming services and have built an entire ecosystem of applications that collaborate in order to provide a reliable and scalable streaming system used across the globe. Spotify: They are one of the leading music streaming services in the world and have built this application using microservices. Every single widget of the application (which is a website exposed as a desktop app using Chromium Embedded Framework (CEF)) is a different microservice that can be updated individually. First, there was the monolith A huge percentage (my estimate is around 90%) of the modern enterprise software is built following a monolithic approach. Huge software components that run in a single container and have a well-defined development life cycle that goes completely against the following agile principles, deliver early and deliver often (https://en.wikipedia.org/wiki/Release_early,_release_often): Deliver early: The sooner you fail, the easier it is to recover. If you are working for two years in a software component and then, it is released, there is a huge risk of deviation from the original requirements, which are usually wrong and changing every few days. Deliver often: Everything of the software is delivered to all the stake holders so that they can have their inputs and see the changes reflected in the software. Errors can be fixed in a few days and improvements are identified easily. Companies build big software components instead of smaller ones that work together as it is the natural thing to do, as follows: The developer has a new requirement. He builds a new method on an existing class on the service layer. The method is exposed on the API via HTTP, SOAP, or any other protocol. Now, repeat it by the number of developers in your company and you will obtain something called organic growth. Organic growth is the type of uncontrolled and unplanned growth on software systems under business pressure without an adequate long-term planning, and it is bad. How to tackle the organic growth? The first thing needed to tackle the organic growth is make sure that business and IT are aligned in the company. Usually, in big companies, IT is not seen as a core part of the business. Organizations outsource their IT systems, keeping the cost in mind, but not the quality so that the partners building these software components are focused on one thing: deliver on time and according to the specification, even if it is incorrect. This produces a less-than-ideal ecosystem to respond to the business needs with a working solution for an existing problem. IT is lead by people who barely understand how the systems are built and usually overlook the complexity of the software development. Fortunately, this is a changing tendency as IT systems have become the drivers of 99% of the businesses around the world, but we need to be smarter about how we build them. The first measure to tackle the organic growth is to align IT and business stakeholders in order to work together, educating the non-technical stakeholders is the key to success. If we go back to the example from the previous section (few releases with quite big changes). Can we do it better? Of course, we can. Divide the work into manageable software artifacts that model a single and well-defined business activity and give it an entity. It does not need to be a microservice at this stage, but keeping the logic inside a separated, well-defined, easy testable, and decoupled module will give us a huge advantage towards future changes in the application. Building microservices – The fallback strategy When you design a system, we usually think about the replaceability of the existing components. For example, when using a persistence technology in Java, we tend to lean towards the standards (Java Persistence API (JPA)) so that we can replace the underneath implementation without too much effort. Microservices take the same approach, but they isolate the problem instead of working towards an easy replaceability. Also, e-mailing is something that, although it seems simple, always ends up giving problems. Consider that we want to replace Mandrill with a plain SMTP server, such as Gmail. We don't need to do anything special, we just change the implementation and rollout the new version of our microservice, as follows: var nodemailer = require('nodemailer'); var seneca = require("seneca")(); var transporter = nodemailer.createTransport({ service: 'Gmail', auth: { user: 'info@micromerce.com', pass: 'verysecurepassword' } }); /** * Sends an email including the content. */ seneca.add({area: "email", action: "send"}, function(args, done) { var mailOptions = { from: 'Micromerce Info ✔ <info@micromerce.com>', to: args.to, subject: args.subject, html: args.body }; transporter.sendMail(mailOptions, function(error, info){ if(error){ done({code: e}, null); } done(null, {status: "sent"}); }); }); For the outer world, our simplest version of the e-mail sender is now at all lights, using SMTP through Gmail to deliver our e-mails. We could even rollout one server with this version and send some traffic to it in order to validate our implementation without affecting all the customers (in other words, contain the failure). Deploying microservices Deployment is usually the ugly friend of the software development life cycle party. There is a missing contact point in between development and system administration, which DevOps is going to solve in the following few years (or has already done it and no one told me). The following is the graph showing the cost of fixing software bugs versus the various phases of development: From the continuous integration up to continuous delivery, the process should be automated as much as possible, where as much as possible means 100%. Remember, humans are imperfect…if we rely on humans carrying on a manual repetitive process for a bug-free software, we are walking the wrong path. Remember that a machine will always be error free (as long as the algorithm that is executed is error free) so…why not let a machine control our infrastructure? Summary In this article, we saw how microservices are required in complex software systems, how the monolithic approach is useful, and how to build and deploy microservices. Resources for Article: Further resources on this subject: Making a Web Server in Node.js [article] Node.js Fundamentals and Asynchronous JavaScript [article] An Introduction to Node.js Design Patterns [article]
Read more
  • 0
  • 0
  • 17495

article-image-net-45-extension-methods-iqueryable
Packt
14 May 2013
4 min read
Save for later

.NET 4.5 Extension Methods on IQueryable

Packt
14 May 2013
4 min read
(For more resources related to this topic, see here.) Getting ready Refer to the IQueryableExtensions.cs file in the ExtensionMethods.Libraryproject for the extension methods. The models are located in Models/PagedList.cs and Models/IPagedList.cs. These methods are used in the IQueryableExtensionTests.cs file in the ExtensionMethods.Tests project. How to do it... The following code snippet shows a general use of extension methods on IQueryables: public static User ByUserId(this IQueryable<User> query, int userId) { return query.First(u => u.UserId == userId); } The following code snippet is a paged list class for pagination of data: public class PagedList<T> : List<T>, IPagedList { public PagedList(IQueryable<T> source, int index, int pageSize) { this.TotalCount = source.Count(); this.PageSize = pageSize; this.PageIndex = index; this.AddRange(source.Skip(index * pageSize).Take(pageSize). ToList()); } public PagedList(List<T> source, int index, int pageSize) { this.TotalCount = source.Count(); this.PageSize = pageSize; this.PageIndex = index; this.AddRange(source.Skip(index * pageSize).Take(pageSize). ToList()); } public int TotalCount { get; set; } public int PageIndex { get; set; } public int PageSize { get; set; } public bool IsPreviousPage { get { return (PageIndex > 0); } } public bool IsNextPage { get { return (PageIndex * PageSize) <=TotalCount; } } } The following code snippet is the extension method that executes and converts the query to the PagedList object: public static PagedList<T> ToPagedList<T>(this IQueryable<T> source, int index, int pageSize) { return new PagedList<T>(source, index, pageSize); } The following code snippet shows how we use these extension methods: [TestMethod] public void UserByIdReturnsCorrectUser() { var query = new List<User> { new User {UserId = 1}, new User {UserId = 2} }.AsQueryable(); var user = query.ByUserId(1); Assert.AreEqual(1, user.UserId); } [TestMethod] public void PagedList_Contains_Correct_Number_Of_Elements() { var query = new List<int>{1,2,3,4,5,6,7,8,9,10}.AsQueryable(); var pagedList = query.ToPagedList(0, 5); Assert.AreEqual(5, pagedList.Count); Assert.AreEqual(10, pagedList.TotalCount); } How it works... The first code snippet ByUserId is the most commonly used type of extension method for IQueryable types. An alternative to this method is to use the repository pattern and add a method of getting a user by the Id. But sometimes, we will expose the query to lower levels of the app such as the service layer where we might need to use this feature at multiple places, hence refactoring that logic into an extension method makes perfect sense. This extension method evaluates and executes the query immediately due to requesting a single value using the First() method: query.First(u => u.UserId == userId); The second code snippet gives us a PagedList model which becomes a valuable class when working with grids or pagination. The constructor accepts an IQueryable or IList and converts that data into a paged list. Take note of the line in which we evaluate the source by calling ToList(). This line executes the query on the provider: this.AddRange(source.Skip(index * pageSize).Take(pageSize).ToList()); In the code snippets using these extension methods, we have created a list and cast it to an IQueryable type. This is purely for the purpose of demonstration. In a real application, the query would be coming from a LINQ to SQL or entities context, which is in charge of executing the query against a database. We need to be careful of how extension methods on IQueryable are written. A poorly written query will result in unexpected behavior, such as premature query execution. If the extension method is simply building up the query (using method chaining), ensure that the query is not evaluated inside the method. If the query is evaluated and executed before the method finishes, any other use of the query outside of the extension method will result in operating on the data in memory. Summary In this article, you have learned a few tricks and caveats when using extending IQueryable. Resources for Article : Further resources on this subject: Working With ASP.NET DataList Control [Article] NHibernate 3.0: Using LINQ Specifications in the data access layer [Article] LINQ to Objects [Article]
Read more
  • 0
  • 0
  • 17478
Modal Close icon
Modal Close icon