Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-aggregators-file-exchange-over-ftpftps-social-integration-and-enterprise-messaging
Packt
20 Feb 2015
26 min read
Save for later

Aggregators, File exchange Over FTP/FTPS, Social Integration, and Enterprise Messaging

Packt
20 Feb 2015
26 min read
In this article by Chandan Pandey, the author of Spring Integration Essentials, we will explore the out-of-the-box capabilities that the Spring Integration framework provides for a seamless flow of messages across heterogeneous components and see what Spring Integration has in the box when it comes to real-world integration challenges. We will cover Spring Integration's support for external components and we will cover the following topics in detail: Aggregators File exchange over FTP/FTPS Social integration Enterprise messaging (For more resources related to this topic, see here.) Aggregators The aggregators are the opposite of splitters - they combine multiple messages and present them as a single message to the next endpoint. This is a very complex operation, so let's start by a real life scenario. A news channel might have many correspondents who can upload articles and related images. It might happen that the text of the articles arrives much sooner than the associated images - but the article must be sent for publishing only when all relevant images have also arrived. This scenario throws up a lot of challenges; partial articles should be stored somewhere, there should be a way to correlate incoming components with existing ones, and also there should be a way to identify the completion of a message. Aggregators are there to handle all of these aspects - some of the relevant concepts that are used are MessageStore, CorrelationStrategy, and ReleaseStrategy. Let's start with a code sample and then we will dive down to explore each of these concepts in detail: <int:aggregator   input-channel="fetchedFeedChannelForAggregatior"   output-channel="aggregatedFeedChannel"   ref="aggregatorSoFeedBean"   method="aggregateAndPublish"   release-strategy="sofeedCompletionStrategyBean"   release-strategy-method="checkCompleteness"   correlation-strategy="soFeedCorrelationStrategyBean"   correlation-strategy-method="groupFeedsBasedOnCategory"   message-store="feedsMySqlStore "   expire-groups-upon-completion="true">   <int:poller fixed-rate="1000"></int:poller> </int:aggregator> Hmm, a pretty big declaration! And why not—a lot of things combine together to act as an aggregator. Let's quickly glance at all the tags used: int:aggregator: This is used to specify the Spring framework's namespace for the aggregator. input-channel: This is the channel from which messages will be consumed. output-channel: This is the channel to which messages will be dropped after aggregation. ref: This is used to specify the bean having the method that is called on the release of messages. method: This is used to specify the method that is invoked when messages are released. release-strategy: This is used to specify the bean having the method that decides whether aggregation is complete or not. release-strategy-method: This is the method having the logic to check for completeness of the message. correlation-strategy: This is used to specify the bean having the method to correlate the messages. correlation-strategy-method: This is the method having the actual logic to correlate the messages. message-store: This is used to specify the message store, where messages are temporarily stored until they have been correlated and are ready to release. This can be in memory (which is default) or can be a persistence store. If a persistence store is configured, message delivery will be resumed across a server crash. Java class can be defined as an aggregator and, as described in the previous bullet points, the method and ref parameters decide which method of bean (referred by ref) should be invoked when messages have been aggregated as per CorrelationStrategy and released after fulfilment of ReleaseStrategy. In the following example, we are just printing the messages before passing them on to the next consumer in the chain: public class SoFeedAggregator {   public List<SyndEntry> aggregateAndPublish(List<SyndEntry>     messages) {     //Do some pre-processing before passing on to next channel     return messages;   } } Let's get to the details of the three most important components that complete the aggregator. Correlation strategy Aggregator needs to group the messages—but how will it decide the groups? In simple words, CorrelationStrategy decides how to correlate the messages. The default is based on a header named CORRELATION_ID. All messages having the same value for the CORRELATION_ID header will be put in one bracket. Alternatively, we can designate any Java class and its method to define a custom correlation strategy or can extend Spring Integration framework's CorrelationStrategy interface to define it. If the CorrelationStrategy interface is implemented, then the getCorrelationKey() method should be implemented. Let's see our correlation strategy in the feeds example: public class CorrelationStrategy {   public Object groupFeedsBasedOnCategory(Message<?> message) {     if(message!=null){       SyndEntry entry = (SyndEntry)message.getPayload();       List<SyndCategoryImpl> categories=entry.getCategories();       if(categories!=null&&categories.size()>0){         for (SyndCategoryImpl category: categories) {           //for simplicity, lets consider the first category           return category.getName();         }       }     }     return null;   } } So how are we correlating our messages? We are correlating the feeds based on the category name. The method must return an object that can be used for correlating the messages. If a user-defined object is returned, it must satisfy the requirements for a key in a map such as defining hashcode() and equals(). The return value must not be null. Alternatively, if we would have wanted to implement it by extending framework support, then it would have looked like this: public class CorrelationStrategy implements CorrelationStrategy {   public Object getCorrelationKey(Message<?> message) {     if(message!=null){       …             return category.getName();           }         }       }       return null;     }   } } Release strategy We have been grouping messages based on correlation strategy—but when will we release it for the next component? This is decided by the release strategy. Similar to the correlation strategy, any Java POJO can define the release strategy or we can extend framework support. Here is the example of using the Java POJO class: public class CompletionStrategy {   public boolean checkCompleteness(List<SyndEntry> messages) {     if(messages!=null){       if(messages.size()>2){         return true;       }     }     return false;   } } The argument of a message must be of type collection and it must return a Boolean indication whether to release the accumulated messages or not. For simplicity, we have just checked for the number of messages from the same category—if it's greater than two, we release the messages. Message store Until an aggregated message fulfils the release criteria, the aggregator needs to store them temporarily. This is where message stores come into the picture. Message stores can be of two types: in-memory and persistence store. Default is in memory, and if this is to be used, then there is no need to declare this attribute at all. If a persistent message store needs to be used, then it must be declared and its reference should be given to the message- store attribute. A mysql message store can be declared and referenced as follows: <bean id=" feedsMySqlStore "   class="org.springframework.integration.jdbc.JdbcMessageStore">   <property name="dataSource" ref="feedsSqlDataSource"/> </bean> Data source is Spring framework's standard JDBC data source. The greatest advantage of using persistence store is recoverability—if the system recovers from a crash, all in-memory aggregated messages will not be lost. Another advantage is capacity—memory is limited, which can accommodate a limited number of messages for aggregation, but the database can have a much bigger space. FTP/FTPS FTP, or File Transfer Protocol, is used to transfer files across networks. FTP communications consist of two parts: server and client. The client establishes a session with the server, after which it can download or upload files. Spring Integration provides components that act as a client and connect to the FTP server to communicate with it. What about the server—which server will it connect to? If you have access to any public or hosted FTP server, use it. Else, the easiest way for trying out the example in this section is to set up a local instance of the FTP server. FTP setup is out of the scope of this article. Prerequisites To use Spring Integration components for FTP/FTPS, we need to add a namespace to our configuration file and then add the Maven dependency entry in the pom.xml file. The following entries should be made: Namespace support can be added by using the following code snippet:   class="org.springframework.integration.     ftp.session.DefaultFtpSessionFactory">   <property name="host" value="localhost"/>   <property name="port" value="21"/>   <property name="username" value="testuser"/>   <property name="password" value="testuser"/> </bean> The DefaultFtpSessionFactory class is at work here, and it takes the following parameters: Host that is running the FTP server Port at which it's running the server Username Password for the server A session pool for the factory is maintained and an instance is returned when required. Spring takes care of validating that a stale session is never returned. Downloading files from the FTP server Inbound adapters can be used to read the files from the server. The most important aspect is the session factory that we just discussed in the preceding section. The following code snippet configures an FTP inbound adapter that downloads a file from a remote directory and makes it available for processing: <int-ftp:inbound-channel-adapter   channel="ftpOutputChannel"   session-factory="ftpClientSessionFactory"   remote-directory="/"   local-directory=   "C:\Chandan\Projects\siexample\ftp\ftplocalfolder"   auto-create-local-directory="true"   delete-remote-files="true"   filename-pattern="*.txt"   local-filename-generator-expression=   "#this.toLowerCase() + '.trns'">   <int:poller fixed-rate="1000"/> </int-ftp:inbound-channel-adapter> Let's quickly go through the tags used in this code: int-ftp:inbound-channel-adapter: This is the namespace support for the FTP inbound adapter. channel: This is the channel on which the downloaded files will be put as a message. session-factory: This is a factory instance that encapsulates details for connecting to a server. remote-directory: This is the directory on the server where the adapter should listen for the new arrival of files. local-directory: This is the local directory where the downloaded files should be dumped. auto-create-local-directory: If enabled, this will create the local directory structure if it's missing. delete-remote-files: If enabled, this will delete the files on the remote directory after it has been downloaded successfully. This will help in avoiding duplicate processing. filename-pattern: This can be used as a filter, but only files matching the specified pattern will be downloaded. local-filename-generator-expression: This can be used to generate a local filename. An inbound adapter is a special listener that listens for events on the remote directory, for example, an event fired on the creation of a new file. At this point, it will initiate the file transfer. It creates a payload of type Message<File> and puts it on the output channel. By default, the filename is retained and a file with the same name as the remote file is created in the local directory. This can be overridden by using local- filename-generator-expression. Incomplete files On the remote server, there could be files that are still in the process of being written. Typically, there the extension is different, for example, filename.actualext.writing. The best way to avoid reading incomplete files is to use the filename pattern that will copy only those files that have been written completely. Uploading files to the FTP server Outbound adapters can be used to write files to the server. The following code snippet reads a message from a specified channel and writes it inside the FTP server's remote directory. The remote server session is determined as usual by the session factory. Make sure the username configured in the session object has the necessary permission to write to the remote directory. The following configuration sets up a FTP adapter that can upload files in the specified directory:   <int-ftp:outbound-channel-adapter channel="ftpOutputChannel"     remote-directory="/uploadfolder"     session-factory="ftpClientSessionFactory"     auto-create-directory="true">   </int-ftp:outbound-channel-adapter> Here is a brief description of the tags used: int-ftp:outbound-channel-adapter: This is the namespace support for the FTP outbound adapter. channel: This is the name of the channel whose payload will be written to the remote server. remote-directory: This is the remote directory where files will be put. The user configured in the session factory must have appropriate permission. session-factory: This encapsulates details for connecting to the FTP server. auto-create-directory: If enabled, this will automatically create a remote directory if it's missing, and the given user should have sufficient permission. The payload on the channel need not necessarily be a file type; it can be one of the following: java.io.File: A Java file object byte[]: This is a byte array that represents the file contents java.lang.String: This is the text that represents the file contents Avoiding partially written files Files on the remote server must be made available only when they have been written completely and not when they are still partial. Spring uses a mechanism of writing the files to a temporary location and its availability is published only when it has been completely written. By default, the suffix is written, but it can be changed using the temporary-file-suffix property. This can be completely disabled by setting use-temporary-file- name to false. FTP outbound gateway Gateway, by definition, is a two-way component: it accepts input and provides a result for further processing. So what is the input and output in the case of FTP? It issues commands to the FTP server and returns the result of the command. The following command will issue an ls command with the option –l to the server. The result is a list of string objects containing the filename of each file that will be put on the reply- channel. The code is as follows: <int-ftp:outbound-gateway id="ftpGateway"     session-factory="ftpClientSessionFactory"     request-channel="commandInChannel"     command="ls"     command-options="-1"     reply-channel="commandOutChannel"/> The tags are pretty simple: int-ftp:outbound-gateway: This is the namespace support for the FTP outbound gateway session-factory: This is the wrapper for details needed to connect to the FTP server command: This is the command to be issued command-options: This is the option for the command reply-channel: This is the response of the command that is put on this channel FTPS support For FTPS support, all that is needed is to change the factory class—an instance of org.springframework.integration.ftp.session.DefaultFtpsSessionFactory should be used. Note the s in DefaultFtpsSessionFactory. Once the session is created with this factory, it's ready to communicate over a secure channel. Here is an example of a secure session factory configuration: <bean id="ftpSClientFactory"   class="org.springframework.integration.ftp.session.   DefaultFtpsSessionFactory">   <property name="host" value="localhost"/>   <property name="port" value="22"/>   <property name="username" value="testuser"/>   <property name="password" value="testuser"/> </bean> Although it is obvious, I would remind you that the FTP server must be configured to support a secure connection and open the appropriate port. Social integration Any application in today's context is incomplete if it does not provide support for social messaging. Spring Integration provides in-built support for many social interfaces such as e-mails, Twitter feeds, and so on. Let's discuss the implementation of Twitter in this section. Prior to Version 2.1, Spring Integration was dependent on the Twitter4J API for Twitter support, but now it leverages Spring's social module for Twitter integration. Spring Integration provides an interface for receiving and sending tweets as well as searching and publishing the search results in messages. Twitter uses oauth for authentication purposes. An app must be registered before we start Twitter development on it. Prerequisites Let's look at the steps that need to be completed before we can use a Twitter component in our Spring Integration example: Twitter account setup: A Twitter account is needed. Perform the following steps to get the keys that will allow the user to use Twitter using the API: Visit https://apps.twitter.com/. Sign in to your account. Click on Create New App. Enter the details such as Application name, Description, Website, and so on. All fields are self-explanatory and appropriate help has also been provided. The value for the field Website need not be a valid one—put an arbitrary website name in the correct format. Click on the Create your application button. If the application is created successfully, a confirmation message will be shown and the Application Management page will appear, as shown here: Go to the Keys and Access Tokens tab and note the details for Consumer Key (API Key) and Consumer Secret (API Secret) under Application Settings, as shown in the following screenshot: You need additional access tokens so that applications can use Twitter using APIs. Click on Create my access token; it takes a while to generate these tokens. Once it is generated, note down the value of Access Token and Access Token Secret. Go to the Permissions tab and provide permission to Read, Write and Access direct messages. After performing all these steps, and with the required keys and access token, we are ready to use Twitter. Let's store these in the twitterauth.properties property file: twitter.oauth.apiKey= lnrDlMXSDnJumKLFRym02kHsy twitter.oauth.apiSecret= 6wlriIX9ay6w2f6at6XGQ7oNugk6dqNQEAArTsFsAU6RU8F2Td twitter.oauth.accessToken= 158239940-FGZHcbIDtdEqkIA77HPcv3uosfFRnUM30hRix9TI twitter.oauth.accessTokenSecret= H1oIeiQOlvCtJUiAZaachDEbLRq5m91IbP4bhg1QPRDeh The next step towards Twitter integration is the creation of a Twitter template. This is similar to the datasource or connection factory for databases, JMS, and so on. It encapsulates details to connect to a social platform. Here is the code snippet: <context:property-placeholder location="classpath: twitterauth.properties "/> <bean id="twitterTemplate" class=" org.springframework.social.   twitter.api.impl.TwitterTemplate ">   <constructor-arg value="${twitter.oauth.apiKey}"/>   <constructor-arg value="${twitter.oauth.apiSecret}"/>   <constructor-arg value="${twitter.oauth.accessToken}"/>   <constructor-arg value="${twitter.oauth.accessTokenSecret}"/> </bean> As I mentioned, the template encapsulates all the values. Here is the order of the arguments: apiKey apiSecret accessToken accessTokenSecret With all the setup in place, let's now do some real work: Namespace support can be added by using the following code snippet: <beans   twitter-template="twitterTemplate"   channel="twitterChannel"> </int-twitter:inbound-channel-adapter> The components in this code are covered in the following bullet points: int-twitter:inbound-channel-adapter: This is the namespace support for Twitter's inbound channel adapter. twitter-template: This is the most important aspect. The Twitter template encapsulates which account to use to poll the Twitter site. The details given in the preceding code snippet are fake; it should be replaced with real connection parameters. channel: Messages are dumped on this channel. These adapters are further used for other applications, such as for searching messages, retrieving direct messages, and retrieving tweets that mention your account, and so on. Let's have a quick look at the code snippets for these adapters. I will not go into detail for each one; they are almost similar to what have been discussed previously. Search: This adapter helps to search the tweets for the parameter configured in the query tag. The code is as follows: <int-twitter:search-inbound-channel-adapter id="testSearch"   twitter-template="twitterTemplate"   query="#springintegration"   channel="twitterSearchChannel"> </int-twitter:search-inbound-channel-adapter> Retrieving Direct Messages: This adapter allows us to receive the direct message for the account in use (account configured in Twitter template). The code is as follows: <int-twitter:dm-inbound-channel-adapter   id="testdirectMessage"   twitter-template="twiterTemplate"   channel="twitterDirectMessageChannel"> </int-twitter:dm-inbound-channel-adapter> Retrieving Mention Messages: This adapter allows us to receive messages that mention the configured account via the @user tag (account configured in the Twitter template). The code is as follows: <int-twitter:mentions-inbound-channel-adapter   id="testmentionMessage"   twitter-template="twiterTemplate"   channel="twitterMentionMessageChannel"> </int-twitter:mentions-inbound-channel-adapter> Sending tweets Twitter exposes outbound adapters to send messages. Here is a sample code:   <int-twitter:outbound-channel-adapter     twitter-template="twitterTemplate"     channel="twitterSendMessageChannel"/> Whatever message is put on the twitterSendMessageChannel channel is tweeted by this adapter. Similar to an inbound gateway, the outbound gateway provides support for sending direct messages. Here is a simple example of an outbound adapter: <int-twitter:dm-outbound-channel-adapter   twitter-template="twitterTemplate"   channel="twitterSendDirectMessage"/> Any message that is put on the twitterSendDirectMessage channel is sent to the user directly. But where is the name of the user to whom the message will be sent? It is decided by a header in the message TwitterHeaders.DM_TARGET_USER_ID. This must be populated either programmatically, or by using enrichers or SpEL. For example, it can be programmatically added as follows: Message message = MessageBuilder.withPayload("Chandan")   .setHeader(TwitterHeaders.DM_TARGET_USER_ID,   "test_id").build(); Alternatively, it can be populated by using a header enricher, as follows: <int:header-enricher input-channel="twitterIn"   output-channel="twitterOut">   <int:header name="twitter_dmTargetUserId" value=" test_id "/> </int:header-enricher> Twitter search outbound gateway As gateways provide a two-way window, the search outbound gateway can be used to issue dynamic search commands and receive the results as a collection. If no result is found, the collection is empty. Let's configure a search outbound gateway, as follows:   <int-twitter:search-outbound-gateway id="twitterSearch"     request-channel="searchQueryChannel"     twitter-template="twitterTemplate"     search-args-expression="#springintegration"     reply-channel="searchQueryResultChannel"/> And here is what the tags covered in this code mean: int-twitter:search-outbound-gateway: This is the namespace for the Twitter search outbound gateway request-channel: This is the channel that is used to send search requests to this gateway twitter-template: This is the Twitter template reference search-args-expression: This is used as arguments for the search reply-channel: This is the channel on which searched results are populated This gives us enough to get started with the social integration aspects of the spring framework. Enterprise messaging Enterprise landscape is incomplete without JMS—it is one of the most commonly used mediums of enterprise integration. Spring provides very good support for this. Spring Integration builds over that support and provides adapter and gateways to receive and consume messages from many middleware brokers such as ActiveMQ, RabbitMQ, Rediss, and so on. Spring Integration provides inbound and outbound adapters to send and receive messages along with gateways that can be used in a request/reply scenario. Let's walk through these implementations in a little more detail. A basic understanding of the JMS mechanism and its concepts is expected. It is not possible to cover even the introduction of JMS here. Let's start with the prerequisites. Prerequisites To use Spring Integration messaging components, namespaces, and relevant Maven the following dependency should be added: Namespace support can be added by using the following code snippet: > Maven entry can be provided using the following code snippet: <dependency>   <groupId>org.springframework.integration</groupId>   <artifactId>spring-integration-jms</artifactId>   <version>${spring.integration.version}</version> </dependency> After adding these two dependencies, we are ready to use the components. But before we can use an adapter, we must configure an underlying message broker. Let's configure ActiveMQ. Add the following in pom.xml:   <dependency>     <groupId>org.apache.activemq</groupId>     <artifactId>activemq-core</artifactId>     <version>${activemq.version}</version>     <exclusions>       <exclusion>         <artifactId>spring-context</artifactId>         <groupId>org.springframework</groupId>       </exclusion>     </exclusions>   </dependency>   <dependency>     <groupId>org.springframework</groupId>     <artifactId>spring-jms</artifactId>     <version>${spring.version}</version>     <scope>compile</scope>   </dependency> After this, we are ready to create a connection factory and JMS queue that will be used by the adapters to communicate. First, create a session factory. As you will notice, this is wrapped in Spring's CachingConnectionFactory, but the underlying provider is ActiveMQ: <bean id="connectionFactory" class="org.springframework.   jms.connection.CachingConnectionFactory">   <property name="targetConnectionFactory">     <bean class="org.apache.activemq.ActiveMQConnectionFactory">       <property name="brokerURL" value="vm://localhost"/>     </bean>   </property> </bean> Let's create a queue that can be used to retrieve and put messages: <bean   id="feedInputQueue"   class="org.apache.activemq.command.ActiveMQQueue">   <constructor-arg value="queue.input"/> </bean> Now, we are ready to send and retrieve messages from the queue. Let's look into each message one by one. Receiving messages – the inbound adapter Spring Integration provides two ways of receiving messages: polling and event listener. Both of them are based on the underlying Spring framework's comprehensive support for JMS. JmsTemplate is used by the polling adapter, while MessageListener is used by the event-driven adapter. As the name suggests, a polling adapter keeps polling the queue for the arrival of new messages and puts the message on the configured channel if it finds one. On the other hand, in the case of the event-driven adapter, it's the responsibility of the server to notify the configured adapter. The polling adapter Let's start with a code sample: <int-jms:inbound-channel-adapter   connection-factory="connectionFactory"   destination="feedInputQueue"   channel="jmsProcessedChannel">   <int:poller fixed-rate="1000" /> </int-jms:inbound-channel-adapter> This code snippet contains the following components: int-jms:inbound-channel-adapter: This is the namespace support for the JMS inbound adapter connection-factory: This is the encapsulation for the underlying JMS provider setup, such as ActiveMQ destination: This is the JMS queue where the adapter is listening for incoming messages channel: This is the channel on which incoming messages should be put There is a poller element, so it's obvious that it is a polling-based adapter. It can be configured in one of two ways: by providing a JMS template or using a connection factory along with a destination. I have used the latter approach. The preceding adapter has a polling queue mentioned in the destination and once it gets any message, it puts the message on the channel configured in the channel attribute. The event-driven adapter Similar to polling adapters, event-driven adapters also need a reference either to an implementation of the interface AbstractMessageListenerContainer or need a connection factory and destination. Again, I will use the latter approach. Here is a sample configuration: <int-jms:message-driven-channel-adapter   connection-factory="connectionFactory"   destination="feedInputQueue"   channel="jmsProcessedChannel"/> There is no poller sub-element here. As soon as a message arrives at its destination, the adapter is invoked, which puts it on the configured channel. Sending messages – the outbound adapter Outbound adapters convert messages on the channel to JMS messages and put them on the configured queue. To convert Spring Integration messages to JMS messages, the outbound adapter uses JmsSendingMessageHandler. This is is an implementation of MessageHandler. Outbound adapters should be configured with either JmsTemplate or with a connection factory and destination queue. Keeping in sync with the preceding examples, we will take the latter approach, as follows: <int-jms:outbound-channel-adapter   connection-factory="connectionFactory"   channel="jmsChannel"   destination="feedInputQueue"/> This adapter receives the Spring Integration message from jmsChannel, converts it to a JMS message, and puts it on the destination. Gateway Gateway provides a request/reply behavior instead of a one-way send or receive. For example, after sending a message, we might expect a reply or we may want to send an acknowledgement after receiving a message. The inbound gateway Inbound gateways provide an alternative to inbound adapters when request-reply capabilities are expected. An inbound gateway is an event-based implementation that listens for a message on the queue, converts it to Spring Message, and puts it on the channel. Here is a sample code: <int-jms:inbound-gateway   request-destination="feedInputQueue"   request-channel="jmsProcessedChannel"/> However, this is what an inbound adapter does—even the configuration is similar, except the namespace. So, what is the difference? The difference lies in replying back to the reply destination. Once the message is put on the channel, it will be propagated down the line and at some stage a reply would be generated and sent back as an acknowledgement. The inbound gateway, on receiving this reply, will create a JMS message and put it back on the reply destination queue. Then, where is the reply destination? The reply destination is decided in one of the following ways: Original message has a property JMSReplyTo, if it's present it has the highest precedence. The inbound gateway looks for a configured, default-reply-destination which can be configured either as a name or as a direct reference of a channel. For specifying channel as direct reference default-reply-destination tag should be used. An exception will be thrown by the gateway if it does not find either of the preceding two ways. The outbound gateway Outbound gateways should be used in scenarios where a reply is expected for the send messages. Let's start with an example: <int-jms:outbound-gateway   request-channel="jmsChannel"   request-destination="feedInputQueue"   reply-channel="jmsProcessedChannel" /> The preceding configuration will send messages to request-destination. When an acknowledgement is received, it can be fetched from the configured reply-destination. If reply-destination has not been configured, JMS TemporaryQueues will be created. Summary In this article, we covered out-of-the-box component provided by the Spring Integration framework such as aggregator. This article also showcased the simplicity and abstraction that Spring Integration provides when it comes to handling complicated integrations, be it file-based, HTTP, JMS, or any other integration mechanism. Resources for Article: Further resources on this subject: Modernizing Our Spring Boot App [article] Home Security by Beaglebone [article] Integrating With Other Frameworks [article]
Read more
  • 0
  • 0
  • 5607

article-image-postgresql-9-balancing-hardware-spending
Packt
27 Oct 2010
10 min read
Save for later

PostgreSQL 9: Balancing Hardware Spending

Packt
27 Oct 2010
10 min read
One of the reasons that working with open-source databases such as PostgreSQL can be so effective is that every dollar you save on software licensing can be put toward better hardware instead. The three main components you'll need to balance in your budget are CPUs, memory, and disks, with the disk controller as a related and critical part too. CPUs Currently, available processors are bundling at least two and possibly as many as eight cores into each CPU, making the core count the figure of merit for most database applications. There are two basic decisions you need to make while deciding which CPU solution would best match your database application: Which processor family? Nowadays, this normally boils down to choosing among the various 64 bit product lines from Intel or AMD, although there are some other less popular choices still floating around (Itanium, SPARC, and so on). Do you get more cores or faster cores? These choices are sometimes more tied together than you might think. Currently, Intel has a lead in delivering individual processor cores that are the fastest around, often due to faster transfers between the processor and system RAM. But the processors and related parts are more expensive too. AMD still is competitive at providing more cores per dollar, and their server class designs normally do a good job making the best of the memory available to each core. But if what you want is many more affordable cores instead, that's where AMD is stronger. AMD also has a better history of making its fastest processors available in configurations with many sockets, when you want to put more than two physical CPUs into a single server. The best way to figure out which class of database app you have—more cores or faster cores—is to monitor an existing server using tools such as top. If there's a small number of processes running using a single CPU each, that's the sort of workload where faster cores are better. That tends to happen if you have giant queries running in a batch fashion, for example when large quantities of data need to be sorted to deliver any single report. But if all the CPUs are active with many more concurrent processes instead, then you'd likely benefit better from more cores. That's normally what you'll see in applications with a larger user count, such as databases backing web applications. If you don't have the benefit of a working system to inspect, you can try to guess which type of situation you're in by noting the limitations of the database. PostgreSQL does not allow splitting a single query across more than one core, what's called parallel query by some other databases that support it. That means that if you have any one query or small number of queries that must run as fast as possible, the only way to do that is to prioritize getting faster cores. Another situation where getting a faster core is the better choice is if you need to prioritize data loading or export situations. PostgreSQL's best performing data import method, COPY, can easily become (but isn't always) limited by CPU performance, where that turns into the bottleneck for operations. While it's possible to split input files into pieces and load them in parallel, that's something you'll need to build or acquire yourself, rather than something the server knows how to do for you. Exporting a copy of the database using the pg_dump utility is another example of something that can become CPU limited on some systems. Memory How much to prioritize memory for your application really depends on the size of the working set of data needed to handle the most common operations. Generally, adding more RAM will provide a significant performance boost. There are a few situations where you'd be better served doing something else instead: If your data set is small enough to fit into a smaller amount of RAM, adding more won't help you much. You probably want faster processors instead. When running applications that scan tables much larger than what you can feasibly purchase as RAM, such as in many data warehouse situations, you might be better served by getting faster disks rather than more memory. The normal situation where more memory helps most is when the data you access frequently will fit with the larger amount, but not with the smaller. This happens more often than you might think because of the way database B-tree indexes are stored. Even if you can't fit the entirety of a table or even its index in memory, being able to store a good sized fraction of the index can mean that index-based data lookups will be significantly sped up. Having the most "popular" blocks from the top of the tree structure cached is helpful even if you can't fit all the leaves into memory too. Once you have an application running, you can usually get a much better idea how memory is being used by looking inside the PostgreSQL buffer cache (and potentially inside the operating system one as well) and seeing what data it prefers to keep around. Disks While it's always possible to run into situations where the CPU in your database server is its bottleneck, it's downright likely that you'll run into a disk bottleneck— particularly if you only have a drive or two in the system. A few years ago, the basic two choices in hard drives were the inexpensive ATA (also known as IDE) drives used in desktops versus the more serious SCSI drives aimed at servers. Both technologies have marched forward, and the current choice you're most likely to run into when configuring a database server is whether to use Serial ATA (SATA) or Serial Attached SCSI (SAS). It's possible to find nearly identical drives available in both interfaces, and there are even drive controllers that allow attaching either kind of drive. Combined with a narrowing performance difference between the two, picking between them is harder than ever. The broad parameters of each technology are straightforward to compare. Here's the state of things as this is being written: SAS disks: The maximum available RPM is higher: 10,000 or 15,000 Not as much drive capacity: 73 GB-1 TB are popular sizes Cost per MB is higher SATA disks: Drives typically have a slower RPM: 7200 is standard, some 10,000 designs exist such as the Western Digital VelociRaptor Higher drive capacity: 2 TB available Cost per MB is lower Generally, you'll find individual SAS disks to be faster even than SATA ones with similar specifications. In particular, you're likely to see better seek performance on random I/O due to faster drive mechanics in SAS, and sometimes a faster transfer rate from the disk too. Also, because the SAS drives have supported advanced features such as command queuing for longer, it's more likely your operating system will have matching support to take advantage of them. RAID The Redundant Array of Inexpensive Disks (RAID) approach is the standard way to handle both the performance and reliability limitations of individual disk drives. A RAID array puts many disks, typically of exactly the same configuration, into a set that acts like a single disk—but with either enhanced performance, reliability, or both. In some cases the extra reliability comes from computing what's called parity information for writes to the array. Parity is a form of checksum on the data, which allows reconstructing it even if some of the information is lost. RAID levels that use parity are efficient from a space perspective at writing data in a way that will survive drive failures, but the parity computation overhead can be significant for database applications. The most common basic forms of RAID arrays used are: RAID 0: It is also called as Striping. Multiple disks are used at the same time, spreading reads and writes over each of them in parallel. This can be almost a linear improvement (two disks reading twice as fast as a single one), but a failure on any volume in the set will lose all the data. RAID 1: It is also called as Mirroring. Here more copies of the same data are put onto multiple disks. This can sometimes improve performance—a good RAID 1 mirroring across two disks might handle two reads by sending one to each drive. Reads executed in parallel against both drives can effectively double average seeks per second. But generally, the reason for RAID 1 is redundancy: if a single drive fails, the system will continue operating using the other one. RAID 10 or 1+0: This first takes pairs of disks and mirrors then using RAID 1. Then, the resulting set is striped using RAID 0. The result provides both high performance and the ability to tolerate any single disk failure, without as many ways for speed to suffer in the average and worst case as RAID 5/6. RAID 10 is particularly appropriate for write-heavy environments, where the parity computation overhead of RAID 5/6 can cause disk performance to suffer. Accordingly, it's the preferred RAID level for high-performance database systems. RAID 5: It is also called as Striped with Parity. This approach sits midway between 0 and 1. You stripe data across multiple drives similarly to RAID 0, which improves read performance. But some redundant data is added to a parity drive. If one of the disks in the array is lost, the missing data can be recomputed from the ones left using that parity information. While this is efficient in terms of how little space is wasted relative to the tolerance for disk failures provided, write performance in particular can suffer in RAID 5. RAID 6: Similar to RAID 5, except with more parity information, enabling survival even with two disk failures. It has the same fundamental advantages and disadvantages. RAID 6 is an increasingly common way to cope with the fact that rebuilding a RAID 5 array after a disk loss can take a really long time on modern, high capacity drives. The array has no additional fault tolerance during that period, and seeing a second drive failure before that rebuild finishes is not that unlikely when it takes many hours of intense disk activity to rebuild. Disks manufactured in the same batch are surprisingly likely to fail in groups. To be fair in any disk performance comparison, you need to consider that most systems are going to have a net performance from several disks, such as in a RAID array. Since SATA disks are individually cheaper, you might be able to purchase considerably more of them for the same budget than had you picked SAS instead. If your believe your application will get faster if it is spread over more disks, being able to buy more of them per dollar spent can result in an overall faster system. Note that the upper limit here will often be your server's physical hardware. You only have so many storage bays or controllers ports available, and larger enclosures can cost more both up-front and over their lifetime. It's easy to find situations where smaller numbers of faster drives—which SAS provides—is the better way to go. This is why it's so important to constantly benchmark both hardware and your database application, to get a feel of how well it improves as the disk count increases.
Read more
  • 0
  • 0
  • 5604

article-image-testing-and-tracing-applications
Packt
22 Apr 2014
7 min read
Save for later

Testing and Tracing Applications

Packt
22 Apr 2014
7 min read
(For more resources related to this topic, see here.) Tracing RabbitMQ Tracing the execution of a program is a convenient way to figure out what is really happening under the hood when reasoning about a particular behavior leads to no firm conclusion. Usually, the buck stops at the border where the application interacts with external resources such as the RabbitMQ broker. The good news is that RabbitMQ provides two tools that can be of tremendous help when it comes to tracing the interactions with a broker. The first of these tools is Tracer, an AMQP-aware network proxy that can be placed between a RabbitMQ client and a broker in order to gain insight into the interactions that are happening between each other. Tracer is available as part of the Java client download available at http://www.rabbitmq.com/download.html. After installation, Tracer can be started with the following: runjava.sh com.rabbitmq.tools.Tracer [listenPort] [connectHost] [connectPort]. All the parameters are optional. If left blank, Tracer will start a local proxy listening on port 5673 and connect to a local RabbitMQ on port 5672. Since you're happy with the defaults, you start Tracer with just the following command line: $ ./runjava.sh com.rabbitmq.tools.Tracer Now you can run the integration tests you've just created through this proxy. Do you remember that we made the connection information configurable on these tests? The approach is going to pay off now as we will configure them to go through the proxy port instead of directly hitting the RabbitMQ broker. You do this by running the following command line: $ mvn -Pintegration_tests -Dtest.rmq.addresses=localhost:5673 verify The output of Tracer is very verbose as it includes the complete details of the AMQP operations. Hereafter only the columns that show the channel ID, interaction direct (-> is client to broker and <- is opposite), and the name of the operation is reproduced, with the interactions related to the subscriber highlighted: ch#0 <- <connection.start> ch#0 -> <connection.start-ok> ch#0 <- <connection.tune> ch#0 -> <connection.tune-ok> ch#0 -> <connection.open> ch#0 <- <connection.open-ok> ch#1 -> <channel.open> ch#1 <- <channel.open-ok> ch#1 -> <queue.declare> ch#1 <- <queue.declare-ok> ch#1 -> <channel.close> ch#1 <- <channel.close-ok>() ch#1 -> <channel.open> ch#1 <- <channel.open-ok> ch#1 -> <basic.consume> ch#1 <- <basic.consume-ok> ch#2 -> <channel.open> ch#2 <- <channel.open-ok> ch#2 -> <basic.publish> ch#2 -> <channel.close> ch#1 <- <basic.deliver> ch#2 <- <channel.close-ok> ch#1 -> <basic.cancel> ch#1 <- <basic.cancel-ok> ch#1 -> <channel.close> ch#1 <- <channel.close-ok> ch#0 -> <connection.close> ch#0 <- <connection.close-ok> You can see the operations from the client and the responses from the broker, typically being named after the operation and suffixed with -ok. In essence, the following is the AMQP synopsis of the test code you're running: Establish a connection Open a channel, use it to declare the test queue, and close it Open a channel, use it to consume a queue Open a channel, use it to publish the test message, and close it Receive the message delivery, cancel the consumer, and close its channel Close the connection Notice how the connection start and tune operations are initiated by the broker as a response to establishing the connection to it. Also, notice that the channel number gets reused after being closed; it may seem that the same channel #1 has been used for creating the test queue and subscribing to it, but that's not the case since this channel has been explicitly closed. Only its identifier has been reused. Tracer is a very powerful tool to easily gain a deep understanding of the AMQP protocol and the usage your applications make of it. However, it requires you to insert a proxy between a client and the broker it connects to. Fear not if this is an issue; RabbitMQ has more than one trick in its bag of tracing tools. Drinking at the Firehose RabbitMQ offers the possibility of spying on all message publications and delivery operations that happens in a particular virtual host of a broker. This feature is called the Firehose tracer. When activated on a virtual host, a copy of all published and all delivered messages is sent to the amq.rabbitmq.trace exchange (which is automatically created in every virtual host). The routing key used for messages published to the amq.rabbitmq.trace exchange is publish.<exchange_name> for publication events and deliver.<queue_name> for message deliveries. The original message body is carried to the copies sent to this exchange. Extra information about the original publication or delivery event are added in a set of headers, including exchange_name for the name of the exchange where the message was originally published or redelivered if the message has been delivered more than once. You want to use the Firehose when running the integration tests to see the exchanged messages from the broker's standpoint. Before activating the Firehose on RabbitMQ, you need first to create a client application that will subscribe to the exchange and print out the messages that come to it. For this, you create the following Python script: #!/usr/bin/env python import amqp connection = amqp.Connection(host='localhost', userid='ccm-dev', password='coney123', virtual_host='ccm-dev-vhost') channel = connection.channel() EXCHANGE = 'amq.rabbitmq.trace' QUEUE = 'firehose-queue' channel.queue_declare(queue=QUEUE, durable=False, auto_delete=True, exclusive=True) channel.queue_bind(queue=QUEUE, exchange=EXCHANGE, routing_key='#') def handle_message(message): print message.routing_key, '->', message.properties, message.body print '--------------------------------' channel.basic_consume(callback=handle_message, queue=QUEUE, no_ack=True) print ' [*] Waiting for messages. To exit press CTRL+C' while channel.callbacks: channel.wait() channel.close() connection.close() After starting this script, you turn the Firehose on by running the following command line: $ sudo rabbitmqctl -p ccm-dev-vhost trace_on Starting tracing for vhost "ccm-dev-vhost" ... ...done. Now you can run the integration tests again, this time on the standard port since no proxying is needed with the Firehose: $ mvn -Pintegration_tests verify Let's now look at the following output of the Firehose consumer Python script: publish. -> {'application_headers': {u'node': u'rabbit@pegasus', u'exchange_name': u'', u'routing_keys': [u'amq.gen-vTMWL--04lap8s8JPbX5gA'], u'properties': {}}} 93b56787-b4f5-41e1-8c6f-d5f9b64275ca -------------------------------- deliver.amq.gen-vTMWL--04lap8s8JPbX5gA -> {'application_headers': {u'node': u'rabbit@pegasus', u'exchange_name': u'', u'redelivered': 0, u'routing_keys': u'amq.gen-vTMWL--04lap8s8JPbX5gA'], u'properties': {}}} 93b56787-b4f5-41e1-8c6f-d5f9b64275ca -------------------------------- As you can see, the publication to the default exchange (remember, its name is an empty string) and the delivery to the automatically named test queue are clearly visible. All the details that concern them are readily available in the message properties. Keep in mind that running the Firehose is taxing for the RabbitMQ broker, so when you're done with your tracing session, shut it down with the following: $ sudo rabbitmqctl -p ccm-dev-vhost trace_off Stopping tracing for vhost "ccm-dev-vhost" ... ...done. The Firehose will come handy when tracing what's happening between your different applications and your RabbitMQ brokers in depth. Keep in mind that using unique message IDs, as you've learned throughout this article, will help you a lot when the time comes to perform forensics analysis and trace the progression of messages across your complete infrastructure. Summary In this article, you have discovered powerful tracing tools to peek deeper under the hood of the AMQP protocol and the RabbitMQ broker. Resources for Article: Further resources on this subject: RabbitMQ Acknowledgements [Article] Troubleshooting in OpenStack Cloud Computing [Article] Introduction to Cloud Computing with Microsoft Azure [Article]
Read more
  • 0
  • 0
  • 5602

article-image-integrating-biztalk-server-and-microsoft-dynamics-crm
Packt
20 Jul 2011
7 min read
Save for later

Integrating BizTalk Server and Microsoft Dynamics CRM

Packt
20 Jul 2011
7 min read
What is Microsoft Dynamics CRM? Customer relationship management is a critical part of virtually every business. Dynamics CRM 2011 offers a solution for the three traditional areas of CRM: sales, marketing, and customer service. For customers interested in managing a sales team, Dynamics CRM 2011 has a strong set of features. This includes organizing teams into territories, defining price lists, managing opportunities, maintaining organization structures, tracking sales pipelines, enabling mobile access, and much more. If you are using Dynamics CRM 2011 for marketing efforts, then you have the ability to import data from multiple sources, plan campaigns and set up target lists, create mass communications, track responses to campaigns, share leads with the sales team, and analyze the success of a marketing program. Dynamics CRM 2011 also serves as a powerful hub for customer service scenarios. Features include rich account management, case routing and management, a built-in knowledge base, scheduling of call center resources, scripted Q&A workflows called Dialogs, contract management, and more. Besides these three areas, Microsoft pitches Dynamics CRM as a general purpose application platform called xRM, where the "X" stands for any sort of relationship management. Dynamics CRM has a robust underlying framework for screen design, security roles, data auditing, entity definition, workflow, and mobility, among others. Instead of building these foundational aspects into every application, we can build our data-driven applications within Dynamics CRM. Microsoft has made a big move into the cloud with this release of Dynamics CRM 2011. For the first time in company history, a product was released online (Dynamics CRM Online) prior to on-premises software. The hosted version of the application runs an identical codebase to the on-premises version meaning that code built to support a local instance will work just fine in the cloud. In addition to the big play in CRM hosting, Microsoft has also baked Windows Azure integration into Dynamics CRM 2011. Specifically, we now have the ability to configure a call-out to an Azure AppFabric Service Bus endpoint. To do this, the downstream service must implement a specific WCF interface and within CRM, the Azure AppFabric plugin is configured to call that downstream service through the Azure AppFabric Service Bus relay service. For BizTalk Server to accommodate this pattern, we would want to build a proxy service that implements the required Dynamics CRM 2011 interface and forwards requests into a BizTalk Server endpoint. This article will not demonstrate this scenario, however, as the focus will be on integrating with an onpremises instance only. Why Integrate Dynamics CRM and BizTalk Server? There are numerous reasons to tie these two technologies together. Recall that BizTalk Server is an enterprise integration bus that connects disparate applications. There can be a natural inclination to hoard data within a particular application, but if we embrace real-time message exchange, we can actually have a more agile enterprise. Consider a scenario when a customer's full "contact history" resides in multiple systems. The Dynamics CRM 2011 contact center may only serve a specific audience, and other systems within the company hold additional details about the company's customers. One design choice could be to bulk load that information into Dynamics CRM 2011 on a scheduled interval. However, it may be more effective to call out to a BizTalk Server service that aggregates data across systems and returns a composite view of a customer's history with a company. In a similar manner, think about how information is shared between systems. A public website for a company may include a registration page where visitors sign up for more information and deeper access to content. That registration event is relevant to multiple systems within the company. We could send that initial registration message to BizTalk Server and then broadcast that message to the multiple systems that want to know about that customer. A marketing application may want to respond with a personalized email welcoming that person to the website. The sales team may decide to follow up with that person if they expressed interest in purchasing products. Our Dynamics CRM 2011 customer service center could choose to automatically add the registration event so that it is ready whenever that customer calls in. In this case, BizTalk Server acts as a central router of data and invokes the exposed Dynamics CRM services to create customers and transactions. Communicating from BizTalk Server to Dynamics CRM The way that you send requests from BizTalk Server to Dynamics CRM 2011 has changed significantly in this release. In the previous versions of Dynamics CRM, a BizTalk "send" adapter was available for communicating with the platform. Dynamics CRM 2011 no longer ships with an adapter and developers are encouraged to use the WCF endpoints exposed by the product. Dynamics CRM has both a WCF REST and SOAP endpoint. The REST endpoint can only be used within the CRM application itself. For instance, you can build what is called a web resource that is embedded in a Dynamics CRM page. This resource could be a Microsoft Silverlight or HTML page that looks up data from three different Dynamics CRM entities and aggregates them on the page. This web resource can communicate with the Dynamics CRM REST API, which is friendly to JavaScript clients. Unfortunately, you cannot use the REST endpoint from outside of the Dynamics CRM environment, but because BizTalk cannot communicate with REST services, this has little impact on the BizTalk integration story. The Dynamics CRM SOAP API, unlike its ASMX web service predecessor, is static and operates with a generic Entity data structure. Instead of having a dynamic WSDL that exposes typed definitions for all of the standard and custom entities in the system, the Dynamics CRM 2011 SOAP API has a set of operations (for example, Create, Retrieve) that function with a single object type. The Entity object has a property identifying which concrete object it represents (for example, Account or Contract), and a name/value pair collection that represents the columns and values in the object it represents. For instance, an Entity may have a LogicalName set to "Account" and columns for "telephone1", "emailaddress", and "websiteurl." In essence, this means that we have two choices when interacting with Dynamics CRM 2011 from BizTalk Server. Our first option is to directly consume and invoke the untyped SOAP API. Doing this involves creating maps from a canonical schema to the type-less Entity schema. In the case of doing a Retrieve operation, we may also have to map the type-less Entity message back to a structured message for more processing. Below, we will walk through an example of this. The second option involves creating a typed proxy service for BizTalk Server to invoke. Dynamics CRM has a feature-rich Solution Development Kit (SDK) that allows us to create typed objects and send them to the Dynamics CRM SOAP endpoint. This proxy service will then expose a typed interface to BizTalk that operates as desired with a strongly typed schema. An upcoming exercise demonstrates this scenario. Which choice is best? For simple solutions, it may be fine to interact directly with the Dynamics CRM 2011 SOAP API. If you are updating a couple fields on an entity, or retrieving a pair of data values, the messiness of the untyped schema is worth the straightforward solution. However, if you are making large scale changes to entities, or getting back an entire entity and publishing to the BizTalk bus for more subscribers to receive, then working strictly with a typed proxy service is the best route. However, we will look at both scenarios below, and you can make that choice for yourself. Integrating Directly with the Dynamics CRM 2011 SOAP API In the following series of steps, we will look at how to consume the native Dynamics CRM SOAP interface in BizTalk Server. We will first look at how to query Dynamics CRM to return an Entity. After that, we will see the steps for creating a new Entity in Dynamics CRM. Querying Dynamics CRM from BizTalk Server In this scenario, BizTalk Server will request details about a specific Dynamics CRM "contact" record and send the result of that inquiry to another system.
Read more
  • 0
  • 0
  • 5601

article-image-themes-and-templates-apache-struts-2
Packt
20 Oct 2009
10 min read
Save for later

Themes and Templates with Apache Struts 2

Packt
20 Oct 2009
10 min read
Extracting the templates The first step to modifying an existing theme or creating our own is to extract the templates from the Struts 2 distribution. This actually has the advantageous performance side effect of keeping the templates in the file system (as opposed to in the library file), which allows FreeMarker to cache the templates properly. Caching the templates provides a performance boost and involves no work other than extracting the templates. The issue with caching templates contained in library files, however, will be fixed. If we examine the Struts 2 core JAR file, we'll see a /template folder. We just need to put that in our application's classpath. The best way to do this depends on your build and deploy environment. For example, if we're using Eclipse, the easiest thing to do is put the /template folder in our source folder; Eclipse should deploy them automatically. A maze of twisty little passages Right now, consider a form having only text fields and a submit button. We'll start by looking at the template for the text field tag. For the most part, Struts 2 custom tags are named similarly to the template file that defines it. As we're using the "xhtml" theme, we'll look in our newly-created /template/xhtml folder. Templates are found in a folder with the same name as the theme. We find the <s:textfield> template in /template/xhtml/text.ftl file. However, when we open it, we are disappointed to find it implemented by the following files—controlheader.ftl file retrieved from the current theme's folder, text.ftl from the simple theme, and controlfooter.ftl file from "xhtml" theme. This is curious, but satisfactory for now. We'll assume what we need is in the controlheader.ftl file. However, upon opening that, we discover we actually need to look in controlheader-core.ftl file. Opening that file shows us the table rows that we're looking for. Going walkabout through source code, both Java and FreeMarker, can be frustrating, but ultimately educational. Developing the habit of looking at framework source can lead to a greater mastery of that framework. It can be frustrating at times, but is a critical skill. Even without a strong understanding of the FreeMarker template language, we can get a pretty good idea of what needs to be done by looking at the controlheader-core.ftl file. We notice that the template sets a convenience variable (hasFieldErrors) when the field being rendered has an error. We'll use that variable to control the style of the table row and cells of our text fields. This is how the class of the text field label is being set. Creating our theme To keep the template clean for the purpose of education, we'll go ahead and create a new theme. (Most of the things will be the same, but we'll strip out some unused code in the templates we modify.) While we have the possibility of extending an existing theme (see the Struts 2 documentation for details), we'll just create a new theme called s2wad by copying the xhtml templates into a folder called s2wad. We can now use the new theme by setting the theme in our <s:form> tag by specifying a theme attribute: <s:form theme="s2wad" ... etc ...> Subsequent form tags will now use our new s2wad theme. Because we decided not to extend the existing "xhtml" theme, as we have a lot of tags with the "xhtml" string hard coded inside. In theory, it probably wasn't necessary to hard code the theme into the templates. However, we're going to modify only a few tags for the time being, while the remaining tags will remain hard coded (although incorrectly). In an actual project, we'd either extend an existing theme or spend more time cleaning up the theme we've created (along with the "xhtml" theme, and provide corrective patches back to the Struts 2 project). First, we'll modify controlheader.ftl to use the theme parameter to load the appropriate controlheader-core.ftl file. Arguably, this is how the template should be implemented anyway, even though we could hard code in the new s2wad theme. Next, we'll start on controlheader-core.ftl. As our site will never use the top label position, we'll remove that. Doing this isn't necessary, but will keep it cleaner for our use. The controlheader-core.ftl template creates a table row for each field error for the field being rendered, and creates the table row containing the field label and input field itself. We want to add a class to both the table row and table cells containing the field label and input field. By adding a class to both, the row itself and each of the two table cells, we maximize our ability to apply CSS styles. Even if we end up styling only one or the other, it's convenient to have the option. We'll also strip out the FreeMarker code that puts the required indicator to the left of the label, once again, largely to keep things clean. Projects will normally have a unified look and feel. It's reasonable to remove unused functionality, and if we're already going through the trouble to create a new theme, then we might as well do that. We're also going to clean up the template a little bit by consolidating how we handle the presence of field errors. Instead of putting several FreeMarker <#if> directives throughout the template, we'll create some HTML attributes at the top of the template, and use them in the table row and table cells later on. Finally, we'll indent the template file to make it easier to read. This may not always be a viable technique in production, as the extra spaces may be rendered improperly, (particularly across browsers), possibly depending on what we end up putting in the tag. For now, imagine that we're using the default "required" indicator, an asterisk, but it's conceivable we might want to use something like an image. Whitespace is something to be aware of when dealing with HTML. Our modified controlheader-core.ftl file now looks like this: <#assign hasFieldErrors = parameters.name?exists && fieldErrors?exists && fieldErrors[parameters.name]?exists/><#if hasFieldErrors> <#assign labelClass = "class='errorLabel'"/> <#assign trClass = "class='hasErrors'"/> <#assign tdClass = "class='tdLabel hasErrors'"/><#else> <#assign labelClass = "class='label'"/> <#assign trClass = ""/> <#assign tdClass = "class='tdLabel'"/></#if><#if hasFieldErrors> <#list fieldErrors[parameters.name] as error> <tr errorFor="${parameters.id}" class="hasErrors"> <td>&nbsp;</td> <td class="hasErrors"><#rt/> <span class="errorMessage">${error?html}</span><#t/> </td><#lt/> </tr> </#list></#if><tr ${trClass}> <td ${tdClass}> <#if parameters.label?exists> <label <#t/> <#if parameters.id?exists> for="${parameters.id?html}" <#t/> </#if> ${labelClass} ><#t/> ${parameters.label?html}<#t/> <#if parameters.required?default(false)> <span class="required">*</span><#t/> </#if> :<#t/> <#include "/${parameters.templateDir}/s2e2e/tooltip.ftl" /> </label><#t/> </#if> </td><#lt/> It's significantly different when compared to the controlheader-core.ftl file of the "xhtml" theme. However, it has the same functionality for our application, with the addition of the new hasErrors class applied to both the table row and cells for the recipe's name and description fields. We've also slightly modified where the field errors are displayed (it is no longer centered around the entire input field row, but directly above the field itself). We'll also modify the controlheader.ftl template to apply the hasErrors style to the table cell containing the input field. This template is much simpler and includes only our new hasErrors class and the original align code. Note that we can use the variable hasFieldErrors, which is defined in controlheader-core.ftl. This is a valuable technique, but has the potential to lead to spaghetti code. It would probably be better to define it in the controlheader.ftl template. <#include "/${parameters.templateDir}/${parameters.theme}/controlheader-core.ftl" /><td<#if hasFieldErrors>class="hasErrors"<#t/></#if><#if parameters.align?exists>align="${parameters.align?html}"<#t/></#if>><#t/> We'll create a style for the table cells with the hasErrors class, setting the background to be just a little red. Our new template sets the hasErrors class on both the label and the input field table cells, and we've collapsed our table borders, so this will create a table row with a light red background. .hasErrors td {background: #fdd;} Now, a missing Name or Description will give us a more noticeable error, as shown in the following screenshot: This is fairly a simple example. However, it does show that it's pretty straightforward to begin customizing our own templates to match the requirements of the application. By encapsulating some of the view layer inside the form tags, our JSP files are kept significantly cleaner. Other uses of templates Anything we can do in a typical JSP page can be done in our templates. We don't have to use Struts 2's template support. We can do many similar things in a JSP custom tag file (or a Java-based tag), but we'd lose some of the functionality that's already been built. Some potential uses of templates might include the addition of accessibility features across an entire site, allowing them to be encapsulated within concise JSP notation. Enhanced JavaScript functionality could be added to all fields, or only specific fields of a form, including things such as detailed help or informational pop ups. This overlaps somewhat with the existing tooltip support, we might have custom usage requirements or our own framework that we need to support. Struts 2 now also ships with a Java-based theme that avoids the use of FreeMarker tags. These tags provide a noticeable speed benefit. However, only a few basic tags are supported at this time. It's bundled as a plug-in, which can be used as a launching point for our own Java-based tags. Summary Themes and templates provide another means of encapsulating functionality and/or appearance across an entire application. The use of existing themes can be a great benefit, particularly when doing early prototyping of a site, and are often sufficient for the finished product. Dealing effectively with templates is largely a matter of digging through the existing template source. It also includes determining what our particular needs are, and modifying or creating our own themes, adding and removing functionality as appropriate. While this article only takes a brief look at templates, it covers the basics and opens the door to implementing any enhancements we may require. If you have read this article you may be interested to view : Exceptions and Logging in Apache Struts 2 Documenting our Application in Apache Struts 2 (part 1) Documenting our Application in Apache Struts 2 (part 2)
Read more
  • 0
  • 0
  • 5601

article-image-oracle-b2b-overview
Packt
17 Sep 2013
12 min read
Save for later

Oracle B2B Overview

Packt
17 Sep 2013
12 min read
B2B environment setup Here is the list of some OFM concepts that will be used in this article: Domain: It is the basic administration unit that includes a special WebLogic Server instance called the Administration Server, and optionally one or many Java components. Java component: It is a Java EE application deployed to an Oracle WebLogic Server domain as part of a domain template. For example, SOA Suite is a Java component. Managed server: It is an additional WebLogic Server included in a domain, to host Java components such as SOA Suite. We will use the UNIX operating system for our tutorials. The following table depicts the directory environment variables used throughout the article for configuring the Oracle SOA Suite deployment: Name Variable What It Is Example Middleware home MW_HOME The top-level directory for all OFM products WebLogic Server home WL_HOME Contains installed files necessary to host a WebLogic Server $MW_HOME/wlserver_10.3 Oracle home SOA_ORACLE_HOME Oracle SOA Suite product directory $MW_HOME/Oracle_SOA1 Oracle Common Home ORACLE_COMMON_HOME Contains the binary and library files required for the Oracle Enterprise Manager Fusion Middleware Control and Java Required Files (JRF) $MW_HOME/oracle_common Domain home SOA_DOMAIN_HOME The absolute path of the source domain containing the SOA Suite Java component $MW_HOME/user_projects/domains/SOADomain Java home JAVA_HOME Specifies the location of JDK (must be 1.6.04 or higher) or JRockit $MW_HOME/jdk160_29 Ant Home ANT_HOME Specifies the location of Ant archive location $MW_HOME/org.apache.ant_1.7.1 The following figure depicts a snapshot of the SOA Suite directory's hierarchical structure: For the recommended SOA Suite directory location, please refer to the OFM Enterprise Development guide for SOA Suite that can be found at http://docs.oracle.com/cd/E16764_01/core.1111/e12036/toc.htm. JDeveloper installation tips JDeveloper is a development tool that will be used in the article. It is a full service Integrated Development Environment (IDE), which allows for the development of SOA projects along with a host of other Oracle products, including Java. If it has not been installed yet, one may consider downloading and installing the VM VirtualBox (VBox) Image of the entire package of SOA Suite, B2B, and JDeveloper, provided by Oracle on the Oracle download site found at http://www.oracle.com/technetwork/middleware/soasuite/learnmore/vmsoa-172279.html. All you need to do is to install Oracle VM VirtualBox, and import the SOA/BPM appliance. This is for evaluation and trial purposes, and is not recommended for production use; however, for the purpose of following, along with the tutorial in the article, it is perfect. The following table shows minimum and recommended requirements for the VBox Image: Minimum Recommended Memory (RAM) 4-6 GB 8 GB Disk Space 25 GB 50 GB While VM's are convenient, they do use quite a bit of disk space and memory. If you don't have a machine that meets the minimum requirements, it will be a challenge to try the exercises. The other alternative is to download the bits for the platform you are using from the Oracle download page, and install each software package, and configure them accordingly, including a JDK, a DB, WebLogic Server, SOA Suite, and JDeveloper, among other things you may need. If you decide that you have enough system resources to run the VBox Image, here are some of the major steps that you need to perform to download and install it. Please follow the detailed instructions found in the Introduction and Readme file that can be downloaded from http://www.oracle.com/technetwork/middleware/soasuite/learnmore/soabpmvirtualboxreadme-1612068.pdf, in order to have the complete set of instructions. Download the Introduction and Readme file, and review. Enable hardware virtualization in your PC BIOS if necessary. To download and install the VirtualBox software (engine that runs the virtual machine on your host machine), click on the link Download and install Oracle VM VirtualBox on the download page. To download the 7 pieces of the ZIP file, click on each file download ending with 7z.00[1-7] on the download page. To download the MD5 Checksum tool if you don't have one, click on the link Download MD5sums if you're on Windows to check your download worked okay on the download page. Run the MD5 Checksum tool to verify the 7 downloaded files: md5sums oel5u5-64bit-soabpm-11gr1-ps5-2-0-M.7z.001. Repeat for all 7 files. (This takes quite a while, but it is best to do it, so that you can verify that your download is complete and accurate.) Compare the results of the program with the results in the download that ends with .mdsum. They should match exactly. Extract the VBox Image from the .001 file using a Zip/Unzip tool. Use a ZIP tool such as 7-Zip (available as freeware for Windows), WinZip, or other to extract the .ova file from the 7 files into a single file on your platform. Using 7-Zip, if you extract from the first file; it will find the other 6 files and combine them all as it extracts. Start VirtualBox and set preferences such as the location of the VBox Image on your disk (follow instructions in the readme file). Import the new .ova file that was extracted from the ZIP file. Check settings and adjust memory/CPU. Start the appliance (VBox Image). Login as oracle with password oracle (check Readme). Choose the domain type dev_soa_osb. Set up a shared folder, you can use to share files between your machine and the virtual machine, and restart the VM. Once you are logged back in, start the admin server using the text based menu. Once the server is started, you can start the graphical desktop using the text based menu. Click on the jDeveloper Icon on the desktop of the VM to start jDeveloper. Choose Default Role when prompted for a role. At the time of writing, the latest available version is 11g PS5 (11.1.1.1.6). The VBox Image comes with SOA Suite, Oracle 11g XE Database, and JDeveloper, pre-installed on a Linux Virtual Machine. Using the VirtualBox technology, you can run this Linux machine virtually on your laptop, desktop, or on a variety of other platforms. For the purpose of this article, you should choose the dev_soa_osb type of domain. System requirements Oracle B2B is installed as a part of the SOA Suite installation. The SOA Suite installation steps are well documented, and are beyond the scope of this article. If you have never installed Oracle SOA Suite 11g, check with the Installation Guide for Oracle SOA Suite and Oracle Business Process Management Suite 11g. It can be downloaded from the Oracle Technology Network (OTN) Documentation downloads page at http://docs.oracle.com/cd/E23943_01/doc.1111/e13925/toc.htm. There are several important topics that did not have enough coverage in the SOA Suite Installation Guide. One of them is how to prepare the environment for the SOA/B2B installation. To begin, it is important to validate whether your environment meets the minimum requirements specified in the Oracle Fusion Middleware System Requirements and Specifications document. It can be downloaded from the Oracle Technology Network (OTN) Documentation downloads page at http://docs.oracle.com/html/E18558_01/fusion_requirements.htm. The spreadsheet provides very important SOA Suite installation recommendations, such as the minimum disk space information and memory requirements that could help the IT hardware team with its procurement planning process. For instance, the Oracle recommended hardware and system requirements for SOA Suite are: Minimum Physical Memory required: 2 gigabytes Minimum available Memory Required: 4 gigabytes CPU: dual-core Pentium, 1.5 GHz or greater Disk Space: 15 gigabytes or more This document also has information about supported databases and database versions. Another important document that has plenty of relevant information is Oracle Fusion Middleware 11g Release 1 (11.1.1.x) Certification Matrix. It can be downloaded from the Oracle Technology Network (OTN) Documentation downloads page at http://www.oracle.com/technetwork/middleware/downloads/fmw-11gr1certmatrix.xls. It is indeed a treasure chest of information. This spreadsheet may save you from a lot of headache. The last thing someone wants to run into is the need to re-install, just because they did not properly read and /or missed some recommendations. Here are some important points from the spreadsheet you don't want to miss: Hardware platform version's compatibility with a particular SOA Suite release Supported JDK versions Interoperability support for SOA Suite with WebLogic Server Supported database versions In conclusion, the following list includes a complete SOA Suite software stack (as used in the article): Oracle WebLogic Server (10.1.3.6) (Required) Repository Creation Utility (RCU) (11.1.1.6.0) (Required) SOA Suite 11g (11.1.1.6.0) (Required) JDeveloper (11.1.1.6.0) (Required) JDeveloper Extension for SOA (Required) Oracle B2B Document Editor release 7.05 (Optional) Oracle Database 11g (Required) Oracle B2B installation and post-installation configuration notes There are several important installation and post-installation steps that may directly or indirectly impact the B2B component's behavior. Creating a WebLogic domain is one of the most important SOA Suite 11g installation steps. The BEA engineers, who used to work with WebLogic prior to 11g, never before had to select SOA Suite components while creating a domain. This process is completely new for the Oracle engineers who are familiar only with prior releases of SOA Suite. There are several steps in this process that, if missed, might require a complete re-installation. A common mistake that people make when creating a new domain is that they don't check the Enterprise Manager checkbox. As a result, Enterprise Manager is not available, meaning that neither instance monitoring and tracking, nor access to the B2B configuration properties is available. Make sure you do not make such a mistake by selecting the Oracle Enterprise Manager checkbox. Oracle Enterprise Manager has been assigned a new name: Enterprise Manager Fusion Middleware Control. While planning SOA Suite deployment architecture, it is recommended to choose ahead of time between the following two WebLogic domain configurations: Developers domain Production domain In the Developers domain configuration, SOA Suite is installed as part of the administration server, implying that a separate managed server will not be created. This configuration could be a good choice for a development server, or a personal laptop, or any environment where available memory is limited. One should always keep in mind that SOA Suite requires up to 4 gigabytes of available memory. To set up the Developers domain, select the Oracle SOA Suite for developers checkbox on the Select Domain Source page, as shown in the following screenshot: Oracle strongly recommends against using this configuration in a production environment by warning that it will not be supported; that is, Oracle Support won't be able to provide assistance for any issues that happen to occur in this environment. Conversely, if the Oracle SOA Suite checkbox is selected, as shown in the following screenshot, a managed server will be created with a default name soa_server1. Creating a separate managed server (which is a WebLogic Java Virtual Machine) and deploying SOA Suite to this managed server, provides a more scalable configuration. If SOA Suite for developers was installed, you need to perform the following steps to activate the B2B user interface: Login to the Oracle WebLogic Server Administration Console using the following URL: http :: //<localserver>:7001/console (note that 7001 is the default port unless a different port was chosen during the installation process). Provide the default administrator account (the WebLogic user, unless it was changed during the installation process). Select Deployments from the Domain Structure list. On the bottom-right side of the page, select b2bui from the Deployments list (as shown in the following screenshot). On the next page, click on the Targets tab. Select the Component checkbox to enable the Change Target button. Click on the Change Target button. Select the AdminServer checkbox and click on the Yes button. Click on the Activate Changes button. Click on the Deployments link in the WebLogic domain structure. The B2B user interface is activated. If the SOA Suite production configuration was chosen, these steps are no longer necessary. However, you must first configure Node Manager. To do that, execute the setNMProps script and start Node Manager. $ORACLE_COMMON_HOME/common/bin/setNMProps.sh $MW_HOME/wlserver_n/server/bin/startNodeManager.sh Oracle B2B web components Oracle B2B Gateway is deployed as part of the Oracle SOA Service Infrastructure, or SOA-Infra. SOA Infrastructure is a Java EE compliant application running on Oracle WebLogic Server. Java EE compliant application: It is a wrapper around web applications and Enterprise Java Bean (EJB) applications Web Application: It usually represents the User Interface Layer, and includes Java Server pages (JSP), Servlets, HTML, and so on Servlet: It is a a module of Java code that runs as a server-side application Java Server Pages (JSP): It is a programming technology used to make dynamic web pages WAR archive: It is an artifact for the web application deployment Enterprise Java Beans: These are server-side domain objects that fit into a standard component-based architecture for building enterprise applications with Java EJB application: It is a collection of Enterprise Java Beans The following table shows a list of B2B web components installed as part of the SOA Infrastructure application. They include an enterprise application, several EJBs, a web application, and a web service. The B2B web application provides a link to the B2B Interface. The B2B MetadataWS Web Service provides Oracle SOA Service Infrastructure with access to the metadata repository. Stateless EJBs are used by the B2B Engine. This table might be helpful to understand how Oracle B2B integrates with Oracle SOA Suite. It could also be useful while developing B2B high availability architecture. Name Application Type b2b Web Application b2bui JEE Application B2BInstanceMessageBean EJB B2BStarterBeanWLS EJB B2BUtilityBean EJB B2BMetadataWS Web Service
Read more
  • 0
  • 0
  • 5600
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at ₹800/month. Cancel anytime
article-image-freeswitch-106-sip-and-user-directory
Packt
05 Aug 2010
7 min read
Save for later

FreeSWITCH 1.0.6: SIP and the User Directory

Packt
05 Aug 2010
7 min read
(For more resources on Telephony, see here.) Understanding the FreeSWITCH user directory The FreeSWITCH user directory is based on a centralized XML document, comprised of one or more <domain> elements. Each <domain> can contain either <user> elements or <groups> elements. A <groups> element contains one or more <groups> elements, each of which contains one or more <user> elements. A small, simple example would look like the following: <section name="directory"> <domain name="example.com"> <groups> <group name="default"> <user id="1001"> <params> <param name="password" value="1234"/> </params> </user> </group> </groups> </domain></section> Some more basic configurations may not have a need to organize the users in groups so it is possible to omit the <groups> element completely, and just insert several <user> elements into the top <domain> element. The important thing is that each user@domain derived from this directory is available to all components in the system—it's a single centralized directory for storing all of your user information. If you register as a user with a SIP phone or if you try to leave a voicemail message for a user, FreeSWITCH looks in the same place for user data. This is important because it limits duplication of data, and makes it more efficient than it would be if each component kept track of its users separately. This system should work well for a small system with a few users in it, but what about a large system with thousands of users? What if a user wants to connect his existing database to FreeSWITCH to provide the user directory? Well, using mod_xml_curl (download here-ch:1,ch:3), we can create a web service that gets the request for the entries in the user directory, in the same way a web page sends the results of a form submission. In turn, that web service can query an existing database of users formatted any way possible, and construct the XML records in the format that FreeSWITCH registry expects. mod_xml_curl returns the data to the module requesting the lookup. This means that instant, seamless integration with your existing setup is possible; your data is still kept in its original, central location. The user directory can be accessed by any subsystem within FreeSWITCH. This includes modules, scripts, and the FSAPI interface among others. In this article, we are going to learn how the Sofia SIP module employs the user directory to authenticate your softphone or hardware SIP phone. If you are a developer you may appreciate some nifty things you can do with your user directory, such as adding a <variables> element to either the <domain>, the <groups>, or the <user> element. In this element you can set many <variable> elements, allowing you to set channel variables that will apply to every call made by a particular authenticated user. This can come in very handy in the Dialplan because it allows you to make user-specific routing decisions. It is also possible to define IP address ranges using CIDR notation, which can be used to authenticate particular users based on what remote network address they connect from. This removes the need for a login and password, if your user always logs in from the same remote IP address. The directory is implemented in pure XML. This is advantageous for several reasons, not the least of which is the "X" in XML: Extensible. Since XML is, by definition, extensible, the directory structure is also extensible. If we need to add a new element into the directory, we can do so simply by adding to the existing XML structure. Authentication versus authorizationAuthentication is the process of identifying a user. Authorization is the process of determining the level of access of a user. Authentication answers the question, "Is this person really who he says he is?" Authorization answers the question, "What is this person allowed to do here?" When you see expressions such as "IP Auth" and "Digest Auth", remember that they are referring to the two primary ways of identifying (that is, authenticating) a user. IP authorization is based upon the user's IP address. Digest authentication is based upon the user supplying a username and password. SIP (and FreeSWITCH) can use either method. Visit http://en.wikipedia.org/wiki/Digest_access_authentication for a discussion of how digest authentication works Working with the FreeSWITCH user directory The default configuration has one domain with a directory of 20 users. Users can be added or removed very easily. There is no set limit to how many users can be defined on the system. The list of users is collectively referred to as the directory. Users can belong to one or more groups. Finally, all the users belong to a single domain. By default, the domain is the IP address of the FreeSWITCH server. In the following sections we will discuss these topics: User features Adding a user Testing voicemail Groups of users User features Let's begin by looking at the XML file that defines a user. Locate the file conf/directory/default/1000.xml and open it in an editor. You should see a file like the following: <include> <user id="1000"> <params> <param name="password" value="$${default_password}"/> <param name="vm-password" value="1000"/> </params> <variables> <variable name="toll_allow" value="domestic,international,local"/> <variable name="accountcode" value="1000"/> <variable name="user_context" value="default"/> <variable name="effective_caller_id_name" value="Extension 1000"/> <variable name="effective_caller_id_number" value="1000"/> <variable name="outbound_caller_id_name" value="$${outbound_caller_name}"/> <variable name="outbound_caller_id_number" value="$${outbound_caller_id}"/> <variable name="callgroup" value="techsupport"/> </variables> </user></include> The XML structure of a user is simple. Within the <include> tags the user has the following: The user element with the id attribute The params element, wherein parameters are specified The variables element, wherein channel variables are defined Even before we know what much of the specifics mean, we can glean from this file that the user id is 1000 and that there is both a password and a vm-password. In this case, the password parameter refers to the SIP authorization password. The expression $${default_password} refers to the value contained in the global variable default_password which is defined in the conf/vars.xml file. If you surmised that vm-password means "voicemail password" then you are correct. This value refers to the digits that the user needs to dial when logging in to check his or her voicemail messages. The value of id is used both as the authorization username and the SIP username. Additionally, there are a number of channel variables that are defined for this user. Most of these are directly related to the default Dialplan. The following table lists each variable and what it is used for: Variable Purpose toll_allow Specifies which types of calls this user can make accountcode Arbitrary value that shows up in CDR data user_context The Dialplan context that is used when this person makes a phone call effective_caller_id_name Caller ID name displayed on called party's phone when calling another registered user effective_caller_id_number Caller ID number displayed on called party's phone when calling another registered user outbound_caller_id_name Caller ID name sent to provider on outbound calls outbound_caller_id_number Caller ID number sent to provider on outbound calls callgroup Arbitrary value that can be used in Dialplan or CDR In summary, a user in the default configuration has the following: A username for SIP and for authorization A voicemail password A means of allowing/restricting dialling A means of handling caller ID being sent out Several arbitrary variables that can be used or ignored as needed Let's now add a new user to our directory.
Read more
  • 0
  • 0
  • 5596

article-image-running-tasks-asynchronously
Packt
29 Dec 2016
12 min read
Save for later

Running tasks asynchronously

Packt
29 Dec 2016
12 min read
In this article by Javier Fernández González, author of the book Java 9 Concurrency Cookbook - Second Edition we will cover how to run tasks asynchronously. When you execute ForkJoinTask in ForkJoinPool, you can do it in a synchronous or asynchronous way. When you do it in a synchronous way, the method that sends the task to the pool doesn't return until the task sent finishes its execution. When you do it in an asynchronous way, the method that sends the task to the executor returns immediately, so the task can continue with its execution. (For more resources related to this topic, see here.) You should be aware of a big difference between the two methods. When you use the synchronized methods, the task that calls one of these methods (for example, the invokeAll() method) is suspended until the tasks it sent to the pool finish their execution. This allows the ForkJoinPool class to use the work-stealing algorithm to assign a new task to the worker thread that executed the sleeping task. On the contrary, when you use the asynchronous methods (for example, the fork() method), the task continues with its execution, so the ForkJoinPool class can't use the work-stealing algorithm to increase the performance of the application. In this case, only when you call the join() or get() methods to wait for the finalization of a task, the ForkJoinPool class can use that algorithm. In addition to RecursiveAction and RecursiveTask classes, Java 8 introduced a new ForkJoinTask with the CountedCompleter class. With this kind of tasks you can include a completion action that will be executed when is launched and there is no child pending tasks. This mechanism is based in a method included in the class (the onCompletion() method) and a counter of pending tasks. This counter is initialized to zero by default and you can increment it when you need in an atomic way. Normally, you will increment this counter one by one when you launch a child task. Finally, when a task has finished is execution, you can try to complete the execution of the task and consequently, executes the onCompletion() method. If the pending count is bigger than zero, it is decremented by one. If it's zero, the onCompletion() method is executed and then the parent task is tried to complete. In this article, you will learn how to use the asynchronous methods provided by the ForkJoinPool and CountedCompleter classes for the management of tasks. You are going to implement a program that will search for files with a determined extension inside a folder and its subfolders. The CountedCompleter class you're going to implement will process the content of a folder. For each subfolder inside that folder, it will send a new task to the ForkJoinPool class in an asynchronous way. For each file inside that folder, the task will check the extension of the file and add it to the result list if it proceeds. When a task is completed, it will insert the result lists of all its child tasks in its result task. How to do it... Follow these steps to implement the example: Create a class named FolderProcessor and specify that it extends the CountedCompleter class parameterized with the List<String> type. public class FolderProcessor extends CountedCompleter<List<String>> { Declare the serial version UID of the class. This element is necessary because the parent class of the RecursiveTask class, the ForkJoinTask class, implements the Serializable interface. private static final long serialVersionUID = -1826436670135695513L; Declare a private String attribute named path. This attribute will store the full path of the folder this task is going to process. private String path; Declare a private String attribute named extension. This attribute will store the name of the extension of the files this task is going to look for. private String extension; Declare two List private attributes named tasks and resultList. We will use the first one to store all the child tasks launched from this task and the other one to store the list of results of this task. private List<FolderProcessor> tasks; private List<String> resultList; Implement one constructor for the class to initialize its attributes and its parent class. We declared this constructor as protected as it will only be used internally protected FolderProcessor (CountedCompleter<?> completer, String path, String extension) { super(completer); this.path=path; this.extension=extension; } We implement other public constructor to be used externally. As the task created by this constructor won't have parent task, we don't include this object as parameter. public FolderProcessor (String path, String extension) { this.path=path; this.extension=extension; } Implement the compute() method. As the base class of our task is the CountedCompleter class, the return type of this method is void. @Override protected void compute() { First, initialize the two list attributes. resultList=new ArrayList<>(); tasks=new ArrayList<>(); Get the content of the folder. File file=new File(path); File content[] = file.listFiles(); For each element in the folder, if there is a subfolder, create a new FolderProcessor object and execute it asynchronously using the fork() method. We use the first constructor of the class and pass the current task as the completer task of the new one. We also increment the counter of pending tasks using the addToPendingCount() method. if (content != null) { for (int i = 0; i < content.length; i++) { if (content[i].isDirectory()) { FolderProcessor task=new FolderProcessor(this, content[i].getAbsolutePath(), extension); task.fork(); addToPendingCount(1); tasks.add(task); Otherwise, compare the extension of the file with the extension you are looking for using the checkFile() method and, if they are equal, store the full path of the file in the list of strings declared earlier. } else { if (checkFile(content[i].getName())){ list.add(content[i].getAbsolutePath()); } } } If the list of the FolderProcessor subtasks has more than 50 elements, write a message to the console to indicate this circumstance. if (tasks.size()>50) { System.out.printf("%s: %d tasks ran.n",file.getAbsolutePath(),tasks.size()); } } Finally, try to complete the current task using the tryComplete() method: tryComplete(); } Implement the onCompletion() method. This method will be executed when all the child tasks (all the tasks that have been forked from the current task) have finished their execution. We add the result list of all the child tasks to the result list of the current task. @Override public void onCompletion(CountedCompleter<?> completer) { for (FolderProcessor childTask : tasks) { resultList.addAll(childTask.getResultList()); } } Implement the checkFile() method. This method compares if the name of a file passed as a parameter ends with the extension you are looking for. If so, the method returns the true value, otherwise it returns the false value. private boolean checkFile(String name) { return name.endsWith(extension); } Finally, implement the getResultList() method to return the result list of a task. The code of this method is very simple so it won't be included. Implement the main class of the example by creating a class named Main with a main() method. public class Main { public static void main(String[] args) { Create ForkJoinPool using the default constructor. ForkJoinPool pool=new ForkJoinPool(); Create three FolderProcessor tasks. Initialize each one with a different folder path. FolderProcessor system=new FolderProcessor("C:\Windows", "log"); FolderProcessor apps=new FolderProcessor("C:\Program Files","log"); FolderProcessor documents=new FolderProcessor("C:\Documents And Settings","log"); Execute the three tasks in the pool using the execute() method. pool.execute(system); pool.execute(apps); pool.execute(documents); Write to the console information about the status of the pool every second until the three tasks have finished their execution. do { System.out.printf("******************************************n"); System.out.printf("Main: Parallelism: %dn",pool.getParallelism()); System.out.printf("Main: Active Threads: %dn",pool.getActiveThreadCount()); System.out.printf("Main: Task Count: %dn",pool.getQueuedTaskCount()); System.out.printf("Main: Steal Count: %dn",pool.getStealCount()); System.out.printf("******************************************n"); try { TimeUnit.SECONDS.sleep(1); } catch (InterruptedException e) { e.printStackTrace(); } } while ((!system.isDone())||(!apps.isDone())||(!documents.isDone())); Shut down ForkJoinPool using the shutdown() method. pool.shutdown(); Write the number of results generated by each task to the console. List<String> results; results=system.join(); System.out.printf("System: %d files found.n",results.size()); results=apps.join(); System.out.printf("Apps: %d files found.n",results.size()); results=documents.join(); System.out.printf("Documents: %d files found.n",results.size()); How it works... The following screenshot shows part of an execution of this example: The key of this example is in the FolderProcessor class. Each task processes the content of a folder. As you know, this content has the following two kinds of elements: Files Other folders If the task finds a folder, it creates another FolderProcessor object to process that folder and sends it to the pool using the fork() method. This method sends the task to the pool that will execute it if it has a free worker-thread or it can create a new one. The method returns immediately, so the task can continue processing the content of the folder. For every file, a task compares its extension with the one it's looking for and, if they are equal, adds the name of the file to the list of results. Once the task has processed all the content of the assigned folder, we try to complete the current task. As we explained in the introduction of this article, when we try to complete a task, the code of the CountedCompleter looks for the value of the pending task counter. If this value is bigger than 0, it decrease of that counter. On the contrary, if the value is 0, the task executes the onCompletion() method and then try to completes its parent task. In our case, when a task is processing a folder and it finds a subfolder, it creates a new child task, launch that task using the fork() method and increment the counter of pending tasks. So, when a task has processed all its content, the counter of pending tasks of the task will be equal to the number of child tasks we have launched. When we call the tryComplete() method, if the folder of the current task has subfolders, this call will decrease the number of pending tasks. Only when all its child tasks have been completed, its onCompletion() method is executed. If the folder of the current task hasn't got any subfolders, the counter of pending tasks will be zero and the onComplete() method will be called immediately and then it will try to complete its parent task. By this way, we create a tree of tasks from top to bottom that are completed from bottom to top. In the onComplete() method, we process all the result lists of the child tasks and add their elements in the result list of the current task. The ForkJoinPool class also allows the execution of tasks in an asynchronous way. You have used the execute() method to send the three initial tasks to the pool. In the Main class, you also finished the pool using the shutdown() method and wrote information about the status and the evolution of the tasks that are running in it. The ForkJoinPool class includes more methods that can be useful for this purpose. There's more... In this example we have used the addToPendingCount() method to increment the counter of pending tasks, but we have other methods we can use to change the value of this counter. setPendingCount(): This method establish the value of the counter of pending tasks. compareAndSetPendingCount(): This method receives two parameters. The first one is the expected value and the second one is the new value. If the value of the counter of pending tasks is equal to the expected value, establish its value to the new one. decrementPendingCountUnlessZero(): This method decrements the value of the counter of pending tasks unless it's equal to zero. The CountedCompleter class also includes other methods to manage the completion of the tasks. These are the most significant ones: complete(): This method executes the onCompletion() method independently of the value of the counter of pending tasks try to complete its completer (parent) task. onExceptionalCompletion(): This method is executed when the completeExceptionally() method has been called or the compute() method has thrown an Exception. Override this method to include your code to process those exceptions. In this example, you have used the join() method to wait for the finalization of tasks and get their results. You can also use one of the two versions of the get() method with this purpose: get(long timeout, TimeUnit unit): This version of the get() method, if the result of the task isn't available, waits the specified time for it. If the specified period of time passes and the result isn't yet available, the method returns a null value. The TimeUnit class is an enumeration with the following constants: DAYS, HOURS, MICROSECONDS, MILLISECONDS, MINUTES, NANOSECONDS, and SECONDS. The join() method can't be interrupted. If you interrupt the thread that called the join() method, the method throws an InterruptedException exception. Summary In this article we learned how to use the asynchronous methods provided by the ForkJoinPool and CountedCompleter classes for the management of tasks. Resources for Article: Further resources on this subject: Why Bother? – Basic [article] Getting Started with Sorting Algorithms in Java [article] Saying Hello to Java EE [article]
Read more
  • 0
  • 0
  • 5592

article-image-ogre-3d-fixed-function-pipeline-and-shaders
Packt
25 Nov 2010
13 min read
Save for later

Ogre 3D: Fixed Function Pipeline and Shaders

Packt
25 Nov 2010
13 min read
  OGRE 3D 1.7 Beginner's Guide Create real time 3D applications using OGRE 3D from scratch Easy-to-follow introduction to OGRE 3D Create exciting 3D applications using OGRE 3D Create your own scenes and monsters, play with the lights and shadows, and learn to use plugins Get challenged to be creative and make fun and addictive games on your own A hands-on do-it-yourself approach with over 100 examples Introduction Fixed Function Pipeline is the rendering pipeline on the graphics card that produces those nice shiny pictures we love looking at. As the prefix Fixed suggests, there isn't a lot of freedom to manipulate the Fixed Function Pipeline for the developer. We can tweak some parameters using the material files, but nothing fancy. That's where shaders can help fill the gap. Shaders are small programs that can be loaded onto the graphics card and then function as a part of the rendering process. These shaders can be thought of as little programs written in a C-like language with a small, but powerful, set of functions. With shaders, we can almost completely control how our scene is rendered and also add a lot of new effects that weren't possible with only the Fixed Function Pipeline. Render Pipeline To understand shaders, we need to first understand how the rendering process works as a whole. When rendering, each vertex of our model is translated from local space into camera space, then each triangle gets rasterized. This means, the graphics card calculates how to represent the model in an image. These image parts are called fragments. Each fragment is then processed and manipulated. We could apply a specific part of a texture to this fragment to texture our model or we could simply assign it a color when rendering a model in only one color. After this processing, the graphics card tests if the fragment is covered by another fragment that is nearer to the camera or if it is the fragment nearest to the camera. If this is true, the fragment gets displayed on the screen. In newer hardware, this step can occur before the processing of the fragment. This can save a lot of computation time if most of the fragments won't be seen in the end result. The following is a very simplified graph showing the pipeline: With almost each new graphics card generation, new shader types were introduced. It began with vertex and pixel/fragment shaders. The task of the vertex shader is to transform the vertices into camera space, and if needed, modify them in any way, like when doing animations completely on the GPU. The pixel/fragment shader gets the rasterized fragments and can apply a texture to them or manipulate them in other ways, for example, for lighting models with an accuracy of a pixel. Time for action – our first shader application Let's write our first vertex and fragment shaders: In our application, we only need to change the used material. Change it to MyMaterial13. Also remove the second quad: manual->begin("MyMaterial13", RenderOperation::OT_TRIANGLE_LIST); Now we need to create this material in our material file. First, we are going to define the fragment shader. Ogre 3D needs five pieces of information about the shader: The name of the shader In which language it is written In which source file it is stored How the main function of this shader is called In what profile we want the shader to be compiled All this information should be in the material file: fragment_program MyFragmentShader1 cg { source Ogre3DBeginnersGuideShaders.cg entry_point MyFragmentShader1 profiles ps_1_1 arbfp1 } The vertex shader needs the same parameter, but we also have to define a parameter that is passed from Ogre 3D to our shader. This contains the matrix that we will use for transforming our quad into camera space: vertex_program MyVertexShader1 cg { source Ogre3DBeginnerGuideShaders.cg entry_point MyVertexShader1 profiles vs_1_1 arbvp1 default_params { param_named_auto worldViewMatrix worldviewproj_matrix } } The material itself just uses the vertex and fragment shader names to reference them: material MyMaterial13 { technique { pass { vertex_program_ref MyVertexShader1 { } fragment_program_ref MyFragmentShader1 { } } } } Now we need to write the shader itself. Create a file named Ogre3DBeginnersGuideShaders.cg in the mediamaterialsprograms folder of your Ogre 3D SDK. Each shader looks like a function. One difference is that we can use the out keyword to mark a parameter as an outgoing parameter instead of the default incoming parameter. The out parameters are used by the rendering pipeline for the next rendering step. The out parameters of a vertex shader are processed and then passed into the pixel shader as an in parameter. The out parameter from a pixel shader is used to create the final render result. Remember to use the correct name for the function; otherwise, Ogre 3D won't find it. Let's begin with the fragment shader because it's easier: void MyFragmentShader1(out float4 color: COLOR) The fragment shader will return the color blue for every pixel we render: { color = float4(0,0,1,0); } That's all for the fragment shader; now we come to the vertex shader. The vertex shader has three parameters—the position for the vertex, the translated position of the vertex as an out variable, and as a uniform variable for the matrix we are using for the translation: void MyVertexShader1( float4 position : POSITION, out float4 oPosition : POSITION, uniform float4x4 worldViewMatrix) Inside the shader, we use the matrix and the incoming position to calculate the outgoing position: { oPosition = mul(worldViewMatrix, position); } Compile and run the application. You should see our quad, this time rendered in blue. What just happened? Quite a lot happened here; we will start with step 2. Here we defined the fragment shader we are going to use. Ogre 3D needs five pieces of information for a shader. We define a fragment shader with the keyword fragment_program, followed by the name we want the fragment program to have, then a space, and at the end, the language in which the shader will be written. As for programs, shader code was written in assembly and in the early days, programmers had to write shader code in assembly because there wasn't another language to be used. But also, as with general programming language, soon there came high-level programming to ease the pain of writing shader code. At the moment, there are three different languages that shaders can be written in: HLSL, GLSL, and CG. The shader language HLSL is used by DirectX and GLSL is the language used by OpenGL. CG was developed by NVidia in cooperation with Microsoft and is the language we are going to use. This language is compiled during the start up of our application to their respective assembly code. So shaders written in HLSL can only be used with DirectX and GLSL shaders with OpenGL. But CG can compile to DirectX and OpenGL shader assembly code; that's the reason why we are using it to be truly cross platform. That's two of the five pieces of information that Ogre 3D needs. The other three are given in the curly brackets. The syntax is like a property file—first the key and then the value. One key we use is source followed by the file where the shader is stored. We don't need to give the full path, just the filename will do, because Ogre 3D scans our directories and only needs the filename to find the file. Another key we are using is entry_point followed by the name of the function we are going to use for the shader. In the code file, we created a function called MyFragmentShader1 and we are giving Ogre 3D this name as the entry point for our fragment shader. This means, each time we need the fragment shader, this function is called. The function has only one parameter out float4 color : COLOR. The prefix out signals that this parameter is an out parameter, meaning we will write a value into it, which will be used by the render pipeline later on. The type of this parameter is called float4, which simply is an array of four float values. For colors, we can think of it as a tuple (r,g,b,a) where r stands for red, g for green, b for blue, and a for alpha: the typical tuple to description colors. After the name of the parameter, we got a : COLOR. In CG, this is called a semantic describing for what the parameter is used in the context of the render pipeline. The parameter :COLOR tells the render pipeline that this is a color. In combination with the out keyword and the fact that this is a fragment shader, the render pipeline can deduce that this is the color we want our fragment to have. The last piece of information we supply uses the keyword profiles with the values ps_1_1 and arbfp1. To understand this, we need to talk a bit about the history of shaders. With each generation of graphics cards, a new generation of shaders have been introduced. What started as a fairly simple C-like programming language without even IF conditions are now really complex and powerful programming languages. Right now, there are several different versions for shaders and each with a unique function set. Ogre 3D needs to know which of these versions we want to use. ps_1_1 means pixel shader version 1.1 and arbfp1 means fragment program version 1. We need both profiles because ps_1_1 is a DirectX specific function set and arbfp1 is a function subset for OpenGL. We say we are cross platform, but sometimes we need to define values for both platforms. All subsets can be found at http://www.ogre3d.org/docs/manual/manual_18.html. That's all needed to define the fragment shader in our material file. In step 3, we defined our vertex shader. This part is very similar to the fragment shader definition code; the main difference is the default_params block. This block defines parameters that are given to the shader during runtime. param_named_auto defines a parameter that is automatically passed to the shader by Ogre 3D. After this key, we need to give the parameter a name and after this, the value keyword we want it to have. We name the parameter worldViewMatrix; any other name would also work, and the value we want it to have has the key worldviewproj_matrix. This key tells Ogre 3D we want our parameter to have the value of the WorldViewProjection matrix. This matrix is used for transforming vertices from local into camera space. A list of all keyword values can be found at http://www.ogre3d.org/docs/manual/manual_23.html#SEC128. How we use these values will be seen shortly. Step 4 used the work we did before. As always, we defined our material with one technique and one pass; we didn't define a texture unit but used the keyword vertex_program_ref. After this keyword, we need to put the name of a vertex program we defined, in our case, this is MyVertexShader1. If we wanted, we could have put some more parameters into the definition, but we didn't need to, so we just opened and closed the block with curly brackets. The same is true for fragment_program_ref. Writing a shader Now that we have defined all necessary things in our material file, let's write the shader code itself. Step 6 defines the function head with the parameter we discussed before, so we won't go deeper here. Step 7 defines the function body; for this fragment shader, the body is extremely simple. We created a new float4 tuple (0,0,1,0), describes the color blue and assigns this color to our out parameter color. The effect is that everything that is rendered with this material will be blue. There isn't more to the fragment shader, so let's move on to the vertex shader. Step 8 defines the function header. The vertex shader has 3 parameters— two are marked as positions using CG semantics and the other parameter is a 4x4 matrix using float4 as values named worldViewMatrix. Before the parameter type definition, there is the keyword uniform. Each time our vertex shader is called, it gets a new vertex as the position parameter input, calculates the position of this new vertex, and saves it in the oPosition parameter. This means with each call, the parameter changes. This isn't true for the worldViewMatrix. The keyword uniform denotes parameters that are constant over one draw call. When we render our quad, the worldViewMatrix doesn't change while the rest of the parameters are different for each vertex processed by our vertex shader. Of course, in the next frame, the worldViewMatrix will probably have changed. Step 9 creates the body of the vertex shader. In the body, we multiply the vertex that we got with the world matrix to get the vertex translated into camera space. This translated vertex is saved in the out parameter to be processed by the rendering pipeline. We will look more closely into the render pipeline after we have experimented with shaders a bit more. Texturing with shaders We have painted our quad in blue, but we would like to use the previous texture. Time for action – using textures in shaders Create a new material named MyMaterial14. Also create two new shaders named MyFragmentShader2 and MyVertexShader2. Remember to copy the fragment and vertex program definitions in the material file. Add to the material file a texture unit with the rock texture: texture_unit { texture terr_rock6.jpg } We need to add two new parameters to our fragment shader. The first is a two tuple of floats for the texture coordinates. Therefore, we also use the semantic to mark the parameter as the first texture coordinates we are using. The other new parameter is of type sampler2D, which is another name for texture. Because the texture doesn't change on a per fragment basis, we mark it as uniform. This keyword indicates that the parameter value comes from outside the CG program and is set by the rendering environment, in our case, by Ogre 3D: void MyFragmentShader2(float2 uv : TEXCOORD0, out float4 color : COLOR, uniform sampler2D texture) In the fragment shader, replace the color assignment with the following line: color = tex2D(texture, uv); The vertex shader also needs some new parameters—one float2 for the incoming texture coordinates and one float2 as the outgoing texture coordinates. Both are our TEXCOORD0 because one is the incoming and the other is the outgoing TEXCOORD0: void MyVertexShader2( float4 position : POSITION, out float4 oPosition : POSITION, float2 uv : TEXCOORD0, out float2 oUv : TEXCOORD0, uniform float4x4 worldViewMatrix) In the body, we calculate the outgoing position of the vertex: oPosition = mul(worldViewMatrix, position); For the texture coordinates, we assign the incoming value to the outgoing value: oUv = uv; Remember to change the used material in the application code, and then compile and run it. You should see the quad with the rock texture.
Read more
  • 0
  • 0
  • 5590

article-image-securing-qlikview-documents
Packt
16 Jan 2014
6 min read
Save for later

Securing QlikView Documents

Packt
16 Jan 2014
6 min read
(For more resources related to this topic, see here.) Making documents available to the correct users can be handled in several different ways, depending on your implementation and license structure. These methods are not mutually exclusive and you may choose to implement a combination of them. By license If you only have named Document users, you can restrict access to documents by simply not granting users a license. If users do not have a Document license for a particular document, they may be able to see that document in AccessPoint, but will not be able to open it. You will need to turn off any automatic allocation of licenses for both Document licenses and Named User licenses, or the system will simply override your security by allocating an available license and giving the user access to that document. This only works for Document license users. The Named User license holders can't be locked out of a document this way as they have a license that allows them to open any number of documents, so they cannot be restricted. The fact that this is user based—a Document license can only be granted to a user, not a group—also means that there is no option to secure by a named group. This is the most basic, least flexible, and least user-friendly way to implement security. While it will certainly stop users getting access to documents—and it will work in either NTFS or DMS security modes—it can be frustrating for users to see a document that they think can open, but for which they will get a NO CAL error when they try to open it. The QlikView file will also need to have appropriate NTFS or DMS security so that users would be able to access it. The easiest way to do this is to grant access to a group that all the users will be in, or even allow access to an Authenticated Users group. Section Access Section Access security is a very effective way of securing a document to the correct set of users. This is because a user must be actually listed in the Section Access user list for the document to be even listed in AccessPoint for them. Additionally, if Section Access is in place, a user cannot even connect by using a direct access URL because they have no security access to the data. This method of securing documents works well in both NTFS and DMS security modes. When using the NTLM (Windows authentication via Internet Explorer) authentication method, you can have Group Names listed in Section Access. However, when using alternative authentication, Section Access does not give us an option to secure by group. As with the license method discussed earlier, appropriate file security needs to be in place in order to allow all the users access the QlikView file. NTFS Access Control List (ACL) NTFS (Microsoft's NT File System) security is the default method of securing access to files in a QlikView implementation. It works very well for installations where all the users are Windows users within the same domain or a set of trusted domains. In NTFS security mode, the Access Control List (ACL) of the QlikView file is used to list the documents for a particular user. This is a very straightforward way of securing access and will be very familiar to Windows system administrators. As with normal Windows file security, the security can be applied at the folder level. Windows security groups can also be used. Between groups and folder security, very flexible levels of security can be applied. By default, Internet Explorer and Google Chrome will pass through the Kerberos/NTLM credentials to sites in the local Intranet zone. For other browsers, such as Safari on the iPad, the user will be prompted for a username and password. When a user connects to AccessPoint and their credentials are established, they are compared against the ACLs for all the files hosted by QVS. Only those files that the user has access to—either directly by name or by group membership—will be listed in AccessPoint. Document Metadata Service (DMS) For non-Windows users, QlikView provides a way of managing user access to files called the Document Metadata Service (DMS). DMS uses a .META file in the same folder as the .QVW file to store the Access Control List. The Windows ACL, which has permissions on the file, now becomes mostly irrelevant as it is not used to authenticate users. It is only the QlikView service account that will need access to the file. It is a binary choice between using NTFS or DMS security on any one QlikView Server. Enabling DMS To enable DMS, we need to make a change to the server configuration. In the QlikView Management Console, on the Security tab of the QVS settings screen, we change Authorization to DMS authorization and then click on the Apply button. The QlikView Server service will need to be restarted for this change to take effect. Once the service has restarted, a new tab, Authorization, becomes available in the document properties: Clicking on the + button to the right of this tab allows you to enter new details of Access, User Type, and specific Users and Groups. Access is either set to Always or Restricted. When Access is set to Always, the associated user or group will have access at any time. If it is set to Restricted, you can specify a time range and specific days when the user or group has access. You can keep clicking on the + button to add as many sets of restricted times as needed for a user or group. The restrictions are additive; that is, if the user only has access on Monday and Tuesday in one group of restrictions, and then Thursday and Friday in another set of restrictions, they will therefore, have access on all four days. The User Type is one of the following categories: User Type Details All Users Essentially no security. Any user, including anonymous, who can access the server will be able to access the file. All Authenticated Users For most implementations, this will also be All Users. However, it will not give access to anonymous users. The Section Access would typically be used to manage the security. Named Users This allows you to specify a list of named users and/or groups that will have specific access to the document. If Named Users is selected, a Manage Users button will appear that allows you to specify users and/or groups. Summary In this article, we have looked at several ways of securing QlikView Documents—by license, using Section Access, utilizing NTFS ACLs, and implementing QlikView's DMS authorization. Resources for Article: Further resources on this subject: Common QlikView script errors [Article] Introducing QlikView elements [Article] Meet QlikView [Article]
Read more
  • 0
  • 0
  • 5587
article-image-building-color-picker-hex-rgb-conversion
Packt
02 Mar 2015
18 min read
Save for later

Building a Color Picker with Hex RGB Conversion

Packt
02 Mar 2015
18 min read
In this article by Vijay Joshi, author of the book Mastering jQuery UI, we are going to create a color selector, or color picker, that will allow the users to change the text and background color of a page using the slider widget. We will also use the spinner widget to represent individual colors. Any change in colors using the slider will update the spinner and vice versa. The hex value of both text and background colors will also be displayed dynamically on the page. (For more resources related to this topic, see here.) This is how our page will look after we have finished building it: Setting up the folder structure To set up the folder structure, follow this simple procedure: Create a folder named Article inside the MasteringjQueryUI folder. Directly inside this folder, create an HTML file and name it index.html. Copy the js and css folder inside the Article folder as well. Now go inside the js folder and create a JavaScript file named colorpicker.js. With the folder setup complete, let's start to build the project. Writing markup for the page The index.html page will consist of two sections. The first section will be a text block with some text written inside it, and the second section will have our color picker controls. We will create separate controls for text color and background color. Inside the index.html file write the following HTML code to build the page skeleton: <html> <head> <link rel="stylesheet" href="css/ui-lightness/jquery-ui- 1.10.4.custom.min.css"> </head> <body> <div class="container"> <div class="ui-state-highlight" id="textBlock"> <p> Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. </p> <p> Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. </p> <p> Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. </p> </div> <div class="clear">&nbsp;</div> <ul class="controlsContainer"> <li class="left"> <div id="txtRed" class="red slider" data-spinner="sptxtRed" data-type="text"></div><input type="text" value="0" id="sptxtRed" data-slider="txtRed" readonly="readonly" /> <div id="txtGreen" class="green slider" dataspinner=" sptxtGreen" data-type="text"></div><input type="text" value="0" id="sptxtGreen" data-slider="txtGreen" readonly="readonly" /> <div id="txtBlue" class="blue slider" dataspinner=" sptxtBlue" data-type="text"></div><input type="text" value="0" id="sptxtBlue" data-slider="txtBlue" readonly="readonly" /> <div class="clear">&nbsp;</div> Text Color : <span>#000000</span> </li> <li class="right"> <div id="bgRed" class="red slider" data-spinner="spBgRed" data-type="bg" ></div><input type="text" value="255" id="spBgRed" data-slider="bgRed" readonly="readonly" /> <div id="bgGreen" class="green slider" dataspinner=" spBgGreen" data-type="bg" ></div><input type="text" value="255" id="spBgGreen" data-slider="bgGreen" readonly="readonly" /> <div id="bgBlue" class="blue slider" data-spinner="spBgBlue" data-type="bg" ></div><input type="text" value="255" id="spBgBlue" data-slider="bgBlue" readonly="readonly" /> <div class="clear">&nbsp;</div> Background Color : <span>#ffffff</span> </li> </ul> </div> <script src="js/jquery-1.10.2.js"></script> <script src="js/jquery-ui-1.10.4.custom.min.js"></script> <script src="js/colorpicker.js"></script> </body> </html> We started by including the jQuery UI CSS file inside the head section. Proceeding to the body section, we created a div with the container class, which will act as parent div for all the page elements. Inside this div, we created another div with id value textBlock and a ui-state-highlight class. We then put some text content inside this div. For this example, we have made three paragraph elements, each having some random text inside it. After div#textBlock, there is an unordered list with the controlsContainer class. This ul element has two list items inside it. First list item has the CSS class left applied to it and the second has CSS class right applied to it. Inside li.left, we created three div elements. Each of these three div elements will be converted to a jQuery slider and will represent the red (R), green (G), and blue (B) color code, respectively. Next to each of these divs is an input element where the current color code will be displayed. This input will be converted to a spinner as well. Let's look at the first slider div and the input element next to it. The div has id txtRed and two CSS classes red and slider applied to it. The red class will be used to style the slider and the slider class will be used in our colorpicker.js file. Note that this div also has two data attributes attached to it, the first is data-spinner, whose value is the id of the input element next to the slider div we have provided as sptxtRed, the second attribute is data-type, whose value is text. The purpose of the data-type attribute is to let us know whether this slider will be used for changing the text color or the background color. Moving on to the input element next to the slider now, we have set its id as sptxtRed, which should match the value of the data-spinner attribute on the slider div. It has another attribute named data-slider, which contains the id of the slider, which it is related to. Hence, its value is txtRed. Similarly, all the slider elements have been created inside div.left and each slider has an input next to id. The data-type attribute will have the text value for all sliders inside div.left. All input elements have also been assigned a value of 0 as the initial text color will be black. The same pattern that has been followed for elements inside div.left is also followed for elements inside div.right. The only difference is that the data-type value will be bg for slider divs. For all input elements, a value of 255 is set as the background color is white in the beginning. In this manner, all the six sliders and the six input elements have been defined. Note that each element has a unique ID. Finally, there is a span element inside both div.left and div.right. The hex color code will be displayed inside it. We have placed #000000 as the default value for the text color inside the span for the text color and #ffffff as the default value for the background color inside the span for background color. Lastly, we have included the jQuery source file, the jQuery UI source file, and the colorpicker.js file. With the markup ready, we can now write the properties for the CSS classes that we used here. Styling the content To make the page presentable and structured, we need to add CSS properties for different elements. We will do this inside the head section. Go to the head section in the index.html file and write these CSS properties for different elements: <style type="text/css">   body{     color:#025c7f;     font-family:Georgia,arial,verdana;     width:700px;     margin:0 auto;   }   .container{     margin:0 auto;     font-size:14px;     position:relative;     width:700px;     text-align:justify;    } #textBlock{     color:#000000;     background-color: #ffffff;   }   .ui-state-highlight{     padding: 10px;     background: none;   }   .controlsContainer{       border: 1px solid;       margin: 0;       padding: 0;       width: 100%;       float: left;   }   .controlsContainer li{       display: inline-block;       float: left;       padding: 0 0 0 50px;       width: 299px;   }   .controlsContainer div.ui-slider{       margin: 15px 0 0;       width: 200px;       float:left;   }   .left{     border-right: 1px solid;   }   .clear{     clear: both;   }     .red .ui-slider-range{ background: #ff0000; }   .green .ui-slider-range{ background: #00ff00; }   .blue .ui-slider-range{ background: #0000ff; }     .ui-spinner{       height: 20px;       line-height: 1px;       margin: 11px 0 0 15px;     }   input[type=text]{     margin-top: 0;     width: 30px;   } </style> First, we defined some general rules for page body and div .container. Then, we defined the initial text color and background color for the div with id textBlock. Next, we defined the CSS properties for the unordered list ul .controlsContainer and its list items. We have provided some padding and width to each list item. We have also specified the width and other properties for the slider as well. Since the class ui-slider is added by jQuery UI to a slider element after it is initialized, we have added our properties in the .controlsContainer div .ui-slider rule. To make the sliders attractive, we then defined the background colors for each of the slider bars by defining color codes for red, green, and blue classes. Lastly, CSS rules have been defined for the spinner and the input box. We can now check our progress by opening the index.html page in our browser. Loading it will display a page that resembles the following screenshot: It is obvious that sliders and spinners will not be displayed here. This is because we have not written the JavaScript code required to initialize those widgets. Our next section will take care of them. Implementing the color picker In order to implement the required functionality, we first need to initialize the sliders and spinners. Whenever a slider is changed, we need to update its corresponding spinner as well, and conversely if someone changes the value of the spinner, we need to update the slider to the correct value. In case any of the value changes, we will then recalculate the current color and update the text or background color depending on the context. Defining the object structure We will organize our code using the object literal. We will define an init method, which will be the entry point. All event handlers will also be applied inside this method. To begin with, go to the js folder and open the colorpicker.js file for editing. In this file, write the code that will define the object structure and a call to it: var colorPicker = {   init : function ()   {       },   setColor : function(slider, value)   {   },   getHexColor : function(sliderType)   {   },   convertToHex : function (val)   {   } }   $(function() {   colorPicker.init(); }); An object named colorPicker has been defined with four methods. Let's see what all these methods will do: init: This method will be the entry point where we will initialize all components and add any event handlers that are required. setColor: This method will be the main method that will take care of updating the text and background colors. It will also update the value of the spinner whenever the slider moves. This method has two parameters; the slider that was moved and its current value. getHexColor: This method will be called from within setColor and it will return the hex code based on the RGB values in the spinners. It takes a sliderType parameter based on which we will decide which color has to be changed; that is, text color or background color. The actual hex code will be calculated by the next method. convertToHex: This method will convert an RGB value for color into its corresponding hex value and return it to get a HexColor method. This was an overview of the methods we are going to use. Now we will implement these methods one by one, and you will understand them in detail. After the object definition, there is the jQuery's $(document).ready() event handler that will call the init method of our object. The init method In the init method, we will initialize the sliders and the spinners and set the default values for them as well. Write the following code for the init method in the colorpicker.js file:   init : function () {   var t = this;   $( ".slider" ).slider(   {     range: "min",     max: 255,     slide : function (event, ui)     {       t.setColor($(this), ui.value);     },     change : function (event, ui)     {       t.setColor($(this), ui.value);     }   });     $('input').spinner(   {     min :0,     max : 255,     spin : function (event, ui)     {       var sliderRef = $(this).data('slider');       $('#' + sliderRef).slider("value", ui.value);     }   });       $( "#txtRed, #txtGreen, #txtBlue" ).slider('value', 0);   $( "#bgRed, #bgGreen, #bgBlue" ).slider('value', 255); } In the first line, we stored the current scope value, this, in a local variable named t. Next, we will initialize the sliders. Since we have used the CSS class slider on each slider, we can simply use the .slider selector to select all of them. During initialization, we provide four options for sliders: range, max, slide, and change. Note the value for max, which has been set to 255. Since the value for R, G, or B can be only between 0 and 255, we have set max as 255. We do not need to specify min as it is 0 by default. The slide method has also been defined, which is invoked every time the slider handle moves. The call back for slide is calling the setColor method with an instance of the current slider and the value of the current slider. The setColor method will be explained in the next section. Besides slide, the change method is also defined, which also calls the setColor method with an instance of the current slider and its value. We use both the slide and change methods. This is because a change is called once the user has stopped sliding the slider handle and the slider value has changed. Contrary to this, the slide method is called each time the user drags the slider handle. Since we want to change colors while sliding as well, we have defined the slide as well as change methods. It is time to initialize the spinners now. The spinner widget is initialized with three properties. These are min and max, and the spin. min and max method has been set to 0 and 255, respectively. Every time the up/down button on the spinner is clicked or the up/down arrow key is used, the spin method will be called. Inside this method, $(this) refers to the current spinner. We find our related slider to this spinner by reading the data-slider attribute of this spinner. Once we get the exact slider, we set its value using the value method on the slider widget. Note that calling the value method will invoke the change method of the slider as well. This is the primary reason we have defined a callback for the change event while initializing the sliders. Lastly, we will set the default values for the sliders. For sliders inside div.left, we have set the value as 0 and for sliders inside div.right, the value is set to 255. You can now check the page on your browser. You will find that the slider and the spinner elements are initialized now, with the values we specified: You can also see that changing the spinner value using either the mouse or the keyboard will update the value of the slider as well. However, changing the slider value will not update the spinner. We will handle this in the next section where we will change colors as well. Changing colors and updating the spinner The setColor method is called each time the slider or the spinner value changes. We will now define this method to change the color based on whether the slider's or spinner's value was changed. Go to the setColor method declaration and write the following code: setColor : function(slider, value) {   var t = this;   var spinnerRef = slider.data('spinner');   $('#' + spinnerRef).spinner("value", value);     var sliderType = slider.data('type')     var hexColor = t.getHexColor(sliderType);   if(sliderType == 'text')   {       $('#textBlock').css({'color' : hexColor});       $('.left span:last').text(hexColor);                  }   else   {       $('#textBlock').css({'background-color' : hexColor});       $('.right span:last').text(hexColor);                  } } In the preceding code, we receive the current slider and its value as a parameter. First we get the related spinner to this slider using the data attribute spinner. Then we set the value of the spinner to the current value of the slider. Now we find out the type of slider for which setColor is being called and store it in the sliderType variable. The value for sliderType will either be text, in case of sliders inside div.left, or bg, in case of sliders inside div.right. In the next line, we will call the getHexColor method and pass the sliderType variable as its argument. The getHexColor method will return the hex color code for the selected color. Next, based on the sliderType value, we set the color of div#textBlock. If the sliderType is text, we set the color CSS property of div#textBlock and display the selected hex code in the span inside div.left. If the sliderType value is bg, we set the background color for div#textBlock and display the hex code for the background color in the span inside div.right. The getHexColor method In the preceding section, we called the getHexColor method with the sliderType argument. Let's define it first, and then we will go through it in detail. Write the following code to define the getHexColor method: getHexColor : function(sliderType) {   var t = this;   var allInputs;   var hexCode = '#';   if(sliderType == 'text')   {     //text color     allInputs = $('.left').find('input[type=text]');   }   else   {     //background color     allInputs = $('.right').find('input[type=text]');   }   allInputs.each(function (index, element) {     hexCode+= t.convertToHex($(element).val());   });     return hexCode; } The local variable t has stored this to point to the current scope. Another variable allInputs is declared, and lastly a variable to store the hex code has been declared, whose value has been set to # initially. Next comes the if condition, which checks the value of parameter sliderType. If the value of sliderType is text, it means we need to get all the spinner values to change the text color. Hence, we use jQuery's find selector to retrieve all input boxes inside div.left. If the value of sliderType is bg, it means we need to change the background color. Therefore, the else block will be executed and all input boxes inside div.right will be retrieved. To convert the color to hex, individual values for red, green, and blue will have to be converted to hex and then concatenated to get the full color code. Therefore, we iterate in inputs using the .each method. Another method convertToHex is called, which converts the value of a single input to hex. Inside the each method, we keep concatenating the hex value of the R, G, and B components to a variable hexCode. Once all iterations are done, we return the hexCode to the parent function where it is used. Converting to hex convertToHex is a small method that accepts a value and converts it to the hex equivalent. Here is the definition of the convertToHex method: convertToHex : function (val) {   var x  = parseInt(val, 10).toString(16);   return x.length == 1 ? "0" + x : x; } Inside the method, firstly we will convert the received value to an integer using the parseInt method and then we'll use JavaScript's toString method to convert it to hex, which has base 16. In the next line, we will check the length of the converted hex value. Since we want the 6-character dash notation for color (such as #ff00ff), we need two characters each for red, green, and blue. Hence, we check the length of the created hex value. If it is only one character, we append a 0 to the beginning to make it two characters. The hex value is then returned to the parent function. With this, our implementation is complete and we can check it on a browser. Load the page in your browser and play with the sliders and spinners. You will see the text or background color changing, based on their value: You will also see the hex code displayed below the sliders. Also note that changing the sliders will change the value of the corresponding spinner and vice versa. Improving the Colorpicker This was a very basic tool that we built. You can add many more features to it and enhance its functionality. Here are some ideas to get you started: Convert it into a widget where all the required DOM for sliders and spinners is created dynamically Instead of two sliders, incorporate the text and background changing ability into a single slider with two handles, but keep two spinners as usual Summary In this article, we created a basic color picker/changer using sliders and spinners. You can use it to view and change the colors of your pages dynamically. Resources for Article: Further resources on this subject: Testing Ui Using WebdriverJs? [article] Important Aspect Angularjs Ui Development [article] Kendo Ui Dataviz Advance Charting [article]
Read more
  • 0
  • 0
  • 5586

article-image-stacked-denoising-autoencoders
Packt
11 Jul 2016
13 min read
Save for later

Stacked Denoising Autoencoders

Packt
11 Jul 2016
13 min read
In this article by John Hearty, author of the book Advanced Machine Learning with Python, we discuss autoencoders as valuable tools in themselves, significant accuracy can be obtained by stacking autoencoders to form a deep network. This is achieved by feeding the representation created by the encoder on one layer into the next layer's encoder as input to that layer. (For more resources related to this topic, see here.) Stacked DenoisingAutoencoders(SdA) are currently in use in many leading data science teams for sophisticated natural language analyses as well as a broad range of signals, images, and text analyses. The implementation of SdA will be very familiar after the previous chapter's discussion of deep belief networks. The SdA is usedin much the same way as the RBMs in our deep belief networks were used. Each layer of the deep architecture will have a dA and sigmoid component, with the autoencoder component being used to pretrain the sigmoid network. The performance measure used by anSdA is the training set error with an intensive period of layer-to-layer (layer-wise) pretraining used to gradually align network parameters before a final period of fine-tuning. During fine-tuning, the network is trained using validation and test data, over fewer epochs but with larger update steps. The goal is to have the network converge at the end of the fine-tuning to deliver an accurate result. In addition to delivering on the typical advantages of deep networks (the ability to learn feature representations for complex or high-dimensional datasets and train a model without extensive feature engineering), stacked autoencoders have an additional, very interesting property. Correctly configured stacked autoencoders can capture a hierarchical grouping of their input data. Successive layers of anSdA may learn increasingly high-level features. While the first layer might learn some first-order features from input data (such as learning edges in a photo image), a second layer may learn some grouping of first-order features (for instance, by learning given configurations of edges that correspond to contours or structural elements in the input image). There's no golden rule to determine how many layers or how large layers should be for a given problem. The best solution is usually to experiment with these model parameters until you find an optimal point. This experimentation is best done with a hyperparameter optimization technique or genetic algorithm (subjects we'll discuss in later chapters of this book). Higher layers may learn increasingly high-order configurations, enabling anSdA to learn to recognise facial features, alphanumerical characters, or the generalised forms of objects (such as a bird). This is what gives SdAs their unique capability to learn very sophisticated, high-level abstractions of their input data. Autoencoders can be stacked indefinitely, and it has been demonstrated that continuing to stack autoencoders can improve the effectiveness of the deep architecture (with the main constraint becoming computing cost in time). In this chapter, we'll look at stacking three autoencoders to solve a natural language processing challenge. Applying SdA Now that we've had a chance to understand the advantages and power of the SdA as a deep learning architecture, let's test our skills on a real-world dataset. For this chapter, let's step away from image datasets and work with the OpinRank Review Dataset, a text dataset of around 259,000 hotel reviews from TripAdvisor,which is accessible via the UCI Machine Learning dataset Repository. This freely-available dataset provides review scores (as floating point numbers from 1 to 5) and review text for a broad range of hotels; we'll be applying our SdA to attempt to identify the scoring of each hotel from its review text. We'll be applying our autoencoder to analyze a preprocessed version of this data, which is accessible from the GitHub share accompanying this chapter. We'll be discussing the techniques by which we prepare text data in an upcoming chapter. The source data is available at https://archive.ics.uci.edu/ml/datasets/OpinRank+Review+Dataset. In order to get started, we're going to need anSdA (hereafter SdA) class! classSdA(object):     def __init__( self, numpy_rng, theano_rng=None, n_ins=280, hidden_layers_sizes=[500, 500], n_outs=5, corruption_levels=[0.1, 0.1]     ): As we previously discussed, the SdA is created by feeding the encoding from one layer's autoencoder as the input to the subsequent layer. This class supports the configuration of the layer count (reflected in, but not set by, the length of the hidden_layers_sizes and corruption_levels vectors). It also supports differentiated layer sizes (in nodes) at each layer, which can be set using hidden_layers_sizes. As we discussed, the ability to configure successive layers of the autoencoder is critical to developing successful representations. Next, we need parameters to store the MLP (self.sigmoid_layers) and dA (self.dA_layers) elements of the SdA. In order to specify the depth of our architecture, we use the self.n_layers parameter to specify the number of sigmoid and dA layers required: self.sigmoid_layers = [] self.dA_layers = [] self.params = [] self.n_layers = len(hidden_layers_sizes)   assertself.n_layers> 0 Next, we need to construct our sigmoid and dAlayers. We begin by setting the hidden layer size to be set either from the input vector size or by the activation of the preceding layer. Following this, sigmoid_layers and dA_layers components are created with the dA layer drawing from the dA class we discussed earlier in this article: for i in xrange(self.n_layers): if i == 0: input_size = n_ins else: input_size = hidden_layers_sizes[i - 1]   if i == 0: layer_input = self.x else: layer_input = self.sigmoid_layers[-1].output   sigmoid_layer = HiddenLayer(rng=numpy_rng, input=layer_input, n_in=input_size, n_out=hidden_layers_sizes[i], activation=T.nnet.sigmoid) self.sigmoid_layers.append(sigmoid_layer) self.params.extend(sigmoid_layer.params)   dA_layer = dA(numpy_rng=numpy_rng, theano_rng=theano_rng, input=layer_input, n_visible=input_size, n_hidden=hidden_layers_sizes[i],                           W=sigmoid_layer.W, bhid=sigmoid_layer.b) self.dA_layers.append(dA_layer) Having implemented the layers of our SdA, we'll need a final, logistic regression layer to complete the MLP component of the network: self.logLayer = LogisticRegression( input=self.sigmoid_layers[-1].output, n_in=hidden_layers_sizes[-1], n_out=n_outs         )   self.params.extend(self.logLayer.params) self.finetune_cost = self.logLayer.negative_log_likelihood(self.y) self.errors = self.logLayer.errors(self.y) This completes the architecture of our SdA. Next up, we need to generate the training functions used by the SdA class. Each function will have the minibatch index (index) as an argument, together with several other elements; corruption_level and learning_rate are enabled here so that we can adjust them (for example, gradually increase or decrease them) during training. Additionally, we identify variables that help identify where the batch starts and ends: batch_begin and batch_end, respectively. defpretraining_functions(self, train_set_x, batch_size): index = T.lscalar('index')  corruption_level = T.scalar('corruption')  learning_rate = T.scalar('lr')  batch_begin = index * batch_size batch_end = batch_begin + batch_size   pretrain_fns = [] fordAinself.dA_layers: cost, updates = dA.get_cost_updates(corruption_level, learning_rate) fn = theano.function( inputs=[ index, theano.Param(corruption_level, default=0.2), theano.Param(learning_rate, default=0.1)                 ], outputs=cost, updates=updates, givens={ self.x: train_set_x[batch_begin: batch_end]                 }             ) pretrain_fns.append(fn)   returnpretrain_fns The ability to dynamically adjust the learning rate particularly is very helpful and may be applied in one of two ways. Once a technique has begun to converge on an appropriate solution, it is very helpful to be able to reduce the learning rate. If you do not do this, you risk creating a situation in which the network oscillates between values located around the optimum, without ever hitting it. In some contexts, it can be helpful to tie the learning rate to the network's performance measure. If the error rate is high, it can make sense to make larger adjustments until the error rate begins to decrease! The pretraining function we've created takes the minibatch index and can optionally take the corruption level or learning rate. It performs one step of pretraining and outputs the cost value and vector of weight updates. In addition to pretraining, we need to build functions to support the fine-tuning stage, where the network is run iteratively over the validation and test data to optimize network parameters. The train_fn implements a single step of fine-tuning. The valid_score is a Python function that computes a validation score using the error measure produced by the SdA over validation data. Similarly, test_score computes the error score over test data. To get this process off the ground, we first need to set up training, validation, and test datasets. Each stage requires two datasets (set x and set y), containing the features and class labels, respectively. The required number of minibatches for validation and test is determined, and an index is created to track batch size (and provide a means of identifying at which entries a batch starts and ends). Training, validation, and testing occurs for each batch and afterward, both valid_score and test_score are calculated across all batches: defbuild_finetune_functions(self, datasets, batch_size, learning_rate):           (train_set_x, train_set_y) = datasets[0]         (valid_set_x, valid_set_y) = datasets[1]         (test_set_x, test_set_y) = datasets[2]   n_valid_batches = valid_set_x.get_value(borrow=True).shape[0] n_valid_batches /= batch_size n_test_batches = test_set_x.get_value(borrow=True).shape[0] n_test_batches /= batch_size   index = T.lscalar('index')      gparams = T.grad(self.finetune_cost, self.params)     updates = [             (param, param - gparam * learning_rate) forparam, gparamin zip(self.params, gparams)         ]   train_fn = theano.function( inputs=[index], outputs=self.finetune_cost, updates=updates, givens={ self.x: train_set_x[ index * batch_size: (index + 1) * batch_size                 ], self.y: train_set_y[ index * batch_size: (index + 1) * batch_size                 ]             }, name='train'         )   test_score_i = theano.function(             [index], self.errors, givens={ self.x: test_set_x[ index * batch_size: (index + 1) * batch_size                 ], self.y: test_set_y[ index * batch_size: (index + 1) * batch_size                 ]             }, name='test'         )   valid_score_i = theano.function(             [index], self.errors, givens={ self.x: valid_set_x[ index * batch_size: (index + 1) * batch_size                 ], self.y: valid_set_y[ index * batch_size: (index + 1) * batch_size                 ]             }, name='valid'         )     defvalid_score(): return [valid_score_i(i) for i inxrange(n_valid_batches)]   deftest_score(): return [test_score_i(i) for i inxrange(n_test_batches)]   returntrain_fn, valid_score, test_score With the training functionality in place, the following code initiates our SdA: numpy_rng = numpy.random.RandomState(89677) print '... building the model' sda = SdA( numpy_rng=numpy_rng, n_ins=280, hidden_layers_sizes=[240, 170, 100], n_outs=5     ) It should be noted that, at this point, we should be trying an initial configuration of layer sizes to see how we do. In this case, the layer sizes used here are the product of some initial testing. As we discussed, training the SdA occurs in two stages. The first is a layer-wise pretraining process that loops over all of the SdA's layers. The second is a process of fine-tuning over validation and test data. To pretrain the SdA, we provide the required corruption levels to train each layer and iterate over the layers using our previously-defined pretraining_fns: print '... getting the pretraining functions' pretraining_fns = sda.pretraining_functions(train_set_x=train_set_x, batch_size=batch_size)   print '... pre-training the model' start_time = time.clock() corruption_levels = [.1, .2, .2] for i inxrange(sda.n_layers):   for epoch inxrange(pretraining_epochs):             c = [] forbatch_indexinxrange(n_train_batches): c.append(pretraining_fns[i](index=batch_index, corruption=corruption_levels[i], lr=pretrain_lr)) print 'Pre-training layer %i, epoch %d, cost ' % (i, epoch), printnumpy.mean(c)   end_time = time.clock()   print>>sys.stderr, ('The pretraining code for file ' + os.path.split(__file__)[1] + ' ran for %.2fm' % ((end_time - start_time) / 60.)) At this point, we're able to initialize our SdA class via calling the preceding code stored within this book's GitHub repository, MasteringMLWithPython/Chapter3/SdA.py. Assessing SdA performance The SdA will take a significant length of time to run. With 15 epochs per layer and each layer typically taking an average of 11 minutes, the network will run for around 500 minutes on a modern desktop system with GPU acceleration and a single-threaded GotoBLAS. On a system without GPU acceleration, the network will take substantially longer to train and it is recommended that you use the alternative, which runs over a significantly smaller input dataset, MasteringMLWithPython/Chapter3/SdA_no_blas.py. The results are of a high quality, with a validation error score of 3.22% and test error score of 3.14%. These results are particularly impressive given the ambiguous and sometimes challenging nature of natural language processing applications. It was noticeable that the network classified more correctly for the 1-star and 5-star rating cases than for the intermediate levels. This is largely due to the ambiguous nature of unpolarized or unemotional language. Part of the reason that this input data was classifiable well was via significant feature engineering. While time-consuming and sometimes problematic, we've seen that well-executed feature engineering combined with an optimized model can deliver an excellent level of accuracy. Summary In this article, we introduced the autoencoder, an effective dimensionality reduction technique with some unique applications. We focused on the theory behind the SdA, an extension of autoencoders whereby any numbers of autoencoders are stacked in a deep architecture. Resources for Article: Further resources on this subject: Exception Handling in MySQL for Python [article] Clustering Methods [article] Machine Learning Using Spark MLlib [article]
Read more
  • 0
  • 0
  • 5584

article-image-comparing-cursor-and-set-approaches-processing-relational-data
Packt
23 Oct 2009
5 min read
Save for later

Comparing Cursor and Set Approaches in Processing Relational Data

Packt
23 Oct 2009
5 min read
To give you an idea about cursor, the following DECLARE statement creates a cursor (named zero_bal_crs), which gives you access to the rows of a payment table, rows that have 100 or less balances. DECLARE zero_bal_crs CURSOR FOR SELECT * FROM payment WHERE payment_bal < 100 You then fetch and process each of the rows sequentially; the process can be, for example, summing up rows by product group and loading the sums into a summary table. You develop the process outside of the cursor, for example, in a stored procedure. (You’ll see in the examples that the cursor-based process is procedural). Instead of processing row-by-row sequentially, you can process relational data set-by-set, without a cursor. For example, to sum all payment rows with 100 or less balances and load it into a table (named payment_100orless_bal in the following SQL statement), you can use the following SQL statement. This single SQL statement completely processes the rows that meet the condition, all at once, as a set. INSERT INTO payment_100orless_bal SELECT SUM(payment_bal) FROM payment WHERE payment_bal < 100 The following three examples, which are among those I most frequently encountered, further compare cursor-based processing with set processing. Example 1: Sequential Loop cf. One SQL Statement Our process in the first example is to summarize sales transactions by product. Listing 1 shows the DDLs for creating the sales transaction table and sales product table that stores the summary. Listing 1.1: DDLs of the Example 1 Tables CREATE TABLE sales_transactions( product_code INT, sales_amount DEC(8,2), discount INT);CREATE TABLE sales_product(product_code INT, sales_amount DEC(8,2)); Listing 1.2 shows a cursor-based stored procedure that implements the process. In this example, what the cursor (sales_crs) does is simply putting all rows in ascending order by their product codes. As you might have expected, this stored procedure applies loop with if-then-else programming construct to process the data row-by-row. Listing 1.2 Cursor Procedural solution DELIMITER $$DROP PROCEDURE IF EXISTS ex1_cursor $$USE sales $$CREATE PROCEDURE ex1_cursor()BEGINDECLARE p INT DEFAULT 0;DECLARE s DECIMAL(8,2) DEFAULT 0;DECLARE d INT DEFAULT 0;DECLARE done INT DEFAULT 0;DECLARE first_row INT DEFAULT 0;DECLARE px INT DEFAULT 0;DECLARE sx DECIMAL(8,2) DEFAULT 0;DECLARE sales_crs CURSOR FOR SELECT * FROM sales_transactions ORDER BY product_code;DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;OPEN sales_crs; REPEAT FETCH sales_crs INTO p, s, d;IF first_row = 0 THEN SET first_row = 1; SET px = p; SET sx = s * (100 - d)/100; ELSEIF NOT done THEN IF p <> px THEN INSERT INTO sales_product VALUES(px, sx); SET sx = s * (100 - d)/100; SET px = p; ELSE SET sx = sx + s * (100 - d)/100; END IF; ELSE INSERT INTO sales_product VALUES(px, sx); END IF; UNTIL done END REPEAT;CLOSE sales_crs;END $$DELIMITER ; The stored procedure in Listing 1.3 implements a set, processing to accomplish the same purpose as its foregoing cursor-based processing. You can see that this stored procedure is simpler than its cursor-based counterpart. Listing 1.3 Set Operation solution DELIMITER //USE sales //DROP PROCEDURE IF EXISTS ex1_sql //CREATE PROCEDURE ex1_sql ()BEGIN INSERT INTO sales_product SELECT product_code , SUM(sales_amount * (100 - discount)/100) FROM sales_transactions GROUP BY product_code;END //DELIMITER ; Example 2: Nested Cursor cf. Join Our 2nd example is to consolidate order by customer. While example 1 has one table, example 2 has two: order and item. The DDLs of the two tables are shown in Listing 2. To process two tables, our cursor implementation uses a nested loop (Listing 2.2) which makes it even more complex than example 1. Listing 2.1: DDL’s of the Example 2 Tables CREATE TABLE order ( , order_number INT , order_date DATE , customer_number INT);CREATE TABLE item ( , order_number INT , product_code INT , quantity INT , unit_price DEC(8,2)); Listing 2.2: Nested Cursor DELIMITER $$DROP PROCEDURE IF EXISTS ex2_cursor $$CREATE PROCEDURE ex2_cursor()BEGINDECLARE so INT;DECLARE sc INT;DECLARE io INT;DECLARE iq INT;DECLARE iu DEC(10,2);DECLARE done1 VARCHAR(5) DEFAULT 'START' ;DECLARE done2 VARCHAR(5) DEFAULT 'START' ;DECLARE sales_crs CURSOR FOR SELECT customer_number, order_number FROM sales_order ORDER BY customer_number, order_number;DECLARE order_crs CURSOR FOR SELECT order_number, quantity, unit_price FROM item WHERE order_number = so ORDER BY order_number;DECLARE CONTINUE HANDLER FOR NOT FOUND SET done1 = 'END';OPEN sales_crs; WHILE done1 <> 'END' DO FETCH sales_crs INTO sc, so; IF done1 <> 'END' THEN/* inner cursor */ OPEN oorder_crs; SET done2 = done1; WHILE done1 <> 'END' DO FETCH order_crs INTO io, iq, iu; IF done1 <> 'END' THEN INSERT INTO customer_order VALUES (sc, iq * iu); END IF;END WHILE; CLOSE order_crs; SET done1 = done2; END IF; END WHILE;CLOSE sales_crs;END $$DELIMITER ; Listing 2.3 is the set solution. We apply join in this 2nd example which makes it just a single SQL statement. You can see that this set stored procedure is again simpler than its cursor-based equivalent.    
Read more
  • 0
  • 0
  • 5580
article-image-setting-namenode-ha
Packt
20 Nov 2013
11 min read
Save for later

Setting Up NameNode HA

Packt
20 Nov 2013
11 min read
(For more resources related to this topic, see here.) We will configure our NameNode HA setup by adding several options to the core-site.xml file. The following is the structure of the file for this particular step. It will give you an idea of the XML structure, if you are not familiar with it. The header comments are stripped out: <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>fs.default.name</name> <value>hdfs://sample-cluster/</value> </property> <property> <name>ha.zookeeper.quorum</name> <value>nn1.hadoop.test.com:2181,nn2.hadoop.test.com:2181,jt1.hadoop. test.com:2181 </value> </property> </configuration> The configuration file format is pretty much self-explanatory; variables are surrounded by the <property> tag, and each variable has a name and a value. There are only two variables that we need to add at this stage. fs.default.name is the logical name of the NameNode cluster. The value hdfs://sample-cluster/ is specific to the HA setup. This is the logical name of the NameNode cluster. We will define the servers that comprise of it in the hdfs-site.xml file. In a non-HA setup, this variable is assigned a host and a port of the NameNode, since there is only one NameNode in the cluster. The ha.zookeeper.quorum variable specifies locations and ports of the ZooKeeper servers. The ZooKeeper cluster can be used by other services, such as HBase, that is why it is defined in core-site.xml. The next step is to configure the hdfs-site.xml file and add all HDFS-specific parameters there. I will omit the <property> tag and only include <name> and <value> to make the list less verbose. <name>dfs.name.dir</name> <value>/dfs/nn/</value> NameNode will use the location specified by the dfs.name.dir variable to store the persistent snapshot of HDFS metadata. This is where the fsimage file will be stored. As discussed previously, the volume on which this directory resides needs to be backed by RAID. Losing this volume means losing NameNode completely. The /dfs/nn path is an example, however you are free to choose your own. You can actually specify several paths with a dfs.name.dir value, separating them by commas. NameNode will mirror the metadata files in each directory specified. If you have a shared network storage available, you can use it as one of the destinations for HDFS metadata. This will provide additional offsite backups. <name>dfs.nameservices</name> <value>sample-cluster</value> The dfs.nameservices variable specifies the logical name of the NameNode cluster and should be replaced by something that makes sense to you, such as prod-cluster or stage-cluster. The value of dfs.nameservices must match the value of fs.default.name from the core-site.xml file. <name>dfs.ha.namenodes.sample-cluster</name> <value>nn1,nn2</value> Here, we specify the NameNodes that make up our HA cluster setup. These are logical names, not real server hostnames or IPs. These logical names will be referenced in other configuration variables. <name>dfs.namenode.rpc-address.sample-cluster.nn1</name> <value>nn1.hadoop.test.com:8020</value> <name>dfs.namenode.rpc-address.sample-cluster.nn2</name> <value>nn2.hadoop.test.com:8020</value> This pair of variables provide mapping from logical names like nn1 and nn2 to the real host and port value. By default, NameNode daemons use port 8020 for communication with clients and each other. Make sure this port is open for the cluster nodes. <name>dfs.namenode.http-address.sample-cluster.nn1</name> <value>nn1.hadoop.test.com:50070</value> <name>dfs.namenode.http-address.sample-cluster.nn2</name> <value>nn2.hadoop.test.com:50070</value> Each NameNode daemon runs a built-in HTTP server, which will be used by the NameNode web interface to expose various metrics and status information about HDFS operations. Additionally, standby NameNode uses HTTP calls to periodically copy the fsimage file from the primary server, perform the checkpoint operation, and ship it back. <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://nn1.hadoop.test.com:8485;nn2.hadoop.test.com:8485; jt1.hadoop.test.com:8485/sample-cluster</value> The dfs.namenode.shared.edits.dir variable specifies the setup of the JournalNode cluster. In our configuration, there are three JournalNodes running on nn1, nn2, and nn3. Both primary and standby nodes will use this variable to identify which hosts they should contact to send or receive new changes from editlog. <name>dfs.journalnode.edits.dir</name> <value>/dfs/journal</value> JournalNodes need to persist editlog changes that are being submitted to them by the active NameNode. The dfs.journalnode.edits.dir variable specifies the location on the local filesystem where editlog changes will be stored. Keep in mind that this path must exist on all JournalNodes and the ownership of all directories must be set to hdfs:hdfs (user and group). <name>dfs.client.failover.proxy.provider.sample-cluster</name> <value>org.apache.hadoop.hdfs.server.namenode.ha. ConfiguredFailoverProxyProvider</value> In an HA setup, clients that access HDFS need to know which NameNode to contact for their requests. The dfs.client.failover.proxy.provider.sample-cluster variable specifies the Java class name, which will be used by clients for determining the active NameNode. At the moment, there is only ConfiguredFailoverProxyProvider available. <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> The dfs.ha.automatic-failover.enabled variable indicates if the NameNode cluster will use a manual or automatic failover. <name>dfs.ha.fencing.methods</name> <value>sshfence shell(/bin/true) </value> Orchestrating failover in a cluster setup is a complicated task involving multiple steps. One of the common problems that is not unique to the Hadoop cluster, but affects any distributed systems, is a "split-brain" scenario. Split-brain is a case where two NameNodes decide they both play an active role and start writing changes to the editlog. To prevent such an issue from occurring, the HA configuration maintains a marker in ZooKeeper, clearly stating which NameNode is active, and JournalNodes accepts writes only from that node. To be absolutely sure that the two NameNodes don't become active at the same time, a technique called fencing is used during failover. The idea is to force the shutdown of the active NameNode before transferring the active state to a standby. There are two fencing methods currently available: sshfence and shell. sshfence. These require a passwordless ssh access as a user that starts the NameNode daemon, from the active NameNode to the standby and vice versa. By default, this is the hdfs user. The fencing process checks if there is anyone listening on a NameNode port using the nc command, and if the port is found busy, it tries to kill the NameNode process. Another option for dfs.ha.fencing.methods is shell. This will execute the specified shell script to perform fencing. It is important to understand that failover will fail if fencing fails. In our case, we specified two options, the second one always returns success. This is done for workaround cases where the primary NameNode machine goes down and the ssh method will fail, and no failover will be performed. We want to avoid this, so the second option would be to failover anyway, even without fencing, which, as already mentioned, is safe with our setup. To achieve this, we specify two fencing methods, which will be tried by ZKFC in the order of: if the first one fails, the second one will be tried. In our case, the second one will always return success and failover will be initiated, even if the server running the primary NameNode is not available via ssh. <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/var/lib/hadoop-hdfs/.ssh/id_rsa</value> The last option we will need to configure for NameNode HA setup is the ssh key, which will be used by sshfence. Make sure you change the ownership for this file to hdfs user. Two keys need to be generated, one for the primary and one for the secondary NameNode. It is a good idea to test ssh access as an hdfs user in both directions to make sure it is working fine. The hdfs-site.xml configuration file is now all set for testing the HA setup. Don't forget to sync these configuration files to all the nodes in the cluster. The next thing that needs to be done is to start JournalNodes. Execute this command on nn1, nn2, and jt1 a root user: # service hadoop-hdfs-journalnode start With CDH, it is recommended to always use the service command instead of calling scripts in /etc/init.d/ directly. This is done to guarantee that all environment variables are set up properly before the daemon is started. Always check the logfiles for daemons. Now, we need to initially format HDFS. For this, run the following command on nn1: # sudo -u hdfs hdfs namenode –format This is the initial setup of the NameNode, so we don't have to worry about affecting any HDFS metadata, but be careful with this command, because it will destroy any previous metadata entries. There is no strict requirement to run format command on nn1, but to make it easier to follow, let's assume we want nn1 to become an active NameNode. Format command will also format the storage for the JournalNodes. The next step is to create an entry for the HA cluster in ZooKeeper, and start NameNode and ZKFC on the first NameNode. In our case, this is nn1: # sudo -u hdfs hdfs zkfc -formatZK # service hadoop-hdfs-namenode start # service hadoop-hdfs-zkfc start Check the ZKFC log file (by default, it is in /var/log/hadoop-hdfs/) to make sure nn1 is now an active NameNode: INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at nn1.hadoop.test.com/192.168.0.100:8020 active... INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at nn1.hadoop.test.com/192.168.0.100:8020 to active state To activate the secondary NameNode, an operation called bootstrapping needs to be performed. To do this, execute the following command on nn2: # sudo -u hdfs hdfs namenode –bootstrapStandby This will pull the current filesystem state from active NameNode and synchronize the secondary NameNode with the JournalNodes Quorum. Now, you are ready to start the NameNode daemon and the ZKFC daemon on nn2. Use the same commands that you used for nn1. Check the ZKFC log file to make sure nn2 successfully acquired the secondary NameNode role. You should see the following messages at the end of the logfile: INFO org.apache.hadoop.ha.ZKFailoverController: ZK Election indicated that NameNode at nn2.hadoop.test.com/192.168.0.101:8020 should become standby INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at nn2.hadoop.test.com/192.168.0.101:8020 to standby state This is the last step in configuring NameNode HA. It is a good idea to verify if automatic failover is configured correctly, and if it will behave as expected in the case of a primary NameNode outage. Testing failover in the cluster setup stage is easier and safer than discovering that failover doesn't work during production stage and causing a cluster outage. You can perform a simple test: kill the primary NameNode daemon and verify if the secondary takes over its role. After that, bring the old primary back online and make sure it takes over the secondary role. You can use execute the following command to get the current status of NameNode nn1: # sudo -u hdfs hdfs haadmin -getServiceState nn1 The hdfs haadmin command can also be used to initiate a failover in manual failover setup. At this point, you have a fully configured and functional NameNode HA setup. Summary We saw in this article how to configure Hadoop's NameNode HA. Resources for Article: Further resources on this subject: Advanced Hadoop MapReduce Administration [Article] Managing a Hadoop Cluster [Article] Making Big Data Work for Hadoop and Solr [Article]
Read more
  • 0
  • 0
  • 5573

article-image-thorium-and-salt-api
Packt
01 Nov 2016
17 min read
Save for later

Thorium and Salt API

Packt
01 Nov 2016
17 min read
In this article by Joseph Hall, the author of the book Mastering SaltStack Second Edition, we learn about the basics of Thorium and how it helps us in a Salt ecosystem. We are also introduced to the Salt API and how to set up its components as well as creating security certificates. (For more resources related to this topic, see here.) Using Thorium The Thorium system is another component of Salt with the ability to watch the event bus and react based on what it sees there. But the ideas behind it are much different than with the reactor. A word on engines Thorium is one of the engines that started shipping with Salt in version 2016.3. Engines are a type of long-running process that can be written to work with the master or minion. Like other module types, they have access to the Salt configuration and certain Salt subsystems. Engines are separate processes that are managed by Salt. The event reactor runs inside the Salt processes themselves, which means that long-running reactor operations can affect the rest of Salt. Because Thorium is an engine, it does not suffer from this limitation. Looking at Thorium basics Like the reactor, Thorium watches the event bus. But unlike the reactor, which is configured entirely via SLS files, Thorium uses its own subsystem of modules (which are written in Python) and SLS files. Because these modules and SLS files use the state compiler, much of the functionality has been carried over. In order to use Thorium, there are a few steps that you must complete. These steps work together to form the basis of your Thorium setup, so be careful not to skip any. Enabling Thorium First, as an engine, you need to enable Thorium in the master configuration file using the engines directive: engines: - thorium: {} Because Thorium is so heavily configured using its own files, no configuration needs to be passed in at this point. However, engines do need a dictionary of some sort passed in, so we pass in an empty one. Setting up the Thorium directory tree With Thorium configured, we need to create a directory to store Thorium SLS files in. By default, this is /srv/thorium/. Go ahead and create that: # mkdir /srv/thorium/ If you’d like to change this directory, you may do so in the master configuration file: thorium_roots: base: - /srv/thorium-alt/ Like the state system, Thorium requires a top.sls file. This is the first of many similarities you’ll find between the two subsystems. As with /srv/salt/top.sls, you need to specify an environment, a target, and a list of SLS files: base: '*': - thorium_test To be honest, the environment and target really don’t mean much; they are artifacts from the state system, which weren’t designed to do anything special inside of Thorium. That said, the target does actually have some useful purposes. The target here doesn’t refer to any minions. Rather, it refers to the master that this top file applies to. For example, if your master’s ID is moe and you set a target of curly, then this top file won’t be evaluated for that master. If you were to check the /var/log/salt/master file, you would find this: Sound confusing? In a single-master, non-syndicated environment, it probably is. In such an environment, go ahead and set the target to *. But in an environment in which multiple masters are present, you may wish to divide the workload between them. Take a look at this top.sls file: base: 'monitoring-master': - alerts - graphs 'packaging-master': - redhat-pkgs - debian-pkgs In this multi-master environment, the masters may work in concert to manage jobs among the minions, but one master will also be tasked with looking for monitoring-related events and processing them, while the other will handle packaging-related events. There is a component of Thorium that we haven’t discussed yet called the register. We’ll get to it in a moment, but this is a good time to point out that the register is not shared. This means that if you assign two different masters to handle the same events, any actions performed by one master will be invisible to the other. The right hand won’t know what the left is doing, as it were. As you may expect, the list following each target specifies a set of SLS files to be evaluated. But whereas state SLS files are only evaluated when you kick off a state run (state.highstate, for instance), Thorium SLS files are evaluated at regular intervals. By default, these intervals are set to every half second. You can change that interval in the master configuration file: thorium_interval: 0.5 Once you have your top.sls file configured, it’s time to set up some SLS files. Writing Thorium SLS files Let’s go ahead and create /srv/thorium/thorium-test.sls with the following content in it: shell_test: local.cmd: - tgt: myminion - func: cmd.run - arg: - echo 'thorium success' > /tmp/thorium.txt I wouldn’t restart your master yet if I were you. First, let’s talk about what we’re looking at here. This should look very familiar to you, with a few differences. As you would expect, shell_test is the ID of this code block; local.cmd refers to the module and function that will be used, and everything that follows is arguments to that function. The local module is a Thorium-specific module. Execution, state, runner, and other modules are not available in Thorium without using a Thorium module that wraps them. The local module is one such wrapper, which provides access to execution modules. As such, tgt refers to the target that the module will be executed on, func is the module and function that will be executed, and arg is a list of ordered arguments. If you like, you may use kwarg instead to specify keyword arguments. Because state modules are accessed via the state execution module, local.cmd would also be used to kick those off; runner.cmd is also available to issue commands using the runner subsystem. Now, why did I tell you not to restart your master yet? Because if you did, this SLS file would run every half second, writing out to /tmp/thorium.txt over and over again. In order to keep it from running so often, we need to gate it somehow. Using requisites Because Thorium uses the state compiler, all state requisites are available, and they all function as you would expect. Let’s go ahead and add another code block and alter our first one a little bit: checker: check.event: - value: trigger - name: salt/thorium/*/test shell_test: local.cmd: - tgt: myminion - func: cmd.run - arg: - echo 'thorium success' > /tmp/thorium.txt - require: - check: checker The check module has a number of functions that are designed to compare a piece of data against an event and return True if the specified conditions are met. In this case, we’re using check.event, which looks at a given tag and returns True if an event comes in and matches it. The tag that we are looking for is salt/thorium/*/test, which is intended to look for salt/thorium/<minion_id>/test. The event must also have a variable called checker in its payload, with a value of trigger. We have also added a require requisite to the shell_test code block, which will prevent that block from running unless the right event comes in. Now that we’re set up, go ahead and restart the master and a minion called myminion, and issue the following command from the minion: # salt-call event.fire_master '{"checker":"trigger"}' 'salt/thorium/myminion/test' local: True You may need to wait a second or two for the event to be processed and the command from shell_test to be sent. But then you should be able to see a file called /tmp/thorium.txt and read its contents: # cat /tmp/thorium.txt thorium success This particular SLS, as it is now, mimics the functionality of the reactor system, albeit with a slightly more complex setup. Let’s take a moment now to go beyond the reactor. Using the register Thorium isn’t just another reactor. Even if it were, just running in its own process space makes it more valuable than the old reactor. But the true value of Thorium comes with the register. The register is Thorium’s own in-memory database, which persists across executions. A value that is placed in the register at one point in time is will still be there a half hour later unless the master is restarted. Is the register really that fragile? At the moment, yes. And as I stated before, it’s also not shared between systems. However, it is possible to make a copy of the register on disk by adding the following code block: myregisterfile: file.save This will cause the register to be written to a file called myregisterfile at the following location: /var/cache/salt/master/thorium/saves/myregisterfile At the time of writing this, that file will not be reloaded into memory when the master restarts. We’re going to go ahead and alter our SLS file. The shell_test code block doesn’t need to change, but the checker code block will. Remove the name field and change the function from check.event to check.contains: checker: check.contains: - value: trigger We’re still looking for a payload with a variable called checker and a value called checker, but we’re going to look at the tag somewhere else: myregister: reg.set: - add: checker - match: salt/thorium/*/test In this code block, myregister is the name of the register that you’re going to write to. The reg.set function will add a variable to that register that contains the specified piece of the payload. In this case, it will grab the variable from the payload called checker and add its associated value. However, it will only add this information to the registry if the tag on the event in question matches the match specification (salt/thorium/*/test). Go ahead and restart the master, and then fire the same event to the master: # salt-call event.fire_master '{"checker":"trigger"}' 'salt/thorium/myminion/test' local: True If you’ve added the file.save code block from before, we can go ahead and take a look at the register: # cat /var/cache/salt/master/thorium/saves/myregisterfile {"myregister": {"val": "set(['trigger'])"}} Looking forward The Thorium system is pretty new, so it’s still filling out. The value of the registry is that data can be aggregated to it and analyzed in real time. Unfortunately, at the time of writing this, the functions to analyze that data do not yet exist. Understanding the Salt API We've spent some time looking at how to send requests, but many users would argue that receiving requests is just as important, if not more so. Let's take a moment to understand the Salt API. What is the Salt API? Very simply, the Salt API is a REST interface wrapped around Salt. But that doesn't tell you the whole story. The salt command is really just a command-line interface for Salt. In fact, each of the other Salt commands (salt-call, salt-cloud, and so on) is really just a way to access various parts of Salt from the command line. The Salt API provides a way to access Salt from a different interface: HTTP (or HTTPS, preferably). Because web protocols are so ubiquitous, the Salt API allows software, written in any language that has the capability of interacting with web servers, to take advantage of it. Setting up the Salt API Being a REST interface, the Salt API acts as a web server over and above Salt. But it doesn't actually provide the server interface itself. It uses other web frameworks to provide those services and then acts as more of a middleman between them and Salt. The modules that are supported for this are: CherryPy Tornado WSGI These modules are set up in the master configuration file. Each has its own set of configuration parameters and possible dependencies. Let's take a look at each one. CherryPy CherryPy is a minimalist web framework that is designed to be very Pythonic. Because it is based around creating web code in the same way that other Python code is created, it is said to result in code that is much smaller and more quickly developed. It has a mature codebase and a number of notable users. It has also been the de facto module of the Salt API for some time. This module does require that the CherryPy package (usually called python-cherrypy) be installed. The basic setup for CherryPy doesn't involve much configuration. At a minimum, you should have the following: rest_cherrypy: port: 8080 ssl_crt: /etc/pki/tls/certs/localhost.crt ssl_key: /etc/pki/tls/certs/localhost.key We'll discuss creating certificates in a moment, but first let's talk about configuration in general. There are a number of configuration parameters available for this module, but we'll focus on the more common ones here: port: This is required. It's the port for the Salt API to listen on. host: Normally, the Salt API listens on all available interfaces (0.0.0.0). If you are in an an environment where you need to provide services only to one interface, then provide the IP address (that is, 10.0.0.1) here. ssl_crt: This is the path to your SSL certificate. We'll cover this in a moment. ssl_key: This is the path to the private key for the SSL certificate. Again, we'll cover this in a moment. debug: If you are setting up the Salt API for the first time, setting this to True can be very helpful. But once you are up and running, make sure to remove this option or explicitly set it to False. disable_ssl: It is highly recommended that the default value of False be used here. Even when just getting started, self-signed certificates are better than setting this to True. Why? Because nothing is as permanent as temporary, and at least self-signed certificates will remind you each time that you need to get a real set of certificates in place. Don't be complacent for the sake of learning. root_prefix: Normally, the Salt API will serve from the root path of the server (that is, https://saltapi.example.com/), but if you have several applications that you're serving from the same host or you just want to be more specific, you can change this. The default is /, but you could set it to /exampleapi in order to serve REST services from https://saltapi.example.com/exampleapi, for example. webhook_url: If you are using webhooks, they need their own entry point. By default, this is set to /hook, which in our example would serve from https://saltapi.example.com/hook. webhook_disable_auth: Normally, the Salt API requires authentication, but this is quite commonly not possible with third-party applications that need to call it over a webhook. This allows webhooks to not require authentication. We'll go more in depth on this in a moment. Tornado Tornado is a somewhat newer framework that was written by Facebook. It is also newer than Salt but is quickly becoming the web framework of choice inside Salt itself. In fact, it is used so much inside Salt that it is now considered a hard dependency for Salt and will be available on all newer installations. Tornado doesn't have as many configuration options inside the Salt API as CherryPy. The ones that are supported (as defined in the CherryPy section) are: port ssl_crt ssl_key debug disable_ssl While the Tornado module doesn't support nearly as much functionality as the CherryPy module just yet, keep an eye on it; it may become the new de facto Salt API module. WSGI WSGI, or Web Server Gateway Interface, is a Python standard, defined in PEP 333. Direct support for it ships with Python itself, so no external dependencies are required, but this module is also pretty basic. The only configuration option to worry about here is port. However, this module is useful in that it allows the Salt API to be run under any WSGI-compliant web server, such as Apache with mod_wsgi or Nginx with FastCGI. Because this module does not provide any sort of SSL-based security, it is recommended that one of these options be used, with those third-party web servers being properly configured with the appropriate SSL settings. Creating SSL certificates It is highly advisable to use an SSL certificate for the Salt API even if you currently only plan to use it on a local, secured network. You should probably also purchase a certificate that is signed by a certificate authority (CA). When you get to this point, the CA will provide instructions on how to create one using their system. However, for now, we can get by with a self-signed certificate. There are a number of guides online for creating self-signed certificates, but finding one that is easy to understand is somewhat more difficult. The following steps will generate both an SSL certificate and the key to use it on a Linux system: First, we'll need to generate the key. Don't worry about the password—just enter one for now, take note of it, and we'll strip it out in a moment. # openssl genrsa -des3 -out server.key 2048 Generating RSA private key, 2048 bit long modulus ................++++++ ..............................................++++++ e is 65537 (0x10001) Enter pass phrase for server.key: Verifying - Enter pass phrase for server.key: Once you have the key, you need to use it to generate a certificate signing request, or CSR. You will be asked a number of questions about you that are important if you want a certificate signed by a CA. On your internal network, it's somewhat less important. # openssl req -new -key server.key -out server.csr Enter pass phrase for server.key: You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:Utah Locality Name (eg, city) []:Salt Lake City Organization Name (eg, company) [Internet Widgits Pty Ltd]:My Company, LLC Organizational Unit Name (eg, section) []: Common Name (e.g. server FQDN or YOUR name) []: Email Address []:me@example.com Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []: An optional company name []: At this point, we can go ahead and strip the password from the key. # cp server.key server.key.org # openssl rsa -in server.key.org -out server.key Enter pass phrase for server.key.org: writing RSA fkey Finally, we'll create a self-signed certificate. # openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt Signature ok subject=/C=US/ST=Utah/L=Salt Lake City/O=My Company, LLC/emailAddress=me@example.com Getting Private key At this point, you will have four files: server.crt server.csr server.key server.key.org Copy server.crt to the path specified for ssl_crt and server.key to the path specified for ssl_key. Summary In this article, we have learned how to setup the Thorium directory tree, using requisites and also using the register. We have also studied how to setup the Salt API. The modules that are supported for Salt API are CherryPy, Tornado, and WSGI. We have also gone through creating SSL certificates that is highly advisable to use an SSL certificate for the Salt API. Resources for Article: Further resources on this subject: Salt Configuration [article] Diving into Salt Internals [article] Introducing Salt [article]
Read more
  • 0
  • 0
  • 5572
Modal Close icon
Modal Close icon