How-To Tutorials

KeystoneJS is a content management framework for node.js. It is an easy-to-use system that does all the hard work of making a website for you. This article works through a simple example to get you started with KeystoneJS. Initial Setup KeystoneJS comes paired with a generator to make setup simple. You'll need to have node.js and mongodb installed before you begin. To generate your site, all you need to do is run npm install -g generator-keystone and then yo keystone. You'll be asked a few questions, and after a while your site is ready. Running node keystone, you'll find a site with a readymade blog, gallery and contact form, but the main feature is of KeystoneJS is the admin UI. Navigate to localhost:3000/keystone and sign in with the default credentials and you'll be able to manage all the content on your site from a user-friendly interface. Take a look around your site and the code so that you're familiar with it, and it's also worth having a read through the documentation. Keystone Models You now have a site up and running, but what if you need more than just a blog and a gallery? Perhaps, you would like a page to display upcoming events. No problem, to achieve this we create a model. Open the models folder in your file browser and you will be able to see the existing models, User.js for example. We're going to add our own model; create a new file called Event.js in the models folder. Say, our event should have both a start and an end time, a name and a description. Then our model will look like this: var keystone = require('keystone'); var Types = keystone.Field.Types; var Event = new keystone.List('Event'); Event.add({ name: { type: Types.Name, required: true, index: true }, description: { type: Types.Textarea }, start: { type: Types.Datetime }, end: { type: Types.Datetime } }); Event.register(); Now restart your app. Under the hood, KeystoneJS is managing all the database schemas for you, and if you sign back in to the admin, UI you'll see that there is now a page to manage your events. All that was required was to create a new model, and Keystone wrote the entire backend for you—this shows the power of Keystone. You don't have to spend your time writing the backend for your site, and are free to focus on the client-facing side of things. Routes and Templates We have created our model and are now able to log in to the admin UI and manage our events. However, we still need to display these events to our website viewers. This is done in two parts; a route is used to obtain the data from the database and makes this data available to the template, which displays the data. First, create the route. Open a new file, routes/views/events.js, and enter the following code: var keystone = require('keystone'); exports = module.exports = function(req, res) { var view = new keystone.View(req, res); var locals = res.locals; // Set locals locals.section = 'events'; // Load the events view.on('init', function(next) { var q = keystone.list('Event').model.find(); q.exec(function(err, results) { locals.data.events = results; next(err); }); }); // Render the view view.render('post'); }; You can now create your template. The events will be available to the template as data.events, because we have set locals.data.events in the route. KeystoneJS gives you the option of which template engine to use. The default is jade, so we will use this as the example here, but you can easily adapt the code to any other engine, and if you get stuck, a good place to start is the blog post template. Templates are stored in templates/views, so create templates/views/events.js with the following code: extends ../layouts/default mixin event(event) h2 event.name p if event.start | start: #{event._.start.format('MMMM Do, YYYY')} p if event.end | end: #{event._.end.format('MMMM Do, YYYY')} p if event.description | details: event.description block content .container: .row .events each event in data.events +event(event) This is by no means a well-designed page, but will do for this example. We're almost done, but if you go to /events in your web browser, you'll get a 404 error. That's because we haven't told our route controllers about the new page yet. This is done in routes/index.js and you just need to add the line app.get('/events', routes.views.events);. This tells your app to send any get requests for /events to your new route, which in turn renders the new template. You can also add your new events page to your header by simply adding { label: 'Events',      key: 'events',     href: '/events' }, to routes/middleware.js. The key in this should match the res.locals.section in the route we created. Conclusion By simply running yo keystone and adding just over 50 lines of code, we've created an events page to display our events. You can log in to the admin UI and create, update and delete events; and your website will update automatically. This really highlights what keystone does. We don’t have to spend our time configuring all the node modules and writing the backend of our server; keystone has done all the work for us. This means we can dedicate all our time to making our client-facing website look as good as possible. About the Author Jake Stockwin is a third-year mathematics and statistics undergraduate at the University of Oxford, and a novice full-stack developer. He has a keen interest in programming, both in his academic studies and in his spare time. Next year, he plans to write his dissertation on reinforcement learning, an area of machine learning. Over the past few months, he has designed websites for various clients and has begun developing in Node.js.

(For more resources related to this topic, see here.) In this article, we will implement the Batch and Service layers to complete the architecture. There are some key concepts underlying this big data architecture: Immutable state Abstraction and composition Constrain complexity Immutable state is the key, in that it provides true fault-tolerance for the architecture. If a failure is experienced at any level, we can always rebuild the data from the original immutable data. This is in contrast to many existing data systems, where the paradigm is to act on mutable data. This approach may seem simple and logical; however, it exposes the system to a particular kind of risk in which the state is lost or corrupted. It also constrains the system, in that you can only work with the current view of the data; it isn't possible to derive new views of the data. When the architecture is based on a fundamentally immutable state, it becomes both flexible and fault-tolerant. Abstractions allow us to remove complexity in some cases, and in others they can introduce complexity. It is important to achieve an appropriate set of abstractions that increase our productivity and remove complexity, but at an appropriate cost. It must be noted that all abstractions leak, meaning that when failures occur at a lower abstraction, they will affect the higher-level abstractions. It is therefore often important to be able to make changes within the various layers and understand more than one layer of abstraction. The designs we choose to implement our abstractions must therefore not prevent us from reasoning about or working at the lower levels of abstraction when required. Open source projects are often good at this, because of the obvious access to the code of the lower level abstractions, but even with source code available, it is easy to convolute the abstraction to the extent that it becomes a risk. In a big data solution, we have to work at higher levels of abstraction in order to be productive and deal with the massive complexity, so we need to choose our abstractions carefully. In the case of Storm, Trident represents an appropriate abstraction for dealing with the data-processing complexity, but the lower level Storm API on which Trident is based isn't hidden from us. We are therefore able to easily reason about Trident based on an understanding of lower-level abstractions within Storm. Another key issue to consider when dealing with complexity and productivity is composition. Composition within a given layer of abstraction allows us to quickly build out a solution that is well tested and easy to reason about. Composition is fundamentally decoupled, while abstraction contains some inherent coupling to the lower-level abstractions—something that we need to be aware of. Finally, a big data solution needs to constrain complexity. Complexity always equates to risk and cost in the long run, both from a development perspective and from an operational perspective. Real-time solutions will always be more complex than batch-based systems; they also lack some of the qualities we require in terms of performance. Nathan Marz's Lambda architecture attempts to address this by combining the qualities of each type of system to constrain complexity and deliver a truly fault-tolerant architecture. We divided this flow into preprocessing and "at time" phases, using streams and DRPC streams respectively. We also introduced time windows that allowed us to segment the preprocessed data. In this article, we complete the entire architecture by implementing the Batch and Service layers. The Service layer is simply a store of a view of the data. In this case, we will store this view in Cassandra, as it is a convenient place to access the state alongside Trident's state. The preprocessed view is identical to the preprocessed view created by Trident, counted elements of the TF-IDF formula (D, DF, and TF), but in the batch case, the dataset is much larger, as it includes the entire history. The Batch layer is implemented in Hadoop using MapReduce to calculate the preprocessed view of the data. MapReduce is extremely powerful, but like the lower-level Storm API, is potentially too low-level for the problem at hand for the following reasons: We need to describe the problem as a data pipeline; MapReduce isn't congruent with such a way of thinking Productivity We would like to think of a data pipeline in terms of streams of data, tuples within the stream and predicates acting on those tuples. This allows us to easily describe a solution to a data processing problem, but it also promotes composability, in that predicates are fundamentally composable, but pipelines themselves can also be composed to form larger, more complex pipelines. Cascading provides such an abstraction for MapReduce in the same way as Trident does for Storm. With these tools, approaches, and considerations in place, we can now complete our real-time big data architecture. There are a number of elements, that we will update, and a number of elements that we will add. The following figure illustrates the final architecture, where the elements in light grey will be updated from the existing recipe, and the elements in dark grey will be added in this article: Implementing TF-IDF in Hadoop TF-IDF is a well-known problem in the MapReduce communities; it is well-documented and implemented, and it is interesting in that it is sufficiently complex to be useful and instructive at the same time. Cascading has a series of tutorials on TF-IDF at http://www.cascading.org/2012/07/31/cascading-for-the-impatient-part-5/, which documents this implementation well. For this recipe, we shall use a Clojure Domain Specific Language (DSL) called Cascalog that is implemented on top of Cascading. Cascalog has been chosen because it provides a set of abstractions that are very semantically similar to the Trident API and are very terse while still remaining very readable and easy to understand. Getting ready Before you begin, please ensure that you have installed Hadoop by following the instructions at http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/. How to do it… Start by creating the project using the lein command: lein new tfidf-cascalog Next, you need to edit the project.clj file to include the dependencies: (defproject tfidf-cascalog "0.1.0-SNAPSHOT" :dependencies [[org.clojure/clojure "1.4.0"] [cascalog "1.10.1"] [org.apache.cassandra/cassandra-all "1.1.5"] [clojurewerkz/cassaforte "1.0.0-beta11-SNAPSHOT"] [quintona/cascading-cassandra "0.0.7-SNAPSHOT"] [clj-time "0.5.0"] [cascading.avro/avro-scheme "2.2-SNAPSHOT"] [cascalog-more-taps "0.3.0"] [org.apache.httpcomponents/httpclient "4.2.3"]] :profiles{:dev{:dependencies[[org.apache.hadoop/hadoop-core "0.20.2-dev"] [lein-midje "3.0.1"] [cascalog/midje-cascalog "1.10.1"]]}}) It is always a good idea to validate your dependencies; to do this, execute lein deps and review any errors. In this particular case, cascading-cassandra has not been deployed to clojars, and so you will receive an error message. Simply download the source from https://github.com/quintona/cascading-cassandra and install it into your local repository using Maven. It is also good practice to understand your dependency tree. This is important to not only prevent duplicate classpath issues, but also to understand what licenses you are subject to. To do this, simply run lein pom, followed by mvn dependency:tree. You can then review the tree for conflicts. In this particular case, you will notice that there are two conflicting versions of Avro. You can fix this by adding the appropriate exclusions: [org.apache.cassandra/cassandra-all "1.1.5" :exclusions [org.apache.cassandra.deps/avro]] We then need to create the Clojure-based Cascade queries that will process the document data. We first need to create the query that will create the "D" view of the data; that is, the D portion of the TF-IDF function. This is achieved by defining a Cascalog function that will output a key and a value, which is composed of a set of predicates: (defn D [src] (let [src (select-fields src ["?doc-id"])] (<- [?key ?d-str] (src ?doc-id) (c/distinct-count ?doc-id :> ?n-docs) (str "twitter" :> ?key) (str ?n-docs :> ?d-str)))) You can define this and any of the following functions in the REPL, or add them to core.clj in your project. If you want to use the REPL, simply use lein repl from within the project folder. The required namespace (the use statement), require, and import definitions can be found in the source code bundle. We then need to add similar functions to calculate the TF and DF values: (defn DF [src] (<- [?key ?df-count-str] (src ?doc-id ?time ?df-word) (c/distinct-count ?doc-id ?df-word :> ?df-count) (str ?df-word :> ?key) (str ?df-count :> ?df-count-str))) (defn TF [src] (<- [?key ?tf-count-str] (src ?doc-id ?time ?tf-word) (c/count ?tf-count) (str ?doc-id ?tf-word :> ?key) (str ?tf-count :> ?tf-count-str))) This Batch layer is only interested in calculating views for all the data leading up to, but not including, the current hour. This is because the data for the current hour will be provided by Trident when it merges this batch view with the view it has calculated. In order to achieve this, we need to filter out all the records that are within the current hour. The following function makes that possible: (deffilterop timing-correct? [doc-time] (let [now (local-now) interval (in-minutes (interval (from-long doc-time) now))] (if (< interval 60) false true)) Each of the preceding query definitions require a clean stream of words. The text contained in the source documents isn't clean. It still contains stop words. In order to filter these and emit a clean set of words for these queries, we can compose a function that splits the text into words and filters them based on a list of stop words and the time function defined previously: (defn etl-docs-gen [rain stop] (<- [?doc-id ?time ?word] (rain ?doc-id ?time ?line) (split ?line :> ?word-dirty) ((c/comp s/trim s/lower-case) ?word-dirty :> ?word) (stop ?word :> false) (timing-correct? ?time))) We will be storing the outputs from our queries to Cassandra, which requires us to define a set of taps for these views: (defn create-tap [rowkey cassandra-ip] (let [keyspace storm_keyspace column-family "tfidfbatch" scheme (CassandraScheme. cassandra-ip "9160" keyspace column-family rowkey {"cassandra.inputPartitioner""org.apache.cassandra.dht.RandomPartitioner" "cassandra.outputPartitioner" "org.apache.cassandra.dht.RandomPartitioner"}) tap (CassandraTap. scheme)] tap)) (defn create-d-tap [cassandra-ip] (create-tap "d"cassandra-ip)) (defn create-df-tap [cassandra-ip] (create-tap "df" cassandra-ip)) (defn create-tf-tap [cassandra-ip] (create-tap "tf" cassandra-ip)) The way this schema is created means that it will use a static row key and persist name-value pairs from the tuples as column:value within that row. This is congruent with the approach used by the Trident Cassandra adaptor. This is a convenient approach, as it will make our lives easier later. We can complete the implementation by a providing a function that ties everything together and executes the queries: (defn execute [in stop cassandra-ip] (cc/connect! cassandra-ip) (sch/set-keyspace storm_keyspace) (let [input (tap/hfs-tap (AvroScheme. (load-schema)) in) stop (hfs-delimited stop :skip-header? true) src (etl-docs-gen input stop)] (?- (create-d-tap cassandra-ip) (D src)) (?- (create-df-tap cassandra-ip) (DF src)) (?- (create-tf-tap cassandra-ip) (TF src)))) Next, we need to get some data to test with. I have created some test data, which is available at https://bitbucket.org/qanderson/tfidf-cascalog. Simply download the project and copy the contents of src/data to the data folder in your project structure. We can now test this entire implementation. To do this, we need to insert the data into Hadoop: hadoop fs -copyFromLocal ./data/document.avro data/document.avro hadoop fs -copyFromLocal ./data/en.stop data/en.stop Then launch the execution from the REPL: => (execute "data/document" "data/en.stop" "127.0.0.1") How it works… There are many excellent guides on the Cascalog wiki (https://github.com/nathanmarz/cascalog/wiki), but for completeness's sake, the nature of a Cascalog query will be explained here. Before that, however, a revision of Cascading pipelines is required. The following is quoted from the Cascading documentation (http://docs.cascading.org/cascading/2.1/userguide/htmlsingle/): Pipe assemblies define what work should be done against tuple streams, which are read from tap sources and written to tap sinks. The work performed on the data stream may include actions such as filtering, transforming, organizing, and calculating. Pipe assemblies may use multiple sources and multiple sinks, and may define splits, merges, and joins to manipulate the tuple streams. This concept is embodied in Cascalog through the definition of queries. A query takes a set of inputs and applies a list of predicates across the fields in each tuple of the input stream. Queries are composed through the application of many predicates. Queries can also be composed to form larger, more complex queries. In either event, these queries are reduced down into a Cascading pipeline. Cascalog therefore provides an extremely terse and powerful abstraction on top of Cascading; moreover, it enables an excellent development workflow through the REPL. Queries can be easily composed and executed against smaller representative datasets within the REPL, providing the idiomatic API and development workflow that makes Clojure beautiful. If we unpack the query we defined for TF, we will find the following code: (defn DF [src] (<- [?key ?df-count-str] (src ?doc-id ?time ?df-word) (c/distinct-count ?doc-id ?df-word :> ?df-count) (str ?df-word :> ?key) (str ?df-count :> ?df-count-str))) The <- macro defines a query, but does not execute it. The initial vector, [?key ?df-count-str], defines the output fields, which is followed by a list of predicate functions. Each predicate can be one of the following three types: Generators: A source of data where the underlying source is either a tap or another query. Operations: Implicit relations that take in input variables defined elsewhere and either act as a function that binds new variables or a filter. Operations typically act within the scope of a single tuple. Aggregators: Functions that act across tuples to create aggregate representations of data. For example, count and sum. The :> keyword is used to separate input variables from output variables. If no :> keyword is specified, the variables are considered as input variables for operations and output variables for generators and aggregators. The (src ?doc-id ?time ?df-word) predicate function names the first three values within the input tuple, whose names are applicable within the query scope. Therefore, if the tuple ("doc1" 123324 "This") arrives in this query, the variables would effectively bind as follows: ?doc-id: "doc1" ?time: 123324 ?df-word: "This" Each predicate within the scope of the query can use any bound value or add new bound variables to the scope of the query. The final set of bound values that are emitted is defined by the output vector. We defined three queries, each calculating a portion of the value required for the TF-IDF algorithm. These are fed from two single taps, which are files stored in the Hadoop filesystem. The document file is stored using Apache Avro, which provides a high-performance and dynamic serialization layer. Avro takes a record definition and enables serialization/deserialization based on it. The record structure, in this case, is for a document and is defined as follows: {"namespace": "storm.cookbook", "type": "record", "name": "Document", "fields": [ {"name": "docid", "type": "string"}, {"name": "time", "type": "long"}, {"name": "line", "type": "string"} ] } Both the stop words and documents are fed through an ETL function that emits a clean set of words that have been filtered. The words are derived by splitting the line field using a regular expression: (defmapcatop split [line] (s/split line #"[[](),.)s]+")) The ETL function is also a query, which serves as a source for our downstream queries, and defines the [?doc-id ?time ?word] output fields. The output tap, or sink, is based on the Cassandra scheme. A query defines predicate logic, not the source and destination of data. The sink ensures that the outputs of our queries are sent to Cassandra. The ?- macro executes a query, and it is only at execution time that a query is bound to its source and destination, again allowing for extreme levels of composition. The following, therefore, executes the TF query and outputs to Cassandra: (?- (create-tf-tap cassandra-ip) (TF src)) There's more… The Avro test data was created using the test data from the Cascading tutorial at http://www.cascading.org/2012/07/31/cascading-for-the-impatient-part-5/. Within this tutorial is the rain.txt tab-separated data file. A new column was created called time that holds the Unix epoc time in milliseconds. The updated text file was then processed using some basic Java code that leverages Avro: Schema schema = Schema.parse(SandboxMain.class.getResourceAsStream("/document.avsc")); File file = new File("document.avro"); DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<GenericRecord>(schema); DataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<GenericRecord>(datumWriter); dataFileWriter.create(schema, file); BufferedReader reader = new BufferedReader(new InputStreamReader(SandboxMain.class.getResourceAsStream("/rain.txt"))); String line = null; try { while ((line = reader.readLine()) != null) { String[] tokens = line.split("t"); GenericRecord docEntry = new GenericData.Record(schema); docEntry.put("docid", tokens[0]); docEntry.put("time", Long.parseLong(tokens[1])); docEntry.put("line", tokens[2]); dataFileWriter.append(docEntry); } } catch (IOException e) { e.printStackTrace(); } dataFileWriter.close(); Persisting documents from Storm In the previous recipe, we looked at deriving precomputed views of our data taking some immutable data as the source. In that recipe, we used statically created data. In an operational system, we need Storm to store the immutable data into Hadoop so that it can be used in any preprocessing that is required. How to do it… As each tuple is processed in Storm, we must generate an Avro record based on the document record definition and append it to the data file within the Hadoop filesystem. We must create a Trident function that takes each document tuple and stores the associated Avro record. Within the tfidf-topology project created in, inside the storm.cookbook.tfidf.function package, create a new class named PersistDocumentFunction that extends BaseFunction. Within the prepare function, initialize the Avro schema and document writer: public void prepare(Map conf, TridentOperationContext context) { try { String path = (String) conf.get("DOCUMENT_PATH"); schema = Schema.parse(PersistDocumentFunction.class .getResourceAsStream("/document.avsc")); File file = new File(path); DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<GenericRecord>(schema); dataFileWriter = new DataFileWriter<GenericRecord>(datumWriter); if(file.exists()) dataFileWriter.appendTo(file); else dataFileWriter.create(schema, file); } catch (IOException e) { throw new RuntimeException(e); } } As each tuple is received, coerce it into an Avro record and add it to the file: public void execute(TridentTuple tuple, TridentCollector collector) { GenericRecord docEntry = new GenericData.Record(schema); docEntry.put("docid", tuple.getStringByField("documentId")); docEntry.put("time", Time.currentTimeMillis()); docEntry.put("line", tuple.getStringByField("document")); try { dataFileWriter.append(docEntry); dataFileWriter.flush(); } catch (IOException e) { LOG.error("Error writing to document record: " + e); throw new RuntimeException(e); } } Next, edit the TermTopology.build topology and add the function to the document stream: documentStream.each(new Fields("documentId","document"), new PersistDocumentFunction(), new Fields()); Finally, include the document path into the topology configuration: conf.put("DOCUMENT_PATH", "document.avro"); How it works… There are various logical streams within the topology, and certainly the input for the topology is not in the appropriate state for the recipes in this article containing only URLs. We therefore need to select the correct stream from which to consume tuples, coerce these into Avro records, and serialize them into a file. The previous recipe will then periodically consume this file. Within the context of the topology definition, include the following code: Stream documentStream = getUrlStream(topology, spout) .each(new Fields("url"), new DocumentFetchFunction(mimeTypes), new Fields("document", "documentId", "source")); documentStream.each(new Fields("documentId","document"), new PersistDocumentFunction(), new Fields()); The function should consume tuples from the document stream whose tuples are populated with already fetched documents.

In this article by Jacob Perkins, author of Python 3 Text Processing with NLTK 3 Cookbook, we will learn how to transform text into feature dictionaries, and how to train a text classifier for sentiment analysis. (For more resources related to this topic, see here.) Bag of words feature extraction Text feature extraction is the process of transforming what is essentially a list of words into a feature set that is usable by a classifier. The NLTK classifiers expect dict style feature sets, so we must therefore transform our text into a dict. The bag of words model is the simplest method; it constructs a word presence feature set from all the words of an instance. This method doesn't care about the order of the words, or how many times a word occurs, all that matters is whether the word is present in a list of words. How to do it... The idea is to convert a list of words into a dict, where each word becomes a key with the value True. The bag_of_words() function in featx.py looks like this: def bag_of_words(words): return dict([(word, True) for word in words]) We can use it with a list of words; in this case, the tokenized sentence the quick brown fox: >>> from featx import bag_of_words >>> bag_of_words(['the', 'quick', 'brown', 'fox']) {'quick': True, 'brown': True, 'the': True, 'fox': True} The resulting dict is known as a bag of words because the words are not in order, and it doesn't matter where in the list of words they occurred, or how many times they occurred. All that matters is that the word is found at least once. You can use different values than True, but it is important to keep in mind that the NLTK classifiers learn from the unique combination of (key, value). That means that ('fox', 1) is treated as a different feature than ('fox', 2). How it works... The bag_of_words() function is a very simple list comprehension that constructs a dict from the given words, where every word gets the value True. Since we have to assign a value to each word in order to create a dict, True is a logical choice for the value to indicate word presence. If we knew the universe of all possible words, we could assign the value False to all the words that are not in the given list of words. But most of the time, we don't know all the possible words beforehand. Plus, the dict that would result from assigning False to every possible word would be very large (assuming all words in the English language are possible). So instead, to keep feature extraction simple and use less memory, we stick to assigning the value True to all words that occur at least once. We don't assign the value False to any word since we don't know what the set of possible words are; we only know about the words we are given. There's more... In the default bag of words model, all words are treated equally. But that's not always a good idea. As we already know, some words are so common that they are practically meaningless. If you have a set of words that you want to exclude, you can use the bag_of_words_not_in_set() function in featx.py: def bag_of_words_not_in_set(words, badwords): return bag_of_words(set(words) - set(badwords)) This function can be used, among other things, to filter stopwords. Here's an example where we filter the word the from the quick brown fox: >>> from featx import bag_of_words_not_in_set >>> bag_of_words_not_in_set(['the', 'quick', 'brown', 'fox'], ['the']) {'quick': True, 'brown': True, 'fox': True} As expected, the resulting dict has quick, brown, and fox, but not the. Filtering stopwords Stopwords are words that are often useless in NLP, in that they don't convey much meaning, such as the word the. Here's an example of using the bag_of_words_not_in_set() function to filter all English stopwords: from nltk.corpus import stopwords def bag_of_non_stopwords(words, stopfile='english'): badwords = stopwords.words(stopfile) return bag_of_words_not_in_set(words, badwords) You can pass a different language filename as the stopfile keyword argument if you are using a language other than English. Using this function produces the same result as the previous example: >>> from featx import bag_of_non_stopwords >>> bag_of_non_stopwords(['the', 'quick', 'brown', 'fox']) {'quick': True, 'brown': True, 'fox': True} Here, the is a stopword, so it is not present in the returned dict. Including significant bigrams In addition to single words, it often helps to include significant bigrams. As significant bigrams are less common than most individual words, including them in the bag of words model can help the classifier make better decisions. We can use the BigramCollocationFinder class to find significant bigrams. The bag_of_bigrams_words() function found in featx.py will return a dict of all words along with the 200 most significant bigrams: from nltk.collocations import BigramCollocationFinder from nltk.metrics import BigramAssocMeasures def bag_of_bigrams_words(words, score_fn=BigramAssocMeasures.chi_sq, n=200): bigram_finder = BigramCollocationFinder.from_words(words) bigrams = bigram_finder.nbest(score_fn, n) return bag_of_words(words + bigrams) The bigrams will be present in the returned dict as (word1, word2) and will have the value as True. Using the same example words as we did earlier, we get all words plus every bigram: >>> from featx import bag_of_bigrams_words >>> bag_of_bigrams_words(['the', 'quick', 'brown', 'fox']) {'brown': True, ('brown', 'fox'): True, ('the', 'quick'): True, 'fox': True, ('quick', 'brown'): True, 'quick': True, 'the': True} You can change the maximum number of bigrams found by altering the keyword argument n. See also In the next recipe, we will train a NaiveBayesClassifier class using feature sets created with the bag of words model. Training a Naive Bayes classifier Now that we can extract features from text, we can train a classifier. The easiest classifier to get started with is the NaiveBayesClassifier class. It uses the Bayes theorem to predict the probability that a given feature set belongs to a particular label. The formula is: P(label | features) = P(label) * P(features | label) / P(features) The following list describes the various parameters from the previous formula: P(label): This is the prior probability of the label occurring, which is the likelihood that a random feature set will have the label. This is based on the number of training instances with the label compared to the total number of training instances. For example, if 60/100 training instances have the label, the prior probability of the label is 60%. P(features | label): This is the prior probability of a given feature set being classified as that label. This is based on which features have occurred with each label in the training data. P(features): This is the prior probability of a given feature set occurring. This is the likelihood of a random feature set being the same as the given feature set, and is based on the observed feature sets in the training data. For example, if the given feature set occurs twice in 100 training instances, the prior probability is 2%. P(label | features): This tells us the probability that the given features should have that label. If this value is high, then we can be reasonably confident that the label is correct for the given features. Getting ready We are going to be using the movie_reviews corpus for our initial classification examples. This corpus contains two categories of text: pos and neg. These categories are exclusive, which makes a classifier trained on them a binary classifier. Binary classifiers have only two classification labels, and will always choose one or the other. Each file in the movie_reviews corpus is composed of either positive or negative movie reviews. We will be using each file as a single instance for both training and testing the classifier. Because of the nature of the text and its categories, the classification we will be doing is a form of sentiment analysis. If the classifier returns pos, then the text expresses a positive sentiment, whereas if we get neg, then the text expresses a negative sentiment. How to do it... For training, we need to first create a list of labeled feature sets. This list should be of the form [(featureset, label)], where the featureset variable is a dict and label is the known class label for the featureset. The label_feats_from_corpus() function in featx.py takes a corpus, such as movie_reviews, and a feature_detector function, which defaults to bag_of_words. It then constructs and returns a mapping of the form {label: [featureset]}. We can use this mapping to create a list of labeled training instances and testing instances. The reason to do it this way is to get a fair sample from each label. It is important to get a fair sample, because parts of the corpus may be (unintentionally) biased towards one label or the other. Getting a fair sample should eliminate this possible bias: import collections def label_feats_from_corpus(corp, feature_detector=bag_of_words): label_feats = collections.defaultdict(list) for label in corp.categories(): for fileid in corp.fileids(categories=[label]): feats = feature_detector(corp.words(fileids=[fileid])) label_feats[label].append(feats) return label_feats Once we can get a mapping of label | feature sets, we want to construct a list of labeled training instances and testing instances. The split_label_feats() function in featx.py takes a mapping returned from label_feats_from_corpus() and splits each list of feature sets into labeled training and testing instances: def split_label_feats(lfeats, split=0.75): train_feats = [] test_feats = [] for label, feats in lfeats.items(): cutoff = int(len(feats) * split) train_feats.extend([(feat, label) for feat in feats[:cutoff]]) test_feats.extend([(feat, label) for feat in feats[cutoff:]]) return train_feats, test_feats Using these functions with the movie_reviews corpus gives us the lists of labeled feature sets we need to train and test a classifier: >>> from nltk.corpus import movie_reviews >>> from featx import label_feats_from_corpus, split_label_feats >>> movie_reviews.categories() ['neg', 'pos'] >>> lfeats = label_feats_from_corpus(movie_reviews) >>> lfeats.keys() dict_keys(['neg', 'pos']) >>> train_feats, test_feats = split_label_feats(lfeats, split=0.75) >>> len(train_feats) 1500 >>> len(test_feats) 500 So there are 1000 pos files, 1000 neg files, and we end up with 1500 labeled training instances and 500 labeled testing instances, each composed of equal parts of pos and neg. If we were using a different dataset, where the classes were not balanced, our training and testing data would have the same imbalance. Now we can train a NaiveBayesClassifier class using its train() class method: >>> from nltk.classify import NaiveBayesClassifier >>> nb_classifier = NaiveBayesClassifier.train(train_feats) >>> nb_classifier.labels() ['neg', 'pos'] Let's test the classifier on a couple of made up reviews. The classify() method takes a single argument, which should be a feature set. We can use the same bag_of_words() feature detector on a list of words to get our feature set: >>> from featx import bag_of_words >>> negfeat = bag_of_words(['the', 'plot', 'was', 'ludicrous']) >>> nb_classifier.classify(negfeat) 'neg' >>> posfeat = bag_of_words(['kate', 'winslet', 'is', 'accessible']) >>> nb_classifier.classify(posfeat) 'pos' How it works... The label_feats_from_corpus() function assumes that the corpus is categorized, and that a single file represents a single instance for feature extraction. It iterates over each category label, and extracts features from each file in that category using the feature_detector() function, which defaults to bag_of_words(). It returns a dict whose keys are the category labels, and the values are lists of instances for that category. If we had label_feats_from_corpus() return a list of labeled feature sets instead of a dict, it would be much harder to get balanced training data. The list would be ordered by label, and if you took a slice of it, you would almost certainly be getting far more of one label than another. By returning a dict, you can take slices from the feature sets of each label, in the same proportion that exists in the data. Now we need to split the labeled feature sets into training and testing instances using split_label_feats(). This function allows us to take a fair sample of labeled feature sets from each label, using the split keyword argument to determine the size of the sample. The split argument defaults to 0.75, which means the first 75% of the labeled feature sets for each label will be used for training, and the remaining 25% will be used for testing. Once we have gotten our training and testing feats split up, we train a classifier using the NaiveBayesClassifier.train() method. This class method builds two probability distributions for calculating prior probabilities. These are passed into the NaiveBayesClassifier constructor. The label_probdist constructor contains the prior probability for each label, or P(label). The feature_probdist constructor contains P(feature name = feature value | label). In our case, it will store P(word=True | label). Both are calculated based on the frequency of occurrence of each label and each feature name and value in the training data. The NaiveBayesClassifier class inherits from ClassifierI, which requires subclasses to provide a labels() method, and at least one of the classify() or prob_classify() methods. The following diagram shows other methods, which will be covered shortly: There's more... We can test the accuracy of the classifier using nltk.classify.util.accuracy() and the test_feats variable created previously: >>> from nltk.classify.util import accuracy >>> accuracy(nb_classifier, test_feats) 0.728 This tells us that the classifier correctly guessed the label of nearly 73% of the test feature sets. The code in this article is run with the PYTHONHASHSEED=0 environment variable so that accuracy calculations are consistent. If you run the code with a different value for PYTHONHASHSEED, or without setting this environment variable, your accuracy values may differ. Classification probability While the classify() method returns only a single label, you can use the prob_classify() method to get the classification probability of each label. This can be useful if you want to use probability thresholds for classification: >>> probs = nb_classifier.prob_classify(test_feats[0][0]) >>> probs.samples() dict_keys(['neg', 'pos']) >>> probs.max() 'pos' >>> probs.prob('pos') 0.9999999646430913 >>> probs.prob('neg') 3.535688969240647e-08 In this case, the classifier says that the first test instance is nearly 100% likely to be pos. Other instances may have more mixed probabilities. For example, if the classifier says an instance is 60% pos and 40% neg, that means the classifier is 60% sure the instance is pos, but there is a 40% chance that it is neg. It can be useful to know this for situations where you only want to use strongly classified instances, with a threshold of 80% or greater. Most informative features The NaiveBayesClassifier class has two methods that are quite useful for learning about your data. Both methods take a keyword argument n to control how many results to show. The most_informative_features() method returns a list of the form [(feature name, feature value)] ordered by most informative to least informative. In our case, the feature value will always be True: >>> nb_classifier.most_informative_features(n=5)[('magnificent', True), ('outstanding', True), ('insulting', True),('vulnerable', True), ('ludicrous', True)] The show_most_informative_features() method will print out the results from most_informative_features() and will also include the probability of a feature pair belonging to each label: >>> nb_classifier.show_most_informative_features(n=5) Most Informative Features magnificent = True pos : neg = 15.0 : 1.0 outstanding = True pos : neg = 13.6 : 1.0 insulting = True neg : pos = 13.0 : 1.0 vulnerable = True pos : neg = 12.3 : 1.0 ludicrous = True neg : pos = 11.8 : 1.0 The informativeness, or information gain, of each feature pair is based on the prior probability of the feature pair occurring for each label. More informative features are those that occur primarily in one label and not on the other. The less informative features are those that occur frequently with both labels. Another way to state this is that the entropy of the classifier decreases more when using a more informative feature. See https://en.wikipedia.org/wiki/Information_gain_in_decision_trees for more on information gain and entropy (while it specifically mentions decision trees, the same concepts are applicable to all classifiers). Training estimator During training, the NaiveBayesClassifier class constructs probability distributions for each feature using an estimator parameter, which defaults to nltk.probability.ELEProbDist. The estimator is used to calculate the probability of a label parameter given a specific feature. In ELEProbDist, ELE stands for Expected Likelihood Estimate, and the formula for calculating the label probabilities for a given feature is (c+0.5)/(N+B/2). Here, c is the count of times a single feature occurs, N is the total number of feature outcomes observed, and B is the number of bins or unique features in the feature set. In cases where the feature values are all True, N == B. In other cases, where the number of times a feature occurs is recorded, then N >= B. You can use any estimator parameter you want, and there are quite a few to choose from. The only constraints are that it must inherit from nltk.probability.ProbDistI and its constructor must take a bins keyword argument. Here's an example using the LaplaceProdDist class, which uses the formula (c+1)/(N+B): >>> from nltk.probability import LaplaceProbDist >>> nb_classifier = NaiveBayesClassifier.train(train_feats, estimator=LaplaceProbDist) >>> accuracy(nb_classifier, test_feats) 0.716 As you can see, accuracy is slightly lower, so choose your estimator parameter carefully. You cannot use nltk.probability.MLEProbDist as the estimator, or any ProbDistI subclass that does not take the bins keyword argument. Training will fail with TypeError: __init__() got an unexpected keyword argument 'bins'. Manual training You don't have to use the train() class method to construct a NaiveBayesClassifier. You can instead create the label_probdist and feature_probdist variables manually. The label_probdist variable should be an instance of ProbDistI, and should contain the prior probabilities for each label. The feature_probdist variable should be a dict whose keys are tuples of the form (label, feature name) and whose values are instances of ProbDistI that have the probabilities for each feature value. In our case, each ProbDistI should have only one value, True=1. Here's a very simple example using a manually constructed DictionaryProbDist class: >>> from nltk.probability import DictionaryProbDist >>> label_probdist = DictionaryProbDist({'pos': 0.5, 'neg': 0.5}) >>> true_probdist = DictionaryProbDist({True: 1}) >>> feature_probdist = {('pos', 'yes'): true_probdist, ('neg', 'no'): true_probdist} >>> classifier = NaiveBayesClassifier(label_probdist, feature_probdist) >>> classifier.classify({'yes': True}) 'pos' >>> classifier.classify({'no': True}) 'neg' See also In the next recipe, we will train the DecisionTreeClassifier classifier. Training a decision tree classifier The DecisionTreeClassifier class works by creating a tree structure, where each node corresponds to a feature name and the branches correspond to the feature values. Tracing down the branches, you get to the leaves of the tree, which are the classification labels. How to do it... Using the same train_feats and test_feats variables we created from the movie_reviews corpus in the previous recipe, we can call the DecisionTreeClassifier.train() class method to get a trained classifier. We pass binary=True because all of our features are binary: either the word is present or it's not. For other classification use cases where you have multivalued features, you will want to stick to the default binary=False. In this context, binary refers to feature values, and is not to be confused with a binary classifier. Our word features are binary because the value is either True or the word is not present. If our features could take more than two values, we would have to use binary=False. A binary classifier, on the other hand, is a classifier that only chooses between two labels. In our case, we are training a binary DecisionTreeClassifier on binary features. But it's also possible to have a binary classifier with non-binary features, or a non-binary classifier with binary features. The following is the code for training and evaluating the accuracy of a DecisionTreeClassifier class: >>> dt_classifier = DecisionTreeClassifier.train(train_feats,binary=True, entropy_cutoff=0.8, depth_cutoff=5, support_cutoff=30)>>> accuracy(dt_classifier, test_feats)0.688 The DecisionTreeClassifier class can take much longer to train than the NaiveBayesClassifier class. For that reason, I have overridden the default parameters so it trains faster. These parameters will be explained later. How it works... The DecisionTreeClassifier class, like the NaiveBayesClassifier class, is also an instance of ClassifierI, as shown in the following diagram: During training, the DecisionTreeClassifier class creates a tree where the child nodes are also instances of DecisionTreeClassifier. The leaf nodes contain only a single label, while the intermediate child nodes contain decision mappings for each feature. These decisions map each feature value to another DecisionTreeClassifier, which itself may contain decisions for another feature, or it may be a final leaf node with a classification label. The train() class method builds this tree from the ground up, starting with the leaf nodes. It then refines itself to minimize the number of decisions needed to get to a label by putting the most informative features at the top. To classify, the DecisionTreeClassifier class looks at the given feature set and traces down the tree, using known feature names and values to make decisions. Because we are creating a binary tree, each DecisionTreeClassifier instance also has a default decision tree, which it uses when a known feature is not present in the feature set being classified. This is a common occurrence in text-based feature sets, and indicates that a known word was not in the text being classified. This also contributes information towards a classification decision. There's more... The parameters passed into DecisionTreeClassifier.train() can be tweaked to improve accuracy or decrease training time. Generally, if you want to improve accuracy, you must accept a longer training time and if you want to decrease the training time, the accuracy will most likely decrease as well. But be careful not to optimize for accuracy too much. A really high accuracy may indicate overfitting, which means the classifier will be excellent at classifying the training data, but not so good on data it has never seen. See https://en.wikipedia.org/wiki/Over_fitting for more on this concept. Controlling uncertainty with entropy_cutoff Entropy is the uncertainty of the outcome. As entropy approaches 1.0, uncertainty increases. Conversely, as entropy approaches 0.0, uncertainty decreases. In other words, when you have similar probabilities, the entropy will be high as each probability has a similar likelihood (or uncertainty of occurrence). But the more the probabilities differ, the lower the entropy will be. The entropy_cutoff value is used during the tree refinement process. The tree refinement process is how the decision tree decides to create new branches. If the entropy of the probability distribution of label choices in the tree is greater than the entropy_cutoff value, then the tree is refined further by creating more branches. But if the entropy is lower than the entropy_cutoff value, then tree refinement is halted. Entropy is calculated by giving nltk.probability.entropy() a MLEProbDist value created from a FreqDist of label counts. Here's an example showing the entropy of various FreqDist values. The value of 'pos' is kept at 30, while the value of 'neg' is manipulated to show that when 'neg' is close to 'pos', entropy increases, but when it is closer to 1, entropy decreases: >>> from nltk.probability import FreqDist, MLEProbDist, entropy >>> fd = FreqDist({'pos': 30, 'neg': 10}) >>> entropy(MLEProbDist(fd)) 0.8112781244591328 >>> fd['neg'] = 25 >>> entropy(MLEProbDist(fd)) 0.9940302114769565 >>> fd['neg'] = 30 >>> entropy(MLEProbDist(fd)) 1.0 >>> fd['neg'] = 1 >>> entropy(MLEProbDist(fd)) 0.20559250818508304 What this all means is that if the label occurrence is very skewed one way or the other, the tree doesn't need to be refined because entropy/uncertainty is low. But when the entropy is greater than entropy_cutoff, then the tree must be refined with further decisions to reduce the uncertainty. Higher values of entropy_cutoff will decrease both accuracy and training time. Controlling tree depth with depth_cutoff The depth_cutoff value is also used during refinement to control the depth of the tree. The final decision tree will never be deeper than the depth_cutoff value. The default value is 100, which means that classification may require up to 100 decisions before reaching a leaf node. Decreasing the depth_cutoff value will decrease the training time and most likely decrease the accuracy as well. Controlling decisions with support_cutoff The support_cutoff value controls how many labeled feature sets are required to refine the tree. As the DecisionTreeClassifier class refines itself, labeled feature sets are eliminated once they no longer provide value to the training process. When the number of labeled feature sets is less than or equal to support_cutoff, refinement stops, at least for that section of the tree. Another way to look at it is that support_cutoff specifies the minimum number of instances that are required to make a decision about a feature. If support_cutoff is 20, and you have less than 20 labeled feature sets with a given feature, then you don't have enough instances to make a good decision, and refinement around that feature must come to a stop. See also The previous recipe covered the creation of training and test feature sets from the movie_reviews corpus. Summary In this article, we learned how to transform text into feature dictionaries, and how to train a text classifier for sentiment analysis. Resources for Article: Further resources on this subject: Python Libraries for Geospatial Development [article] Python Testing: Installing the Robot Framework [article] Ten IPython essentials [article]

Building checklists Anyone who preformed Dynamics AX application installation or upgrade has to be familiar with standard checklists. Normally, a checklist is a list of menu items displayed in logical sequence. Each item represents either mandatory or optional actions to be executed by the user in order to complete the whole procedure. In custom Dynamics AX implementations, checklists can be used as a convenient way to configure non standard settings. Checklists can also be implemented as a part of third-party modules for their initial setup. In this recipe, we will create a checklist for user-friendly ledger budget setup. The checklist will consists of two mandatory and one optional item. How to do it... Open AOT, and create a new class called SysCheckListInterfaceBudget: interface SysCheckListInterfaceBudgetextends SysCheckListInterface{} Create three more classes—one for each checklist item, with the following code: class SysCheckListItem_BudgetModelextends SysCheckListItemimplements SysCheckListInterfaceBudget{}public str getCheckListGroup(){ return "Setup";}public str getHelpLink(){ #define.TopicId('AxShared.chm::/html/' + '84030522-0057-412C-BFC7-DBEB4D40E5A1.htm') ; return SysCheckListItem::sharedGuide(#TopicId);}public MenuItemName getMenuItemName(){ return menuitemdisplaystr(BudgetModel);}public MenuItemType getMenuItemType(){ return MenuItemType::Display;}str label(){ return "Models";}class SysCheckListItem_BudgetRevisionextends SysCheckListItemimplements SysCheckListInterfaceBudget{}public void new(){; super(); this.placeAfter(classnum(SysCheckListItem_BudgetModel)); this.indeterminate(true);}public str getCheckListGroup(){ return "Setup";}public str getHelpLink(){ #define.TopicId('AxShared.chm::/html/' + 'AACC4353-C3EB-4982-BB7F-2B36D97FF25B.htm') ; return SysCheckListItem::sharedGuide(#TopicId);}public MenuItemName getMenuItemName(){ return menuitemdisplaystr(BudgetRevision);}public MenuItemType getMenuItemType(){ return MenuItemType::Display;}str label(){ return "Revisions";}class SysCheckListItem_Budgetextends SysCheckListItemimplements SysCheckListInterfaceBudget{}public void new(){; super(); this.addDependency( classnum(SysCheckListItem_BudgetModel)); this.placeAfter( classnum(SysCheckListItem_BudgetRevision));}public str getCheckListGroup(){ return "Create budgets";}public str getHelpLink(){ #define.TopicId('AxShared.chm::/html/' + '6A596E1E-6803-4410-B4E4-EDE4EF44AF6D.htm') ; return SysCheckListItem::sharedGuide(#TopicId);}public MenuItemName getMenuItemName(){ return menuitemdisplaystr(LedgerBudget);}public MenuItemType getMenuItemType(){ return MenuItemType::Display;}str label(){ return "Budgets";} Create another class for the checklist itself: class SysCheckList_Budget extends SysCheckList{ container log;}protected str getCheckListCaption(){ return "Budget checklist";}protected str getHtmlHeader(){ return "Budget checklist";}protected classId getInterfaceId(){ return classnum(SysCheckListInterfaceBudget);}public void save( identifiername _name, ClassDescription _description){; if (!confind(log, _name)) { log = conins(log, conlen(log)+1, _name); }}public boolean find( identifiername _name, ClassDescription _description){ return confind(log, _name) ? true : false;}static void main(Args _args){; SysCheckList::runCheckListSpecific( classnum(SysCheckList_Budget), true);} Open the SysCheckList class in AOT, and replace its checkListItemsHook() and checkListsHook() with the following code: protected static container checkListsHook(){ return [classnum(SysCheckList_Budget)];}protected static container checkListItemsHook(){ return [classnum(SysCheckListItem_Budget), classnum(SysCheckListItem_BudgetRevision), classnum(SysCheckListItem_BudgetModel)];} Open the BudgetModel form in AOT, and override its close() with the following code: public void close(){; super(); SysCheckList::finished( classnum(SysCheckListItem_BudgetModel));} Open the BudgetRevision form in AOT, and override its close() with the following code: public void close(){; super(); SysCheckList::finished( classnum(SysCheckListItem_BudgetRevision));} Open the LedgerBudget form in AOT, and override its close() with the following code: public void close(){; super(); SysCheckList::finished(classnum(SysCheckListItem_Budget));} Create a new Display menu item SysCheckList_Budget with the following properties: Property Value Name SysCheckList_Budget Label Budget checklist ObjectType Class Object SysCheckList_Budget To test the checklist, run the SysCheckList_Budget menu item from AOT. The following should appear on the right-hand side of the Dynamics AX window: Click on the listed items to start and complete relevant actions. Notice how the status icons change upon completion of each task. How it works... The main principle behind checklists is that we have to create a main class, which represents the checklist itself and a number of SysCheckListItem item classes, which act as list items. The relation between the main class and the items is made by the use of an interface, that is, each list item implements it, and the main class holds the reference to it. In this example, we create an interface SysCheckListInterfaceBudget and specify it in the getInterfaceId() of the main checklist class SysCheckList_Budget. Next, we implement the interface in three SysCheckListItem classes, which correspond to Models, Revisions, and Budgets items in the checklist. Each SysCheckListItem class contains a set of inherited methods, which allows us to define a number of different parameters for individual items: All initialization code can be added to the new() methods. In this example, we use placeAfter() to determine the position of the item in the list relative to other items, indeterminate() to make item optional and addDependency() to make an item inactive until another specified item is completed. getCheckListGroup() defines item dependency to a specific group. The Budget checklist has two groups, Setup and Create budgets. getHelpLink() is responsible for placing the relevant help link in the form of a question mark next to the item. getMenuItemName() and getMenuItemType() contain a name and a type of menu item, which is executed upon user request. Here, we have Budget model, Budget revisions, and Ledger budget forms respectively in each class. And finally custom labels can be set in label(). Once the items are ready, we create the main checklist class SysCheckList_Budget, which extends the standard SysCheckList. We override some of the methods to add custom functionality to the checklist: getCheckListCaption() sets the title of the checklist. getHtmlHeader() could be used to add some descriptive text. As mentioned before, getInterfaceId() is the place where we specify the name of the checklist item interface. The methods save() and find() are used to store and retrieve respectively the status of each item in the list. In this example, we store statuses in the local variable log to make sure that statuses are reset every time we run the checklist. The static method main() runs the class. Here, we use runCheckListSpecific() of the system SysCheckList class to start the checklist. The display menu item we created is pointing to the checklist class and may be used to add the checklist to a user menu. When building checklists, it is necessary to add them and their items to the global checklist and checklist item list. The SysCheckList class contains two methods—checkLists() and checkListItems()—where all system checklists and their items are registered. The same class provides two more methods—checkListsHook() and checkListItemsHook()—where custom checklists should be added. As a part of this example, we also add our budget checklist and its items to the SysCheckList. Final modifications have to be done in all checklist forms. We call the finished() of the SysCheckList class in the close() of each form to update the status of the corresponding checklist item. In other words, it means that item status will be set as completed when the user closes the form. This code does not affect the normal use of the form when it is opened from the regular menu. Of course, more logic could be added here if the completion of a specific item is not that straightforward. Also notice that the system automatically adds a link called Information, which describes the checklist statuses: There's more... The checklist in this example stores item statuses per each run. This means that every time you close the checklist, its statuses are lost and are set to their initial states upon checklist start. By replacing save() and find() in SysCheckList_Budget with the following code, we can store statuses permanently in the SysSetupLog table: public boolean find( identifiername _name, ClassDescription _description){ return SysSetupLog::find(_name, _description).RecId != 0;}public void save( identifiername _name, ClassDescription _description){; SysSetupLog::save(_name, _description);} In this case, every time the checklist starts, the system will pick up its last status from the SysSetupLog table and allow the user to continue the checklist. Adding a "Go to the Main Table Form" link Go to the Main Table Form is a feature of Dynamics AX, which allows users to jump to the main record just by right-clicking on the field and selecting the Go to the Main Table Form option. It is based on table relations and is available for those controls whose data fields have foreign key relationships with other tables. Because of the data structure integrity, this feature works most of the time. However, when it comes to complex table relations, it does not work correctly or does not work at all. Another example of when this feature does not work automatically is when the form control is not bound to a table field. In such situations, Go to the Main Table Form has to be implemented manually. In this recipe, to demonstrate how it works, we will modify the Business relations form in the CRM module to make sure that the Employee filter at the top of the form allows users to use the Go to the Main Table Form feature from the context menu. How to do it... Open the smmBusRelTable form in AOT, and override jumpRef() of the EmployeeFilter control with: public void jumpRef(){ EmplTable emplTable; Args args; MenuFunction menuFunction; ; emplTable = EmplTable::find(this.text()); if (!emplTable) { return; } args = new Args(); args.caller(element); args.record(emplTable); menuFunction = new MenuFunction( menuitemdisplaystr(EmplTable), MenuItemType::Display); menuFunction.run(args);} To test the result, open CRM | Business Relation Details, make sure an employee number is specified in the Employee filter, and right-click on the filter control. Notice that the Go to the Main Table Form option, which will open the Employee form, is now available: How it works... Normally, the Go to the Main Table Form feature is controlled by the relations between tables. If there are no relations or the form control is not bound to a table field, then this option is not available. But, we can force this option to appear by overriding the control's jumpRef() method. In this method, we have to add code that opens the relevant form. This can be done by creating, initializing, and running a FormRun object, but the easier way is to simply run the relevant menu item. In this recipe, the code in jumpRef() does exactly that. First, we check if the value in the control is a valid employee number. If yes, then we run the Display menu item EmplTable with an Args object containing the proper employee record. The rest is done automatically by the system, that is, the Employee form is opened with the employee information.

All truths are easy to understand once they are discovered; the point is to discover them.-Galileo Galilei What is UDDI? Universal Description and Discovery Information (UDDI) is a type of registry whose primary purpose is to represent information about web services. It describes the service providers, the services that provider offers, and in some cases, the specific technical specifications for interacting with those services. While UDDI was originally envisioned as a public, platform independent registry that companies could exploit for listing and consuming services, it seems that many have chosen instead to use UDDI as an internal resource for categorizing and describing their available enterprise services. Besides simply listing available services for others to search and peruse, UDDI is arguably most beneficial for those who wish to perform runtime binding to service endpoints. Instead of hard-coding a service path in a client application, one may query UDDI for a particular service's endpoint and apply it to their active service call. While UDDI is typically used for web services, nothing prevents someone from storing information about any particular transport and allowing service consumers to discover and do runtime resolution to these endpoints. As an example, this is useful if you have an environment with primary, backup, and disaster access points and want your application be able to gracefully look up and failover to the next available service environment. In addition, UDDI can be of assistance if an application is deployed globally but you wish for regional consumers to look up and resolve against the closest geographical endpoint. UDDI has a few core hierarchy concepts that you must grasp to fully comprehend how the registry is organized. The most important ones are included here. Name Purpose Name in Microsoft UDDI services BusinessEntity These are the service providers. May be an organization, business unit or functional area. Provider BusinessService General reference to a business service offered by a provider. May be a logical grouping of actual services. Service BindingTemplate Technical details of an individual service including endpoint Binding tModel (Technical Model) Represents metadata for categorization or description such as transport or protocol tModel As far as relationships between these entities go, a Business Entity may contain many Business Services, which in turn can have multiple Binding Templates. A binding may reference multiple tModels and tModels may be reused across many Binding Templates. What's new in UDDI version three? The latest UDDI specification calls out multiple-registry environments, support for digital signatures applied to UDDI entries, more complex categorization, wildcard searching, and a subscription API. We'll spend a bit of time on that last one in a few moments. Let's take a brief lap around at the Microsoft UDDI Services offering. For practical purposes, consider the UDDI Services to be made up of two parts: an Administration Console and a web site. The website is actually broken up into both a public facing and administrative interface, but we'll talk about them as one unit. The UDDI Configuration Console is the place to set service-wide settings ranging from the extent of logging to permissions and site security. The site node (named UDDI) has settings for permission account groups, security settings (see below), and subscription notification thresholds among others. The web node, which resides immediately beneath the parent, controls web site setting such as logging level and target database. Finally, the notification node manages settings related to the new subscription notification feature and identically matches the categories of the web node. The UDDI Services web site, found at http://localhost/uddi/, is the destination or physically listing, managing, and configuring services. The Search page enables querying by a wide variety of criteria including category, services, service providers, bindings, and tModels. The Publish page is where you go to add new services to the registry or edit the settings of existing ones. Finally, the Subscription page is where the new UDDI version three capability of registry notification is configured. We will demonstrate this feature later in this article. How to add services to the UDDI registry? Now we're ready to add new services to our UDDI registry. First, let's go to the Publish page and define our Service Provider and a pair of categorical tModels. To add a new Provider, we right-click the Provider node in the tree and choose Add Provider. Once a provider is created and named, we have the choice of adding all types of context characteristics such as a contact name(s), categories, relationships, and more. I'd like to add two tModel categories to my environment : one to identify which type of environment the service references (development, test, staging, production) and another to flag which type of transport it uses (Basic HTTP, WS HTTP, and so on). To add atModel, simply right-click the tModels node and choose Add tModel. This first one is named biztalksoa:runtimeresolution:environment. After adding one more tModel for biztalksoa:runtimeresolution:transporttype, we're ready to add a service to the registry. Right-click the BizTalkSOA provider and choose Add Service. Set the name of this service toBatchMasterService. Next, we want to add a binding (or access point) for this service, which describes where the service endpoint is physically located. Switch to the Bindings tab of the service definition and choose New Binding. We need a new access point, so I pointed to our proxy service created earlier and identified it as an endPoint. Finally, let's associate the two new tModel categories with our service. Switch to the Categories tab, and choose to Add Custom Category. We're asked to search for atModel, which represents our category, so a wildcard entry such as %biztalksoa%  is a valid search criterion. After selecting the environment category, we're asked for the key name and value. The key "name" is purely a human-friendly representation of the data whereas the tModel identifier and the key value comprise the actual name-value pair. I've entered production as the value on the environment category, and WS-Http as the key value on thetransporttype category. At this point, we have a service sufficiently configured in the UDDI directory so that others can discover and dynamically resolve against it.

Making a complete yet small Linux distribution There's an endless list of actively maintained Linux distributions on (Distrowatch). With modern Open Source applications demanding more and more resources, the most popular Linux distros are also the most resource hungry. Zenwalk (http://www.zenwalk.org/) and Vector Linux (http://vectorlinux.com/) are two Linux distros that promise the ease-of-use of a modern desktop Linux distro but keep the resource utilization in check. So how do they do it? In this discussion with Jean-Philippe Guillemin of Zenwalk and Robert Lange of Vector Linux, I try to understand what it takes to make a distro that's equally capable of running on the first generation of Pentiums as well as the latest. Mayank Sharma: Any advantages of choosing Slackware ( http://www.slackware.com/) as base? [Both distros are based on Slackware Linux.] Robert Lange: I think slackware gives you the ideal base to work from. The most note-worthy advantages are simplicity of design, lack of dependency hell and easy configuration.Jean-Philippe Guillemin: There are some advantages, and also some disadvantages. Of course it depends on the goals of a project, so what I'm going to tell you applies to Zenwalk but may not apply to other projects. Let's begin with goals, then we'll talk about advantages, and finally about disadvantages. The main goal of the Zenwalk project is to build a "rational GNU/Linux operating system". That means that the system is designed to be very simple and responsive, with one application for a given task. The main advantage of Slackware as a starting point to build a GNU/Linux OS with this kind of objective is that Slackware is a pure Linux/Unix system, without bloated subsystems, very simple and flexible. It is one of the last systems of this kind. You can guess that such a "pure" system is ideal when you want to build your own vision of desktop, hardware management and administration tools. The main disadvantage of Slackware as a base system is implied by the main advantage. It is the amount of work needed to transform it into a desktop OS. With most other "major" Linux systems , 80% of the work is already done. It seems simpler to me to build a desktop derivative from one of these big Linux distributions because the desktop system is already nearly finished, but it would give less freedom.   MS: Talking of Slackware, do you contribute upstream? Any changes you'd like to see up there? RL: The short answer is no. Slackware really doesn't appreciate so called forks in its design philosophy so even if offered I highly doubt that there would be any level of acceptance. However, we make the changes that we apply to the system available in our FTP repository. JP: Not really. I sometimes contact Patrick Volkerding [Slackware maintainer] when I need information about something I don't understand in Slackware or to get some news about him. I also ask for his opinion when I am about to decide for a structural change in Zenwalk. I want Zenwalk to remain Slackware compatible, although the fork is really huge now. The only contribution to Slackware was Zenwalk's Xorg 7.X system that Pat plans to use as a base to build his own version. At the moment, Pat is working to update the base toolchain (glibc, gcc, etc). When he has finished this on Slackware's side, we will have to rebuild nearly everything in Zenwalk on this new environment. Apart from this awaited modernization of the toolchain, I wouldn't change anything in Slackware as that's not really important for Zenwalk. We are independent now and the only important condition is to maintain application compatibility between the two projects. MS: Any new distribution that you think would make a good base, if you were to start afresh today? Any particular reasons? RL: There is at least to my knowledge nothing new in what I call the absolute base in a linux distro. 99% percent are based on Debian, Redhat/Fedora, Gentoo and Slackware. The exception being Linux from Scratch (http://www.linuxfromscratch.org/). I don't think I would make any different choice today then I made almost nine years ago. JP: Easy question. To start a new project I would not use anything else than Slackware. At least Linux From Scratch (http://www.packtpub.com/article/Linux_From_Scratch) would be an option, or maybe Zenwalk itself ;). MS: What's the lowest hardware configuration that people have managed to run Vector on? What would you suggest as the minimum config for a usable system? RL: It depends on what version of Vector you are talking about. Our very early versions would run on an i486 machine with 32 MB RAM although X, the Graphical User Interface (GUI) would be pretty slow. Todays 5.x versions have run on Pentium 166's with 128 megs of RAM. To have a pleasant experience we suggest at least a Pentium 2 with 128 megs of RAM. The X GUI is the system hog. If you want to run a console based system a Pentium 100 with 24 MB of RAM would probably work. JP: Zenwalk can be run on a Pentium 2 processor, assuming that you have enough RAM to support X window applications. 256MB is enough to drive such an old box with acceptible usability. The minimum recommended configuration for true responsiveness is a Pentium III system with 512MB RAM.  MS: What other software are essential? I mean components like Window Managers, multimedia players, office applications, etc. Which are the ones that you've included and how have they helped you with your purpose? RL: We try to give our users the basics from most computing categories. The Window Manager we liked the most up till recently was IceWM in combination with the roxfiler system, which allowed for a very light weight X system with much of the same functionality that say Windows 98 users were used to. We have recently jumped on the Xfce 4 bandwagon as its the perfect desktop for lower spec computers but still offers many of the configuration options that only GNOME and KDE users were able to get. The browser market has always been a problem for old computers. Mozilla, Opera and Firefox are RAM eaters, but we offer dillo as an alternate and it does a reasonable job. Office chores are handled by Abiword, which is a great program for low spec machines. Xmms has been our mutimedia player along with Mplayer and GMplayer. JP: The Window Manager and file browser are essential and mandatory parts of the desktop system. We use a customized version of XFCE along with the Thunar file manager. We provide one mainstream application for each task with the Zenwalk desktop. Quoting my friend Claus Futtrup, our Zenwalk columnist, as late as December 2005 the most critical applications for a desktop user were : No 1 : Email client, rated critical by 62% of users, No 2 : Productivity, rated critical by 51%, and No 3 : Web browser, rated critical by 50%. Apart from the applications, one very important aspect is "Desktop integration". All applications should use the same widgets, same color theme, same icons, etc. This is important because icons and windows layout must be consistent to keep the usability as smooth as possible, especially widgets layout and icons. We achieve this by providing exclusively GTK+2 (http://en.wikipedia.org/wiki/GTK+) based applications. MS: I'd appreciate if you could list or briefly explain some of the key issues of assembling a distro that's capable of running on slower hardware. Do you strip kernels, or use the stock ones? RL: The key is to keep things as simple as possible. The less background processes you have running the more memory is required for executing programs. We use a very vanilla kernel with as much modularized as possible to keep its resident size to a minimum. Compile code optimized for i586 that seems the best for compatibility and speed on a wide range of hardware. Pay attention to the init process since you want as much control as possible over what does and does not get loaded and pass this on to the user. JP: Let me first clarify that Zenwalk is not designed for use on old hardware (for example there is no ISA (Industry Standard Architecture) bus support in Zenwalk). Zenwalk is designed to provide the optimal responsiveness that you can expect from a GNU/Linux operating system. The entire system is tuned towards this target. The kernel is configured in a very uncommon way as would be in an embedded system with preemptive processus and IO schedulers, installed libraries are kept to a minimum. We also use only one graphical toolkit for this particular reason, GTK+2. We avoid using bloated subsystems and prefer simple low level systems like udev or inotify for devices polling - we developed our own alternative automount system. Using Xfce, which is a feature-rich yet light desktop environment with its Thunar file manager also helps to improve the responsiveness. There are many other enhancements, but I can't sum three years of research here in a few lines :) MS: How long has it been since you've been rolling out the distros? Why do you think people still use them, despite hardware becoming cheaper? RL: Its been eight or nine years now. I think its the simplicity and speed that keeps them coming back for more. We have also developed a substantial online user community that is second to none in being a friendly place that new users can join the Linux ranks without fear. JP: The project started three years ago. Our user base really began to grow significantly one year ago. At that time the project was mature enough to compete with major GNU/Linux distributions. I really believe that our user base is growing because people find something unique in Zenwalk. For example, in as little as 20 minutes utilizing a very easy setup, the users obtain a true modern working environment ready to use with only the best application for each need. Thus, Zenwalk is usually considered as one of the fastest GNU/Linux systems available. People can run it with blazingly good performance on modern, and also semi-modern computers.

Library CherryPy comes with a set of modules covering common tasks when building a web application such as session management, static resource service, encoding handling, or basic caching. The Autoreload Feature CherryPy is a long-running Python process, meaning that if we modify a Python module of the application, it will not be propagated in the existing process. Since stopping and restarting the server manually can be a tedious task, the CherryPy team has included an autoreload module that restarts the process as soon as it detects a modification to a Python module imported by the application. This feature is handled via configuration settings. If you need the autoreload module to be enabled while in production you will set it up as below. Note the engine.autoreload_frequency option that sets the number of seconds the autoreloader engine has to wait before checking for new changes. It defaults to one second if not present. [global]server.environment = "production"engine.autoreload_on = Trueengine.autoreload_frequency = 5 Autoreload is not properly a module but we mention it here as it is a common feature offered by the library. The Caching Module Caching is an important side of any web application as it reduces the load and stress of the different servers in action—HTTP, application, and database servers. In spite of being highly correlated to the application itself, generic caching tools such as the ones provided by this module can help in achieving decent improvements in your application's performance. The CherryPy caching module works at the HTTP server level in the sense that it will cache the generated output to be sent to the user agent and will retrieve a cached resource based on a predefined key, which defaults to the complete URL leading to that resource. The cache is held in the server memory and is therefore lost when stopping it. Note that you can also pass your own caching class to handle the underlying process differently while keeping the same high-level interface. The Coverage Module When building an application it is often beneficial to understand the path taken by the application based on the input it processes. This helps to determine potential bottlenecks and also see if the application runs as expected. The coverage module provided by CherryPy does this and provides a friendly browseable output showing the lines of code executed during the run. The module is one of the few that rely on a third-party package to run. The Encoding/Decoding Module Publishing over the Web means dealing with the multitude of existing character encoding. To one extreme you may only publish your own content using US-ASCII without asking for readers' feedback and to the other extreme you may release an application such as bulletin board that will handle any kind of charset. To help in this task CherryPy provides an encoding/decoding module that filters the input and output content based on server or user-agent settings. The HTTP Module This module offers a set of classes and functions to handle HTTP headers and entities. For example, to parse the HTTP request line and query string: s = 'GET /note/1 HTTP/1.1' # no query stringr = http.parse_request_line(s) # r is now ('GET', '/note/1', '','HTTP/1.1')s = 'GET /note?id=1 HTTP/1.1' # query string is id=1r = http.parse_request_line(s) # r is now ('GET', '/note', 'id=1','HTTP/1.1')http.parseQueryString(r[2]) # returns {'id': '1'}Provide a clean interface to HTTP headers:For example, say you have the following Accept header value:accept_value = "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5"values = http.header_elements('accept', accept_value)print values[0].value, values[0].qvalue # will print text/html 1.0 The Httpauth Module This module provides an implementation of the basic and digest authentication algorithm as defined in RFC 2617. The Profiler Module This module features an interface to conduct a performance check of the application. The Sessions Module The Web is built on top of a stateless protocol, HTTP, which means that requests are independent of each other. In spite of that, a user can navigate an e-commerce website with the impression that the application more or less follows the way he or she would call the store to pass an order. The session mechanism was therefore brought to the Web to allow servers to keep track of users' information. CherryPy's session module offers a straightforward interface to the application developer to store, retrieve, amend, and delete chunks of data from a session object. CherryPy comes natively with three different back-end storages for session objects: Back-end type Advantages Drawbacks RAM Efficient Accepts any type of objects No configuration needed Information lost when server is shutdown Memory consumption can grow fast File system Persistence of the information Simple setup File system locking can be inefficient Only serializable (via the pickle module) objects can be stored Relational database (PostgreSQL built-in support) Persistence of the information Robust Scalable Can be load balanced Only serializable objects can be stored Setup less straightforward

Fast Short-Running BPEL Let's begin with a discussion on compiled BPEL. Uses of Short-Running Processes Having developed an approach to keep SOA processes running for an arbitrarily long time, we now turn our attention to short-running processes and ask: howcan we make them run as fast as possible? The two most common uses of a short-running process are: To implement a synchronous web service operation. The process begins with an input message, runs through a quick burst of logic to process it, sends back the output message, and completes. The client application blocks for the duration, as diagram (a) in the next figure shows. If the process moves too slowly, the client will complain about the response time. To perform complex routing for the ESB. As David Chapelle discusses in his book Enterprise Service Bus (O'Reilly, 2004), a good ESB can natively perform basic content-based- and itinerary-based-routing, but it needs orchestration processes to handle more complex routing patterns. In diagram (b) in the figure, when the ESB receives a message, it passes it to an orchestration process that proceeds to perform in eight steps a series of transformation and invocation maneuvers that could never be achieved with the basic branching capabilities of the ESB. Again, speed is critical. The ESB prefers to get rid of messages as soon as it gets them. When it delegates work to an orchestration process, it expects that process to move quickly and lightly. Architecture for Short-Running Processes In considering a design to optimize the performance of these two cases, we assume that our stack has both an ESB and a process integration layer. All messages in and out of the stack go through the ESB. The ESB, when it receives an inbound message, routes it to the process integration engine for processing. The process integration engine, in turn, routes all outbound messages through the ESB. Further, we assume that the ESB uses message queues to converse with the process integration layer. Client applications, on the other hand, typically use web services to converse with the ESB. The following figure shows how we might enhance this architecture for faster short-running processes. (The implementation we consider is a Java-based BPEL process engine.) When a client application or partner process calls through the ESB, the ESB routes the event, based on the event's type, either to the general process integration engine or to an engine optimized for short-running processes. To route to the general engine, the ESB places the message on the Normal PI In Queue. That engine is drawn as a cloud; we are not concerned in this discussion with its inner workings. To route to the optimized engine, the ESB either queues the message on SR In Queue or, to reduce latency, directly calls the short-running engine's main class, ProcessManager. (Direct calls are suitable for the orchestration routing case described in the previous figure; there, processes run as an extension of the ESB, so it makes sense for the ESB to invoke them straightaway.) A set of execution threads pulls messages from SR In Queue and invokes ProcessManager to inject these inbound events to the processes themselves. The role of ProcessManager is to keep the state of, and to execute, short-running processes. Each process is represented in compiled form as a Java class (for example, ProcessA or ProcessB) that inherits from a base class called CompiledProcess. Compiled classes are generated by a tool called BPELCompiler, which creates Java code that represents the flow of control specified in the BPEL XML representation of the process. ProcessManager runs processes by creating and calling the methods of instances of CompiledProcess-derived classes. It also uses TimeManager to manage timed events. Processes, whether running on the general engine or on the optimized engine, send messages to partners by placing messages on the outbound queue Out Queue, which the ESB picks up and routes to the relevant partner. A general process engine is built to handle processes of all durations, long and short alike, and, with a mandate this extensive, does not handle the special case of time-critical short-running processes very effectively. There are three optimizations we require, and we build these into the short-running engine: Process state is held in memory. Process state is never persisted, even for processes with intermediate events. Completed process instances are cleaned out of memory immediately, so as to reduce the memory required. Processes are compiled, not interpreted. That is, the process definition is coded in Java class form, rather than as an XML document. Compilation speeds the execution time of a burst. The process may define timed events of a very short duration, to the order of milliseconds. Furthermore, the engine generates a fault when the process exceeds its SLA. The process may catch the fault or let it bubble up to the calling application. The architecture we sketched in this section, as we discover presently, is designed to meet these requirements. Example of a Very Fast Process The next figure shows a short-running process with multiple bursts that benefits from these optimizations. When the process starts, it initializes its variables (InitVars) and asynchronously invokes a partner process called the Producer (Call Producer Asynx). It then enters into a loop (FetchLoop) that, on each iteration, waits for one of the two events from the Producer: result or noMore. If it gets the result event, it, in parallel, invokes two handler services (Call Handler A and Call Handler B), and loops back. If it gets the noMore event, the process sets the loop's continuation flag to false (Set Loop Stop). The loop exits, and the process completes. While it waits for the producer events, the process also sets a timed event (too long) that fires if neither event arrives in sufficient time. If the timer expires, the process sends an exception message to the producer (Send Exception Msg Producer Async), and loops back. The timing characteristics are shown in parentheses. The producer, on average, sends a result or noMore event in 80 milliseconds. The handlers that the process invokes to handle a result event average 50 milliseconds and 70 milliseconds, but because they run in parallel, their elapsed time is the greater of these two times, or 70 milliseconds. Thus, an iteration of the loop with a result event averages roughly 150 milliseconds. Iteration with a noMore event averages just 80 milliseconds, because the activity Set Loop Stop runs nearly instantaneously. The cycle time of an instance with one result iteration and one noMore iteration is just 220 milliseconds. The too long timed event has a duration of 200 milliseconds, which in itself is rather a small interval, but is a huge chunk of time compared to the normal cycle time. The cycle time of an instance whose three intermediate events are result, too long, and noMore is 420 milliseconds on average. Times this fast cannot be achieved on a general-purpose engine. Running the Very Fast Process on the Optimized Engine The sequence diagram in the following figure illustrates how this process runs on the short-running engine: The process starts when client application sends a message intended to trigger the process' start event. The ProcessManager receives this event (either as a direct call or indirectly via an execution thread that monitors the short-running inbound queue) in its routeMessageEvent() method. It then checks with the process class—shown as Process in the figure, a subclass of the CompiledProcess class we discuss presently—whether it supports the given start event type (hasStartEvent()), and if so, injects the event into the process (onStartEvent()). The process, as part of its logic, performs the activities InitVars and CallProducerAsync and enters the first iteration of the while loop, in which it records in its data structures that it is now waiting for three pending events (Set Pending Events). Because one of these events is a timed event, it also registers that event with the TimeManager (addEvent()).The first burst is complete. In the second burst, the producer process responds with a result event (result: routeMessageEvent()). The ProcessManager checks whether the process instance is waiting for that event (hasPendingEvent()) and injects it (onIntermediateEvent()). The process invokes the two handlers (that is, it invokes CallHandler on HandlerA and HandlerB), completing the first iteration of the loop. It now loops back, resets the pending events (Set Pending Events), and registers a new timed event (addEvent()). The second burst is complete. Assuming the producer does not respond in sufficient time, the timer expires, and the TimeManager which checks for expired events on its own thread notifies the Process Manager (routeTimedEvent()). ProcessManager gives the event to the process (calling hasPendingEvent() to confirm that the process is waiting for it and onIntermediateEvent() to inject it), and the process in turn performs the SendExceptionMsg activity, completing the second iteration of the loop. The next iteration starts, and the process resets its pending events. The third burst is complete, and we leave it there.

1.    Packt: Firstly, I would like to thank you for taking part in this Q&A and we expect great things from the Sencha community for 2011. How important do you think the various communities of Open Source projects are to the progression and growth of its software? James Pearce: We have a fantastic community – something over 1 million developers worldwide – and we’re proud to work with them. Our libraries are specifically designed to be extensible in an OO-style way (with community-authored controls, widgets and so on), and we use GPL and MIT licensing throughout (although we do also provide dual commercial and OEM licenses on certain products). And importantly, our JavaScript libraries are often used in conjunction with other web software – content management systems, design systems, mobile frameworks and so on – so we are very comfortable in working with other communities that need to interact with ours.    2.    Packt: For those that do not know, how long has Sencha been around? How did the creation of Sencha come about? James Pearce: Sencha was previously known as Ext JS, a company formed several years ago with a fork of Yahoo’s YUI JavaScript library, and providing product support and professional services for developers creating rich internet applications. In 2010, the company earned VC funding from Sequoia Capital and Radar Partners, and was renamed to Sencha. At the same time the company launched Sencha Touch – the world’s first HTML5, CSS3, and JavaScript framework for building rich web applications for modern mobile devices. These are now our flagship framework products, and are complemented with tools, services and a range of exciting, world-class open-source lab projects 3.    Packt: How would you describe the Sencha community? James Pearce: Very vibrant, and very loyal. Our forums are always a busy place, with tips, advice, problem solving and – yes – bug reports :) Our annual SenchaCon event is a sell-out, and we’re looking at more regular events to cater for the worldwide demand. Groups all over the world run local meetups, and our SenchaDevs developer directory grew incredibly quickly beyond our wildest expectations. Generally our community is very professional and polite. We have a strong following in the enterprise space, where our frameworks are very popular – so perhaps we’re a little different from a lot of regular Open Source communities in this respect. 4.    Packt: What would you describe as the current strengths and weaknesses of the Sencha community? James Pearce: One challenge we face is the huge bell-curve in skill sets. Sencha Touch and Ext JS can be very powerful and experienced developers do astonishing things with them. But at the other end of the spectrum, we want to make it easy for new entrants to the web application world. One of our challenges is working out how to smooth this learning curve and use the existing skills of the ‘advanced’ community to bring newcomers up to speed. But it’s happening around the world, with meetups, training and workshops and so on, so I’m very optimistic. 5.    Packt: How many code contributions does the Sencha project receive on a monthly basis? James Pearce: It’s hard to generalize. We use GitHub for all our projects, and for several projects, we have thousands of followers and hundreds of forks. It’s also a tricky question because we’re avidly hiring at the moment. Several core community members are in the midst of joining the team as employees – and of course their contributions are going through the roof! 6.    Packt: How has Sencha encouraged its community to evangelize the software? James Pearce: I'm always amazed by how fervent our developers are without any prompting. I think it might be something to do with the satisfaction of being able to build incredible apps with web technology – once you’ve seen what can actually be done with HTML5, CSS3 and JavaScript, you want to tell as many people as possible about it. We do also provide assistance and shwag for local community groups who want to get meetups off the ground and run their own training workshops, but frankly our community doesn’t need much encouragement! 7.    Packt: 2010 was a big year for the Sencha community, how does the project expect to improve in 2011 and build on the growth experienced? James Pearce: Good question. In 2010 we had to work with the community to understand how and why we were evolving our business and embracing mobile as the next big frontier for the web. In 2011, the reality that the mobile web is a great long-term future is starting to be well accepted, and we need to make sure we help our users (Sencha Touch in particular) work with that opportunity. The Ext JS project is about to reach a significant milestone too: v4, and we’ll need to work sympathetically with our users and community members to go through the upgrade and re-education process that that will bring. 8.    Packt: as you may know, we’ve recently announced that we’ve hit the landmark of over $300k donated to Open Source projects, how important are donations of this kind to the Sencha community? James Pearce: We’re in a fortunate position: since we provide commercial services (such as training and support) for our software, we are able to maintain the project and support the community ourselves. For us, it is our community’s time and enthusiasm that are the most valuable donation. Nevertheless, your support is invaluable for many types of project and you’re doing a great job helping keep the Open Source ecosystem healthy. 9.    Packt: We specialize in refining and distilling advice, provided by the community around Open Source projects, into easy to follow specialist information. How important is the sharing of information for the Sencha community? How do you plan to improve on your role as the hub of distributing information for the Sencha community in 2011? James Pearce: Yes, this is a big area for us. We want to make our projects as approachable as possible to newcomers to web technology in general – and you can never have too many getting-started guides and introductory stuff. That said, there is so much going on in the mobile and HTML5 space right now, that a lot of our work is just about educating users about what’s possible in contemporary web environments. We live in exciting times!10.    Packt: Thanks for your time James, lastly what projects, if any, are you/Sencha working on at the moment? James Pearce: Ah… you really need to check out some of the cool stuff in our Sencha Labs area! My favorite right now is the PhiloGL project that makes it easy for JavaScript developers to hook into WebGL and hardcore graphics capabilities – there are some cool demos. But some of the vector-graphics stuff is fantastic too (e.g. Raphael and InfoVis) – together with CSS3, these sorts of things demonstrate possibilities that previously required proprietary browser plugins. The web is a changing place and we hope we can be a helpful part of its evolution. To read more about Sencha, go to www.sencha.com

So what exactly is a service? A service is essentially a well-defined interface to an autonomous chunk of functionality, which usually corresponds to a specific business process. That might sound a lot like a regular old object-oriented component to you. While both services and components have commonality in that they expose discrete interfaces of functionality, a service is more focused on the capabilities offered than the packaging. Services are meant to be higher-level, business-oriented offerings that provide technology abstraction and interoperability within a multipurpose "services" tier of your architecture. What makes up a service? Typically you'll find: Contract: Explains what operations the service exposes, types of messages, and exchange patterns supported by this service, and any policies that explain how this service is used. Messages: The data payload exchanged between the service consumer and provider. Implementation: The portion of the service which actually processes the requests, executes the expected business functionality, and optionally returns a response. Service provider: The host of the service which publishes the interface and manages the lifetime of the service. Service consumer: Ideally, a service has someone using it. The service consumer is aware of the available service operations and knows how to discover the provider and determine what type of messages to transmit. Facade: Optionally, a targeted facade may be offered to particularly service consumers. This sort of interface may offer a more simplified perspective on the service, or provide a coarse-grained avenue for service invocation. What is the point of building a service? I'd say it's to construct an asset capable of being reused which means that it's a discrete, discoverable, self-describing entity that can be accessed regardless of platform or technology. Service-oriented architecture is defined as an architectural discipline based on loosely-coupled, autonomous chunks of business functionality which can be used to construct composite applications. Through the rest of this article we get a chance to flesh out many of the concepts that underlie that statement. Let's go ahead and take a look at a few of the principles and characteristics that I consider most important to a successful service-oriented BizTalk solution. As part of each one, I'll explain the thinking behind the principle and then call out how it can be applied to BizTalk Server solutions. Loosely coupled Many of the fundamental SOA principles actually stem from this particular one. In virtually all cases, some form of coupling between components is inevitable. The only way we can effectively build software is to have interrelations between the various components that make up the delivered product. However, when architecting solutions, we have distinct design decisions to make regarding the extent to which application components are coupled. Loose coupling is all about establishing relationships with minimal dependencies. What would a tightly-coupled application look like? In such an application, we'd find components that maintained intimate knowledge of each others' working parts and engaged in frequent, chatty synchronous calls amongst themselves. Many components in the application would retain state and allow consumers to manipulate that state data. Transactions that take place in a tightly coupled application probably adhere to a two-phase commit strategy where all components must succeed together in order for each data interaction to be finalized. The complete solution has its ensemble of components compiled together and singularly deployed to one technology platform. In order to run properly, these tightly-coupled components rely on the full availability of each component to fulfill the requests made of them. On the other hand, a loosely-coupled application employs a wildly different set of characteristics. Components in this sort of application share only a contract and keep their implementation details hidden. Rarely preserving state data, these components rely on less frequent communication where chunky input containing all the data the component needs to satisfy its requestors is shared. Any transactions in these types of applications often follow a compensation strategy where we don't assume that all components can or will commit their changes at the same time. This class of solution can be incrementally deployed to a mix of host technologies. Asynchronous communication between components, often through a broker, enables a less stringent operational dependency between the components that comprise the solution. What makes a solution loosely coupled then? Notably, the primary information shared by a component is its interface. The consuming component possesses no knowledge of the internal implementation details. The contract relationship suffices as a means of explaining how the target component is used. Another trait of loosely coupled solutions is coarse-grained interfaces that encourage the transmission of full data entities as opposed to fine-grained interfaces, which accept small subsets of data. Because loosely-coupled components do not share state information, a thicker input message containing a complete impression of the entity is best. Loosely-coupled applications also welcome the addition of a broker which proxies the (often asynchronous) communication between components. This mediator permits a rich decoupling where runtime binding between components can be dynamic and components can forgo an operational dependency on each other. Let's take a look at an example of loose coupling that sits utterly outside the realm of technology. Completely non-technical loose coupling exampleWhen I go to a restaurant and place an order with my waiter, he captures the request on his pad and sends that request to the kitchen. The order pad (the contract) contains all the data needed by the kitchen chef to create my meal. The restaurant owner can bring in a new waiter or rotate his chefs and the restaurant shouldn't skip a beat as both roles (services) serve distinct functions where the written order is the intersection point and highlight of their relationship. Why does loose coupling matter? By designing a loosely-coupled solution, you provide a level of protection against the changes that the application will inevitably require over its life span. We have to reduce the impact of such changes while making it possible to deploy necessary updates in an efficient manner. How does this apply to BizTalk Server solutions? A good portion of the BizTalk Server architecture was built with loose coupling in mind. Think about the BizTalk MessageBox which acts as a broker facilitating communication between ports and orchestrations while limiting any tight coupling. Receive ports and send ports are very loosely coupled and in many cases, have absolutely no awareness of each other. The publish-and-subscribe bus thrives on the asynchronous transfer of self-describing messages between stateless endpoints. Let's look at a few recommendations of how to build loosely-coupled BizTalk applications. Orchestrations are a prime place where you can either go with a tightly-coupled or loosely-coupled design route. For instance, when sketching out your orchestration process, it's sure tempting to use that Transform shape to convert from one message type to another. However, a version change to that map will require a modification of the calling orchestration. When mapping to or from data structures associated with external systems, it's wiser to push those maps to the edges (receive/send ports) and not embed a direct link to the map within the orchestration. BizTalk easily generates schemas for line-of-business (LOB) systems and consumed services. To interact with these schemas in a very loosely coupled fashion, consider defining stable entity schemas (i.e. "canonical schemas") that are used within an orchestration, and only map to the format of the LOB system in the send port. For example, if you need to send a piece of data into an Oracle database table, you can certainly include a map within an orchestration which instantiates the Oracle message. However, this will create a tight coupling between the orchestration and the database structure. To better insulate against future changes to the database schema, consider using a generic intermediate data format in the orchestration and only transforming to the Oracle-specific format in the send port. How about those logical ports that we add to orchestrations to facilitate the transfer of messages in and out of the workflow process? When configuring those ports, the Port Configuration Wizard asks you if you want to associate the port to a physical endpoint via the Specify Now option. Once again, pretty tempting. If you know that the message will arrive at an orchestration via a FILE adapter, why not just go ahead and configure that now and let Visual Studio.NET create the corresponding physical ports during deployment? While you can independently control the auto-generated physical ports later on, it's a bad idea to embed transport details inside the orchestration file. On each subsequent deployment from Visual Studio.NET, the generated receive port will have any out-of-band changes overwritten by the deployment action. Chaining orchestration together is a tricky endeavor and one that can leave you in a messy state if you are too quick with a design decision. By "chaining orchestrations", I mean exploiting multiple orchestrations to implement a business process. There are a few options at your disposal listed here and ordered from most coupled to least coupled. Call Orchestration or Start Orchestration shape: An orchestration uses these shapes in order to kick off an additional workflow process. The Call Orchestration is used for synchronous connection with the new orchestration while the Start Orchestration is a fire-and-forget action. This is a useful tactic for sharing state data (for example variables, messages, ports) from the source orchestration to the target. However, both options require a tight coupling of the source orchestration to the target. Version changes to the target orchestration would likely require a redeployment of the source orchestration. Partner direct bound ports: These provide you the capability to communicate between orchestrations using ports. In the forward partner direct binding scenario, the sender has a strong coupling to the receiver, while the receiver knows nothing about the sender. This works well in situations where there are numerous senders and only one receiver. Inverse partner direct binding means that there is a tight coupling between the receiver and the sender. The sender doesn't know who will receive the command, so this scenario is intended for cases where there are many receivers for a single sender. In both cases, you have tight coupling on one end, with loose-coupling on the other. MessageBox direct binding: This is the most loosely-coupled way to share data between orchestrations. When you send a message out of an orchestration through a port marked for MessageBox direct binding, you are simply placing a message onto the bus for anyone to consume. The source orchestration has no idea where the data is going, and the recipients have no idea where it's been. MessageBox direct binding provides a very loosely-coupled way to send messages between different orchestrations and endpoints. Critical pointWhile MessageBox direct binding is great, you do lose the ability to send the additional state data that a Call Orchestration shape will provide you. So, as with all architectural decisions, you need to decide if the sacrifice (loose coupling, higher latency) is worth the additional capabilities. Decisions can be made during BizTalk messaging configuration that promote a loosely-coupled BizTalk landscape. For example, both receive ports and send ports allow for the application of maps to messages flying past. In each case, multiple maps can be added. This does NOT mean that all the maps will be applied to the message, but rather, it allows for sending multiple different message types in, and emitting a single type (or even multiple types) out the other side. By applying transformation at the earliest and latest moments of bus processing, you loosely couple external formats and systems from internal canonical formats. We should simply assume that all upstream and downstream systems will change over time, and configure our application accordingly. Another means for loosely coupling BizTalk solutions involves the exploitation of the publish-subscribe architecture that makes up the BizTalk message bus. Instead of building solely point-to-point solutions and figuring that a SOAP interface makes you service oriented, you should also consider loosely coupling the relationship between the service input and where the data actually ends up. We can craft a series of routing decision that take into account message content or context and direct the message to one or more relevant processes/endpoints. While point-to-point solutions may be appropriate for many cases, don't neglect a more distributed pattern where the data publisher does not need to explicitly know exactly how their data will be processed and routed by the message bus. When identifying subscriptions for our send ports, we should avoid tight coupling to metadata attributes that might limit the reuse of the port. For instance, you should try to create subscriptions on either the message type or message content instead of context attributes such as the inbound receive port name. Ports should be tightly coupled to the MessageBox and messages it stores, not to attributes of its publisher. That said, there are clearly cases where a subscriber is specifically looking for data that corresponds to a targeted piece of metadata such as the subject line of the email received by BizTalk. As always, design your solution in a way that solves your business problem in an efficient manner.

(For more resources related to this topic, see here.) Getting ready Content in Edublogs can take many different forms—posts, pages, uploaded media, and embedded media. The first step needs to be developing an understanding of what each of these types of content are, and how they fit into the Edublogs framework. Pages: Pages are generally static content, such as an About or a Frequently Asked Questions page. Posts: Posts are the content that is continually updated on a blog. When you write an article, it is referred to as a post. Media [uploaded]: Edublogs has a media manager that allows you to upload pictures, videos, audio files, and other files that readers would be able to interact with or download. Media [embedded]: Embedded media is different than internal media in that it is not stored on your Edublogs account. If you record a video and upload it, the video resides on your website and is considered internal to that website. If you want to add a YouTube video, a Prezi presentation, a slideshow, or any content that actually resides on another website, that is considered embedding. How to do it... Posts and pages are very similar. When you click on the Pages link on the left navigation column, if you are just beginning, you will see an empty list or the Sample Page that Edublogs provides. However, this page will show a list of all of the pages that you have written, as shown in the following screenshot: Click on any column header (Title, Author, Comments, and Date) to sort the pages by that criterion. A page can be any of several types: Published (anyone can see), Drafts, Private, Password Protected, or in the Trash. You can filter by those pages as well. You will only see the types of pages that you are currently using. For example, in the following screenshot, I have 3 Draft pages. If I had none, Drafts would not show as an option. When you hover over a page, you are provided with several options, such as Edit, Quick Edit, Trash, and View. View: This option shows you the actual live post, the same way that a reader would see it. Trash: This deletes the page. Edit: This brings you back to the main editing screen, where you can change the actual body of the page. Quick Edit: This allows you to change some of the main options of the post: Title, Slug (the end of the URL to access the page), Author, if the page has a parent, and if it should be published. The following screenshot demonstrates these options: How it works... Everything above about Pages also applies to Posts. Posts, though, have several additional options. It's also more common to use the additional options to customize Posts than Pages. Right away, hovering over Posts, it shows two new links: Categories and Tags. These tools are optional, and serve the dual purpose of aiding the author by providing an organizational structure, and helping the reader to find posts more effectively. A Category is usually very general; on one of my educational blogs, I limit my categories to a few: technology integration, assessment, pedagogy, and lessons. If I happen to write a post that does not fit, I do not categorize it. Tags are becoming ubiquitous in many applications and operating systems. They provide an easy way to browse a store of information thematically. On my educational blog, I have over 160 tags. On one post about Facebook's new advertising system, I added the following tags: Digital Literacy, Facebook, Privacy. Utilizing tags can help you to see trends in your writing and makes it much easier for new readers to find posts that interest them, and regular readers to find old posts that they want to re-reference. Let's take a look at some of the advanced features. When adding or editing a post, the following features are all located on the right-hand side column: Publish: The Publish box is necessary any time you want to remove your Post (or Page) from the draft stage, and allow readers to be able to see it. Most new bloggers simply click on Publish/Update when they are done writing a Post, which works fine. It is limited though. People often find that there are certain times of day that result in higher readership. If you click on Edit next to Publish Immediately, you can choose a date and time to schedule the publication. In addition, the Visibility line also allows you to set a Post as private, password protected, or always at the top of the page (if you have a post you particularly want to highlight, for example). Format: Most of the time, changing the format is not necessary, particularly if you run a normal, text driven blog. However, different formats lend themselves to different types of content. For example, if publishing a picture as a Post, as is often done on the microblogging site Tumblr, choosing Image would format the post more effectively. Categories: Click on + Add New Category, or check any existing categories to append them to the Post. Tags: Type any tags that you want to use, separated by commas (such as writing, blogging, Edublogs). Featured Image: Uploading and choosing a feature image adds a thumbnail image, to provide a more engaging browsing experience for the viewer. All of these features are optional, but they are useful for improving the experience, both for yourself and your readers. There's more... While for most people, the heart of a blog is the actual writing that they do. Media serves help to both make the experience more memorable and engaging, as well as to illustrate a point more effectively than text would alone. Media is anything other than text that a user can interact with; primarily, it is video, audio, or pictures. As teachers know, not everyone learns ideally through a text-based medium; media is an important part of engaging readers just as it is an important part of engaging students. There are a few ways to get media into your posts. The first is through the Media Library. On a free account, space is limited to 32 MB, a relatively small account. Pro accounts get 10 GB of space. Click on Media from the navigation menu on the left; it brings up the library. This will have a list of your media, similar to that which is used for Posts and Pages. To add media, simply click on Add New and choose an image, audio file, or video from your computer. This will then be available to any post or page to use. The following screenshot shows the Media Library page: If you are already in a post, you have even more options. Click on the Add Media button above the text editor, as shown in the following screenshot: Following are some of the options you have to embed media: Insert Media: This allows you to directly upload a file or choose one from the Media Library. Create Gallery: Creating a gallery allows you to create a set of images that users can browse through. Set Featured Image: As described above, set a thumbnail image representative of the post. Insert from URL: This allows you to insert an image by pasting in the direct URL. Make sure you give attribution, if you use someone else's image. Insert Embed Code: Embed code is extremely helpful. Many sites provide embed code (often referred to as share code) to allow people to post their content on other websites. One of the most common examples is adding a YouTube video to a post. The following screenshot is from the Share menu of a YouTube video. Copying the code provided and pasting it into the Insert Embed Code field will put the YouTube video right in the post, as shown in the following screenshot. This is much more effective than just providing a link, because readers can watch the video without ever having to leave the blog. Embedding is an Edublogs Pro feature only. Utilizing media effectively can dramatically improve the experience for your readers. Summary This article on managing content provided details about managing different types of content, in the form of posts, pages, uploaded media, and embedded media. It taught us the different features such as publish, format, categories, tags and features image. Resources for Article : Further resources on this subject: Customizing WordPress Settings for SEO [Article] Getting Started with WordPress 3 [Article] Dynamic Menus in WordPress [Article]

Today, collaboration and team effort have become critical factors, both inside and outside of the workplace. More and more users want simplicity and familiarity with the tools they use day in and day out. They achieve this by searching in Google, reading in Wikipedia, writing on a blog, finding people on Facebook, being notified in a simple RSS reader, viewing friends, activities on Facebook, and bringing all of this together in iGoogle. Alfresco Share delivers all of this functionality to enterprise users, projects, and teams. Imagine a business user, if given the permission, being able to set up their project web site quickly, being able to invite users to the site, and assign permissions to users within that web site. What previously required a customized solution is now offered out of the box by Alfresco Share. Alfresco Share Alfresco Share (referred to simply as Share from now on) is built on the Alfresco enterprise-class document repository, and delivers out of the box collaborative content management. It simplifies the capture, sharing, and retrieval of information across virtual teams. Team members or project members can rapidly find relevant content or excerpts, look at past or similar projects, and stay on top of any relevant changes, in order to make them more efficient. Share is focused on collaboration tasks and includes integration with popular blogging, Wiki, and forum or discussion products, out of the box. It provides a great interface into more traditional document management libraries (think folders) as well. Keep in mind that all of the web site's contents and documents are still stored in the Alfresco repository. Therefore they are secured, versioned, searchable, and auditable. Share is an independent client application that accesses the repository through web scripts. It is built on the Alfresco Surf (referred to simply as Surf from now on) platform. Alfresco Share is a web application that runs on a machine that is separate from that of the repository. Share provides a paradigm for creating collaborative applications by aggregating Surf components, and incorporating new Surf components as they are developed. With Share, users can modify their workspaces to fit their collaborative requirements inside or outside of the organization. Users can invite their peers to share and collaborate on the project and the content. With the addition of Share, Alfresco delivers a Web 2.0 application that leverages Flash and AJAX with a polished interface, which any business person can enjoy. Features like Document Libraries, Search, Activity Feeds, Virtual Teams, personalized dashboard, N-tier Architecture, and draft CMIS support make it a really competent tool for collaborative content management. Share allows you to: Bulk-upload content, select content from thumbnails, and view it in a Flash viewer. The content is automatically generated in Flash format. This allows users to view content regardless of the original format. Search for people and experts to contribute to their projects as easily as searching for content. Share provides updates on what is new in a project, especially details of content that has been added, edited, or commented upon. Share can track deliverables and import the information into your personal calendar by using iCal. Use an interactive interface to configure a customizable dashboard, and sites, based on what is critical to a specific role or project. Share allows you to create a virtual team for projects and communities. Develop applications in an environment that uses lightweight scripting and reusable components, as well as deliver scalability and allow more users to access existing resources. The URL to access the Alfresco Share application is different from the URL used to access Alfresco Explorer. The Alfresco Share application can be launched in the web browser by visiting the URL, http://<server name> /share/ If you have already installed Alfresco in Windows, then you can invoke the Alfresco Share application by selecting the application, as shown in the following screenshot: You need to provide your authentication credentials, which are similar to those used in the Alfresco Share application. For the administrator, the default username and password are both admin. Once you have been authenticated, the Administrator Dashboard will be displayed, as shown in the following screenshot. At the top of the page you will find the application toolbar. This toolbar contains links to the various Share pages. Your Administrator Dashboard will look similar to the following screenshot: These components are as follows: Getting Started: This dashlet displays the instructions for getting started, and provides you with links to perform common tasks My Profile: This dashlet contains a summary of the personal details provided in your user profile Sites: This component displays the Site Finder page, where you can search for specific sites and manage the membership of Share sites People: This component displays the People Finder page, where you search for specific Share users Help: This component displays the online help available for Alfresco Share Logout: This component logs you out of the Alfresco Share application Search: This component enables you to perform a quick search for content in the current site, or across all of the sites

Java EE 6 Development with NetBeans 7 Develop professional enterprise Java EE applications quickly and easily with this popular IDE In addition to being an IDE, NetBeans is also a platform. Developers can use NetBeans' APIs to create both NetBeans plugins and standalone applications. For a brief history of Netbeans, see http://netbeans.org/about/history.html. Although the NetBeans IDE supports several programming languages, because of its roots as a Java only IDE it is a lot more popular with this language. As a Java IDE, NetBeans has built-in support for Java SE (Standard Edition) applications, which typically run in the user's desktop or notebook computer; Java ME (Micro Edition), which typically runs in small devices such as cell phones or PDAs; and for Java EE (Enterprise Edition) applications, which typically run on "big iron" servers and can support thousands of concurrent users.   Obtaining NetBeans NetBeans can be obtained by downloading it from http://www.netbeans.org. To download NetBeans, we need to click on the button labeled Download Free NetBeans IDE 7.0 (the exact name of the button may vary depending on the current version of NetBeans). Clicking on this button will take us to a page displaying all of NetBeans download bundles. NetBeans download includes different NetBeans bundles that provide different levels of functionality. The following table summarizes the different available NetBeans bundles and describes the functionality they provide: NetBeans bundleDescriptionJava SEAllows development of Java desktop applications.Java EEAllows development of Java Standard Edition (typically desktop applications), and Java Enterprise Edition (enterprise application running on "big iron" servers) applications.C/C++Allows development of applications written in the C or C++ languages.PHPAllows development of web applications using the popular open source PHP programming language.AllIncludes functionality of all NetBeans bundles. To follow the examples, either the Java EE or the All bundle is needed. The screenshots were taken with the Java EE bundle. NetBeans may look slightly different if the All Pack is used, particularly, some additional menu items may be seen. The following platforms are officially supported: Windows 7/Vista/XP/2000 Linux x86 Linux x64 Solaris x86 Solaris x64 Mac OS X Additionally, NetBeans can be executed in any platform containing Java 6 or newer. To download a version of NetBeans to be executed in one of these platforms, an OS independent version of NetBeans is available for download. Although the OS independent version of NetBeans can be executed in all of the supported platforms, it is recommended to obtain the platform-specific version of NetBeans for your platform. The NetBeans download page should detect the operating system being used to access it, and the appropriate platform should be selected by default. If this is not the case, or if you are downloading NetBeans with the intention of installing it in another workstation on another platform, the correct platform can be selected from the drop down labeled, appropriately enough, Platform. Once the correct platform has been selected, we need to click on the appropriate Download button for the NetBeans bundle we wish to install. For Java EE development, we need either the Java EE or the All bundle. NetBeans will then be downloaded to a directory of our choice. Java EE applications need to be deployed to an application server. Several application servers exist in the market, both the Java EE and the All NetBeans bundles come with GlassFish and Tomcat bundled. Tomcat is a popular open source servlet container, it can be used to deploy applications using the Servlets, JSP and JSF, however it does not support other Java EE technologies such as EJBs or JPA. GlassFish is a 100 percent Java EE-compliant application server. We will be using the bundled GlassFish application server to deploy and execute our examples.   Installing NetBeans NetBeans requires a Java Development Kit (JDK) version 6.0 or newer to be available before it can be installed. NetBeans installation varies slightly between the supported platforms. In the following few sections we explain how to install NetBeans on each supported platform. Microsoft Windows For Microsoft Windows platforms, NetBeans is downloaded as an executable file named something like netbeans-7.0-ml-java-windows.exe, (exact name depends on the version of NetBeans and the NetBeans bundle that was selected for download). To install NetBeans on Windows platforms, simply navigate to the folder where NetBeans was downloaded and double-click on the executable file. Mac OS X For Mac OS X, the downloaded file is called something like netbeans-7.0-ml-javamacosx.dmg (exact name depends on the NetBeans version and the NetBeans bundle that was selected for download). In order to install NetBeans, navigate to the location where the file was downloaded and double-click on it. The Mac OS X installer contains four packages, NetBeans, GlassFish, Tomcat, and OpenESB, these four packages need to be installed individually, They can be installed by simply double-clicking on each one of them. Please note that GlassFish must be installed before OpenESB. Linux and Solaris For Linux and Solaris, NetBeans is downloaded in the form of a shell script. The name of the file will be similar to netbeans-7.0-ml-java-linux.sh, netbeans-7.0-mljava-solaris-x86.sh, or netbeans-7.0-ml-java-solaris-sparc.sh, depending on the version of NetBeans, the selected platform and the selected NetBeans bundle. Before NetBeans can be installed in these platforms, the downloaded file needs to be made executable. This can be done in the command line by navigating to the directory where the NetBeans installer was downloaded and executing the following command: chmod +x ./filename.sh Substitute filename.sh with the appropriate file name for the platform and the NetBeans bundle. Once the file is executable it can be installed from the command line: ./filename.sh Again substitute filename.sh with the appropriate file name for the platform and the NetBeans bundle. Other platforms For other platforms, NetBeans can be downloaded as a platform-independent zip file. The name of the zip file will be something like netbeans-7.0-201007282301-mljava.zip (exact file name may vary, depending on the exact version of NetBeans downloaded and the NetBeans bundle that was selected). To install NetBeans on one of these platforms, simply extract the zip file to any suitable directory.  

In this article by Roberto Freato, author of Microsoft Azure Development Cookbook, we mix some of the recipes of of this book, to build a complete overview of what we need to set up a software infrastructure on the cloud. (For more resources related to this topic, see here.) Microsoft Azure is Microsoft’s Platform for Cloud Computing. It provides developers with elastic building blocks to build scalable applications. Those building blocks are services for web hosting, storage, computation, connectivity, and more, which are usable as stand-alone services or mixed together to build advanced scenarios. Building an application with Microsoft Azure could really mean choosing the appropriate services and mix them together to run our application. We start by creating a SQL Database. Creating a SQL Database server and database SQL Database is a multitenanted database system in which many distinct databases are hosted on many physical servers managed by Microsoft. SQL Database administrators have no control over the physical provisioning of a database to a particular physical server. Indeed, to maintain high availability, a primary and two secondary copies of each SQL Database are stored on separate physical servers, and users can't have any control over them. Consequently, SQL Database does not provide a way for the administrator to specify the physical layout of a database and its logs when creating a SQL Database. The administrator merely has to provide a name, maximum size, and service tier for the database. A SQL Database server is the administrative and security boundary for a collection of SQL Databases hosted in a single Azure region. All connections to a database hosted by the server go through the service endpoint provided by the SQL Database server. At the time of writing this book, an Azure subscription can create up to six SQL Database servers, each of which can host up to 150 databases (including the master database). These are soft limits that can be increased by arrangement with Microsoft Support. From a billing perspective, only the database unit is counted towards, as the server unit is just a container. However, to avoid a waste of unused resources, an empty server is automatically deleted after 90 days of non-hosting user databases. The SQL Database server is provisioned on the Azure Portal. The Region as well as the administrator login and password must be specified during the provisioning process. After the SQL Database server has been provisioned, the firewall rules used to restrict access to the databases associated with the SQL Database server can be modified on the Azure Portal, using Transact SQL or the SQL Database Service Management REST API. The result of the provisioning process is a SQL Database server identified by a fully -qualified DNS name such as SERVER_NAME.database.windows.net, where SERVER_NAME is an automatically generated (random and unique) string that differentiates this SQL Database server from any other. The provisioning process also creates the master database for the SQL Database server and adds a user and associated login for the administrator specified during the provisioning process. This user has the rights to create other databases associated with this SQL Database server as well as any logins needed to access them. Remember to distinguish between the SQL Database service and the famous SQL Server engine available on the Azure platform, but as a plain installation over VMs. In the latter case, you will continue to own the complete control of the instance that runs the SQL Server, the installation details, and the effort to maintain it during the time. Also, remember that the SQL Server virtual machines have a different pricing from the standard VMs due to their license costs. An administrator can create a SQL Database either on the Azure Portal or using the CREATE DATABASE Transact SQL statement. At the time of this writing this book, SQL Database runs in the following two different modes: Version 1.0: This refers to Web or Business Editions Version 2.0: This refers to Basic, Standard, or Premium service tiers with performance levels The first version is deprecating in few months. Web Edition was designed for small databases under 5 GB and Business Edition for databases of 10 GB and larger (up to 150 GB). There is no difference in these editions other than the maximum size and billing increment. The second version introduced service tiers (the equivalent of Editions) with an additional parameter (performance level) that sets the amount of dedicated resource to a given database. The new service tiers (Basic, Standard, and Premium) introduced a lot of advanced features such as active/passive Geo-replication, point-in-time restore, cross-region copy, and restore. Different performance levels have different limits such as the Database Throughput Unit (DTU) and the maximum DB size. An updated list of service tiers and performance levels can be found at http://msdn.microsoft.com/en-us/library/dn741336.aspx. Once a SQL Database has been created, the ALTER DATABASE Transact SQL statement can be used to alter either the edition or the maximum size of the database. The maximum size is important as the database is made read only once it reaches that size (with the The database has reached its size quota error message and number 40544). In this recipe, we'll learn how to create a SQL Database server and a database using the Azure Portal and T-SQL. Getting Ready To perform the majority of operations of the recipe, just a plain internet browser is needed. However, to connect directly to the server, we will use the SQL Server Management Studio (also available in the Express version). How to do it... First, we are going to create a SQL Database server using the Azure Portal. We will do this using the following steps: On the Azure Portal, go to the SQL DATABASES section and then select the SERVERS tab. In the bottom menu, select Add. In the CREATE SERVER window, provide an administrator login and password. Select a Subscription and Region that will host the server. To enable access from the other service in WA to the server, you can check the Allow Windows Azure Services to access the server checkbox; this is a special firewall rule that allows the 0.0.0.0 to 0.0.0.0 IP range. Confirm and wait a few seconds to complete the operation. After that, using the Azure Portal,.go to the SQL DATABASES section and then the SERVERS tab. Select the previously created server by clicking on its name. In the server page, go to the DATABASES tab. In the bottom menu, click on Add; then, after clicking on NEW SQL DATABASE, the CUSTOM CREATE window will open. Specify a name and select the Web Edition. Set the maximum database size to 5 GB and leave the COLLATION dropdown to its default. SQL Database fees are charged differently if you are using the Web/Business Edition rather than the Basic/Standard/Premium service tiers. The most updated pricing scheme for SQL Database can be found at http://azure.microsoft.com/en-us/pricing/details/sql-database/ Verify the server on which you are creating the database (it is specified correctly in the SERVER dropdown) and confirm it. Alternatively, using Transact SQL, launch Microsoft SQL Server Management Studio and open the Connect to Server window. In the Server name field, specify the fully qualified name of the newly created SQL Database server in the following form: serverName.database.windows.net. Choose the SQL Server Authentication method. Specify the administrative username and password associated earlier. Click on the Options button and specify the Encrypt connection checkbox. This setting is particularly critical while accessing a remote SQL Database. Without encryption, a malicious user could extract all the information to log in to the database himself, from the network traffic. Specifying the Encrypt connection flag, we are telling the client to connect only if a valid certificate is found on the server side. Optionally check the Remember password checkbox and connect to the server. To connect remotely to the server, a firewall rule should be created. In the Object Explorer window, locate the server you connected to, navigate to Databases | System Databases folder, and then right-click on the master database and select New Query. 18. Copy and execute this query and wait for its completion:. CREATE DATABASE DATABASE_NAME ( MAXSIZE = 1 GB ) How it works... The first part is pretty straightforward. In steps 1 and 2, we go to the SQL Database section of the Azure portal, locating the tab to manage the servers. In step 3, we fill the online popup with the administrative login details, and in step 4, we select a Region to place the SQL Database server. As a server (with its database) is located in a Region, it is not possible to automatically migrate it to another Region. After the creation of the container resource (the server), we create the SQL Database by adding a new database to the newly created server, as stated from steps 6 to 9. In step 10, we can optionally change the default collation of the database and its maximum size. In the last part, we use the SQL Server Management Studio (SSMS) (step 12) to connect to the remote SQL Database instance. We notice that even without a database, there is a default database (the master one) we can connect to. After we set up the parameters in step 13, 14, and 15, we enable the encryption requirement for the connection. Remember to always set the encryption before connecting or listing the databases of a remote endpoint, as every single operation without encryption consists of plain credentials sent over the network. In step 17, we connect to the server if it grants access to our IP. Finally, in step 18, we open a contextual query window, and in step 19, we execute the creation query, specifying a maximum size for the database. Note that the Database Edition should be specified in the CREATE DATABASE query as well. By default, the Web Edition is used. To override this, the following query can be used: CREATE DATABASE MyDB ( Edition='Basic' ) There's more… We can also use the web-based Management Portal to perform various operations against the SQL Database, such as invoking Transact SQL commands, altering tables, viewing occupancy, and monitoring the performance. We will launch the Management Portal using the following steps: Obtain the name of the SQL Database server that contains the SQL Database. Go to https://serverName.database.windows.net. In the Database fields, enter the database name (leave it empty to connect to the master database). Fill the Username and Password fields with the login information and confirm. Increasing the size of a database We can use the ALTER DATABASE command to increase the size (or the Edition, with the Edition parameter) of a SQL Database by connecting to the master database and invoking the following Transact SQL command: ALTER DATABASE DATABASE_NAME MODIFY ( MAXSIZE = 5 GB ) We must use one of the allowable database sizes. Connecting to a SQL Database with Entity Framework The Azure SQL Database is a SQL Server-like fully managed relation database engine. In many other recipes, we showed you how to connect transparently to the SQL Database, as we did in the SQL Server, as the SQL Database has the same TDS protocol as its on-premise brethren. In addition, using the raw ADO.NET could lead to some of the following issues: Hardcoded SQL: In spite of the fact that a developer should always write good code and make no errors, there is the finite possibility to make mistake while writing stringified SQL, which will not be verified at design time and might lead to runtime issues. These kind of errors lead to runtime errors, as everything that stays in the quotation marks compiles. The solution is to reduce every line of code to a command that is compile time safe. Type safety: As ADO.NET components were designed to provide a common layer of abstraction to developers who connect against several different data sources, the interfaces provided are generic for the retrieval of values from the fields of a data row. A developer could make a mistake by casting a field to the wrong data type, and they will realize it only at run time. The solution is to reduce the mapping of table fields to the correct data type at compile time. Long repetitive actions: We can always write our own wrapper to reduce the code replication in the application, but using a high-level library, such as the ORM, can take off most of the repetitive work to open a connection, read data, and so on. Entity Framework hides the complexity of the data access layer and provides developers with an intermediate abstraction layer to let them operate on a collection of objects instead of rows of tables. The power of the ORM itself is enhanced by the usage of LINQ, a library of extension methods that, in synergy with the language capabilities (anonymous types, expression trees, lambda expressions, and so on), makes the DB access easier and less error prone than in the past. This recipe is an introduction to Entity Framework, the ORM of Microsoft, in conjunction with the Azure SQL Database. Getting Ready The database used in this recipe is the Northwind sample database of Microsoft. It can be downloaded from CodePlex at http://northwinddatabase.codeplex.com/. How to do it… We are going to connect to the SQL Database using Entity Framework and perform various operations on data. We will do this using the following steps: Add a new class named EFConnectionExample to the project. Add a new ADO.NET Entity Data Model named Northwind.edmx to the project; the Entity Data Model Wizard window will open. Choose Generate from database in the Choose Model Contents step. In the Choose Your Data Connection step, select the Northwind connection from the dropdown or create a new connection if it is not shown. Save the connection settings in the App.config file for later use and name the setting NorthwindEntities. If VS asks for the version of EF to use, select the most recent one. In the last step, choose the object to include in the model. Select the Tables, Views, Stored Procedures, and Functions checkboxes. Add the following method, retrieving every CompanyName, to the class: private IEnumerable<string> NamesOfCustomerCompanies() { using (var ctx = new NorthwindEntities()) { return ctx.Customers .Select(p => p.CompanyName).ToArray(); } } Add the following method, updating every customer located in Italy, to the class: private void UpdateItalians() { using (var ctx = new NorthwindEntities()) { ctx.Customers.Where(p => p.Country == "Italy") .ToList().ForEach(p => p.City = "Milan"); ctx.SaveChanges(); } } Add the following method, inserting a new order for the first Italian company alphabetically, to the class: private int FirstItalianPlaceOrder() { using (var ctx = new NorthwindEntities()) { var order = new Orders() { EmployeeID = 1, OrderDate = DateTime.UtcNow, ShipAddress = "My Address", ShipCity = "Milan", ShipCountry = "Italy", ShipName = "Good Ship", ShipPostalCode = "20100" }; ctx.Customers.Where(p => p.Country == "Italy") .OrderBy(p=>p.CompanyName) .First().Orders.Add(order); ctx.SaveChanges(); return order.OrderID; } } Add the following method, removing the previously inserted order, to the class: private void RemoveTheFunnyOrder(int orderId) { using (var ctx = new NorthwindEntities()) { var order = ctx.Orders .FirstOrDefault(p => p.OrderID == orderId); if (order != null) ctx.Orders.Remove(order); ctx.SaveChanges(); } } Add the following method, using the methods added earlier, to the class: public static void UseEFConnectionExample() { var example = new EFConnectionExample(); var customers=example.NamesOfCustomerCompanies(); foreach (var customer in customers) { Console.WriteLine(customer); } example.UpdateItalians(); var order=example.FirstItalianPlaceOrder(); example.RemoveTheFunnyOrder(order); } How it works… This recipe uses EF to connect and operate on a SQL Database. In step 1, we create a class that contains the recipe, and in step 2, we open the wizard for the creation of Entity Data Model (EDMX). We create the model, starting from an existing database in step 3 (it is also possible to write our own model and then persist it in an empty database), and then, we select the connection in step 4. In fact, there is no reference in the entire code to the Windows Azure SQL Database. The only reference should be in the App.config settings created in step 5; this can be changed to point to a SQL Server instance, leaving the code untouched. The last step of the EDMX creation consists of concrete mapping between the relational table and the object model, as shown in step 6. This method generates the code classes that map the table schema, using strong types and collections referred to as Navigation properties. It is also possible to start from the code, writing the classes that could represent the database schema. This method is known as Code-First. In step 7, we ask for every CompanyName of the Customers table. Every table in EF is represented by DbSet<Type>, where Type is the class of the entity. In steps 7 and 8, Customers is DbSet<Customers>, and we use a lambda expression to project (select) a property field and another one to create a filter (where) based on a property value. The SaveChanges method in step 8 persists to the database the changes detected in the disconnected object data model. This magic is one of the purposes of an ORM tool. In step 9, we use the navigation property (relationship) between a Customers object and the Orders collection (table) to add a new order with sample data. We use the OrderBy extension method to order the results by the specified property, and finally, we save the newly created item. Even now, EF automatically keeps track of the newly added item. Additionally, after the SaveChanges method, EF populates the identity field of Order (OrderID) with the actual value created by the database engine. In step 10, we use the previously obtained OrderID to remove the corresponding order from the database. We use the FirstOrDefault() method to test the existence of the ID, and then, we remove the resulting object like we removed an object from a plain old collection. In step 11, we use the methods created to run the demo and show the results. Deploying a Website Creating a Website is an administrative task, which is performed in the Azure Portal in the same way we provision every other building block. The Website created is like a "deployment slot", or better, "web space", since the abstraction given to the user is exactly that. Azure Websites does not require additional knowledge compared to an old-school hosting provider, where FTP was the standard for the deployment process. Actually, FTP is just one of the supported deployment methods in Websites, since Web Deploy is probably the best choice for several scenarios. Web Deploy is a Microsoft technology used for copying files and provisioning additional content and configuration to integrate the deployment process. Web Deploy runs on HTTP and HTTPS with basic (username and password) authentication. This makes it a good choice in networks where FTP is forbidden or the firewall rules are strict. Some time ago, Microsoft introduced the concept of Publish Profile, an XML file containing all the available deployment endpoints of a particular website that, if given to Visual Studio or Web Matrix, could make the deployment easier. Every Azure Website comes with a publish profile with unique credentials, so one can distribute it to developers without giving them grants on the Azure Subscription. Web Matrix is a client tool of Microsoft, and it is useful to edit live sites directly from an intuitive GUI. It uses Web Deploy to provide access to the remote filesystem as to perform remote changes. In Websites, we can host several websites on the same server farm, making administration easier and isolating the environment from the neighborhood. Moreover, virtual directories can be defined from the Azure Portal, enabling complex scenarios or making migrations easier. In this recipe, we will cope with the deployment process, using FTP and Web Deploy with some variants. Getting ready This recipe assumes we have and FTP client installed on the local machine (for example, FileZilla) and, of course, a valid Azure Subscription. We also need Visual Studio 2013 with the latest Azure SDK installed (at the time of writing, SDK Version 2.3). How to do it… We are going to create a new Website, create a new ASP.NET project, deploy it through FTP and Web Deploy, and also use virtual directories. We do this as follows: Create a new Website in the Azure Portal, specifying the following details: The URL prefix (that is, TestWebSite) is set to [prefix].azurewebsites.net The Web Hosting Plan (create a new one) The Region/Location (select West Europe) Click on the newly created Website and go to the Dashboard tab. Click on Download the publish profile and save it on the local computer. Open Visual Studio and create a new ASP.NET web application named TestWebSite, with an empty template and web forms' references. Add a sample Default.aspx page to the project and paste into it the following HTML: <h1>Root Application</h1> Press F5 and test whether the web application is displayed correctly. Create a local publish target. Right-click on the project and select Publish. Select Custom and specify Local Folder. In the Publish method, select File System and provide a local folder where Visual Studio will save files. Then click on Publish to complete. Publish via FTP. Open FileZilla and then open the Publish profile (saved in step 3) with a text editor. Locate the FTP endpoint and specify the following: publishUrl as the Host field username as the Username field userPWD as the Password field Delete the hostingstart.html file that is already present on the remote space. When we create a new Azure Website, there is a single HTML file in the root folder by default, which is served to the clients as the default page. By leaving it in the Website, the file could be served after users' deployments as well if no valid default documents are found. Drag-and-drop all the contents of the local folder with the binaries to the remote folder, then run the website. Publish via Web Deploy. Right-click on the Project and select Publish. Go to the Publish Web wizard start and select Import, providing the previously downloaded Publish Profile file. When Visual Studio reads the Web Deploy settings, it populates the next window. Click on Confirm and Publish the web application. Create an additional virtual directory. Go to the Configure tab of the Website on the Azure Portal. At the bottom, in the virtual applications and directories, add the following: /app01 with the path siteapp01 Mark it as Application Open the Publish Profile file and duplicate the <publishProfile> tag with the method FTP, then edit the following: Add the suffix App01 to profileName Replace wwwroot with app01 in publishUrl Create a new ASP.NET web application called TestWebSiteApp01 and create a new Default.aspx page in it with the following code: <h1>App01 Application</h1> Right-click on the TestWebSiteApp01 project and Publish. Select Import and provide the edited Publish Profile file. In the first step of the Publish Web wizard (go back if necessary), select the App01 method and select Publish. Run the Website's virtual application by appending the /app01 suffix to the site URL. How it works... In step 1, we create the Website on the Azure Portal, specifying the minimal set of parameters. If the existing web hosting plan is selected, the Website will start in the specified tier. In the recipe, by specifying a new web hosting plan, the Website is created in the free tier with some limitations in configuration. The recipe uses the Azure Portal located at https://manage.windowsazure.com. However, the new Azure Portal will be at https://portal.azure.com. New features will be probably added only in the new Portal. In steps 2 and 3, we download the Publish Profile file, which is an XML containing the various endpoints to publish the Website. At the time of writing, Web Deploy and FTP are supported by default. In steps 4, 5, and 6, we create a new ASP.NET web application with a sample ASPX page and run it locally. In steps 7, 8, and 9, we publish the binaries of the Website, without source code files, into a local folder somewhere in the local machine. This unit of deployment (the folder) can be sent across the wire via FTP, as we do in steps 10 to 13 using the credentials and the hostname available in the Publish Profile file. In steps 14 to 16, we use the Publish Profile file directly from Visual Studio, which recognizes the different methods of deployment and suggests Web Deploy as the default one. If we perform the steps 10-13, with steps14-16 we overwrite the existing deployment. Actually, Web Deploy compares the target files with the ones to deploy, making the deployment incremental for those file that have been modified or added. This is extremely useful to avoid unnecessary transfers and to save bandwidth. In steps 17 and 18, we configure a new Virtual Application, specifying its name and location. We can use an FTP client to browse the root folder of a website endpoint, since there are several folders such as wwwroot, locks, diagnostics, and deployments. In step 19, we manually edit the Publish Profile file to support a second FTP endpoint, pointing to the new folder of the Virtual Application. Visual Studio will correctly understand this while parsing the file again in step 22, showing the new deployment option. Finally, we verify whether there are two applications: one on the root folder / and one on the /app01 alias. There's more… Suppose we need to edit the website on the fly, editing a CSS of JS file or editing the HTML somewhere. We can do this using Web Matrix, which is available from the Azure Portal itself through a ClickOnce installation: Go to the Dashboard tab of the Website and click on WebMatrix at the bottom. Follow the instructions to install the software (if not yet installed) and, when it opens, select Edit live site directly (the magic is done through the Publish Profile file and Web Deploy). In the left-side tree, edit the Default.aspx file, and then save and run the Website again. Azure Websites gallery Since Azure Websites is a PaaS service, with no lock-in or particular knowledge or framework required to run it, it can hosts several Open Source CMS in different languages. Azure provides a set of built-in web applications to choose while creating a new website. This is probably not the best choice for production environments; however, for testing or development purposes, it should be a faster option than starting from scratch. Wizards have been, for a while, the primary resources for developers to quickly start off projects and speed up the process of creating complex environments. However, the Websites gallery creates instances of well-known CMS with predefined configurations. Instead, production environments are manually crafted, customizing each aspect of the installation. To create a new Website using the gallery, proceed as follows: Create a new Website, specifying from gallery. Select the web application to deploy and follow the optional configuration steps. If we create some resources (like databases) while using the gallery, they will be linked to the site in the Linked Resources tab. Building a simple cache for applications Azure Cache is a managed service with (at the time of writing this book) the following three offerings: Basic: This service has a unit size of 128 MB, up to 1 GB with one named cache (the default one) Standard: This service has a unit size of 1 GB, up to 10 GB with 10 named caches and support for notifications Premium: This service has a unit size of 5 GB, up to 150 GB with ten named caches, support for notifications, and high availability Different offerings have different unit prices, and remember that when changing from one offering to another, all the cache data is lost. In all offerings, users can define the items' expiration. The Cache service listens to a specific TCP port. Accessing it from a .NET application is quite simple, with the Microsoft ApplicationServer Caching library available on NuGet. In the Microsoft.ApplicationServer.Caching namespace, the following are all the classes that are needed to operate: DataCacheFactory: This class is responsible for instantiating the Cache proxies to interpret the configuration settings. DataCache: This class is responsible for the read/write operation against the cache endpoint. DataCacheFactoryConfiguration: This is the model class of the configuration settings of a cache factory. Its usage is optional as cache can be configured in the App/Web.config file in a specific configuration section. Azure Cache is a key-value cache. We can insert and even get complex objects with arbitrary tree depth using string keys to locate them. The importance of the key is critical, as in a single named cache, only one object can exist for a given key. The architects and developers should have the proper strategy in place to deal with unique (and hierarchical) names. Getting ready This recipe assumes that we have a valid Azure Cache endpoint of the standard type. We need the standard type because we use multiple named caches, and in later recipes, we use notifications. We can create a Standard Cache endpoint of 1 GB via PowerShell. Perform the following steps to create the Standard Cache endpoint : Open the Azure PowerShell and type Add-AzureAccount. A popup window might appear. Type your credentials connected to a valid Azure subscription and continue. Optionally, select the proper Subscription, if not the default one. Type this command to create a new Cache endpoint, replacing myCache with the proper unique name: New-AzureManagedCache -Name myCache -Location "West Europe" -Sku Standard -Memory 1GB After waiting for some minutes until the endpoint is ready, go to the Azure Portal and look for the Manage Keys section to get one of the two Access Keys of the Cache endpoint. In the Configure section of the Cache endpoint, a cache named default is created by default. In addition, create two named caches with the following parameters: Expiry Policy: Absolute Time: 10 Notifications: Enabled Expiry Policy could be Absolute (the default expiration time or the one set by the user is absolute, regardless of how many times the item has been accessed), Sliding (each time the item has been accessed, the expiration timer resets), or Never (items do not expire). This Azure Cache endpoint is now available in the Management Portal, and it will be used in the entire article. How to do it… We are going to create a DataCache instance through a code-based configuration. We will perform simple operations with Add, Get, Put, and Append/Prepend, using a secondary-named cache to transfer all the contents of the primary one. We will do this by performing the following steps: Add a new class named BuildingSimpleCacheExample to the project. Install the Microsoft.WindowsAzure.Caching NuGet package. Add the following using statement to the top of the class file: using Microsoft.ApplicationServer.Caching; Add the following private members to the class: private DataCacheFactory factory = null; private DataCache cache = null; Add the following constructor to the class: public BuildingSimpleCacheExample(string ep, string token,string cacheName) { DataCacheFactoryConfiguration config = new DataCacheFactoryConfiguration(); config.AutoDiscoverProperty = new DataCacheAutoDiscoverProperty(true, ep); config.SecurityProperties = new DataCacheSecurity(token, true); factory = new DataCacheFactory(config); cache = factory.GetCache(cacheName); } Add the following method, creating a palindrome string into the cache: public void CreatePalindromeInCache() { var objKey = "StringArray"; cache.Put(objKey, ""); char letter = 'A'; for (int i = 0; i < 10; i++) { cache.Append(objKey, char.ConvertFromUtf32((letter+i))); cache.Prepend(objKey, char.ConvertFromUtf32((letter + i))); } Console.WriteLine(cache.Get(objKey)); } Add the following method, adding an item into the cache to analyze its subsequent retrievals: public void AddAndAnalyze() { var randomKey = DateTime.Now.Ticks.ToString(); var value="Cached string"; cache.Add(randomKey, value); DataCacheItem cacheItem = cache.GetCacheItem(randomKey); Console.WriteLine(string.Format( "Item stored in {0} region with {1} expiration", cacheItem.RegionName,cacheItem.Timeout)); cache.Put(randomKey, value, TimeSpan.FromSeconds(60)); cacheItem = cache.GetCacheItem(randomKey); Console.WriteLine(string.Format( "Item stored in {0} region with {1} expiration", cacheItem.RegionName, cacheItem.Timeout)); var version = cacheItem.Version; var obj = cache.GetIfNewer(randomKey, ref version); if (obj == null) { //No updates } } Add the following method, transferring the contents of the cache named initially into a second one: public void BackupToDestination(string destCacheName) { var destCache = factory.GetCache(destCacheName); var dump = cache.GetSystemRegions() .SelectMany(p => cache.GetObjectsInRegion(p)) .ToDictionary(p=>p.Key,p=>p.Value); foreach (var item in dump) { destCache.Put(item.Key, item.Value); } } Add the following method to clear the cache named first: public void ClearCache() { cache.Clear(); } Add the following method, using the methods added earlier, to the class: public static void RunExample() { var cacheName = "[named cache 1]"; var backupCache = "[named cache 2]"; string endpoint = "[cache endpoint]"; string token = "[cache token/key]"; BuildingSimpleCacheExample example = new BuildingSimpleCacheExample(endpoint, token, cacheName); example.CreatePalindromeInCache(); example.AddAndAnalyze(); example.BackupToDestination(backupCache); example.ClearCache(); } How it works... From steps 1 to 3, we set up the class. In step 4, we add private members to store the DataCacheFactory object used to create the DataCache object to access the Cache service. In the constructor that we add in step 5, we initialize the DataCacheFactory object using a configuration model class (DataCacheFactoryConfiguration). This strategy is for code-based initialization whenever settings cannot stay in the App.config/Web.config file. In step 6, we use the Put() method to write an empty string into the StringArray bucket. We then use the Append() and Prepend() methods, designed to concatenate strings to existing strings, to build a palindrome string in the memory cache. This sample does not make any sense in real-world scenarios, and we must pay attention to some of the following issues: Writing an empty string into the cache is somehow useless. Each Append() or Prepend() operation travels on TCP to the cache and goes back. Though it is very simple, it requires resources, and we should always try to consolidate calls. In step 7, we use the Add() method to add a string to the cache. The difference between the Add() and Put() methods is that the first method throws an exception if the item already exists, while the second one always overwrites the existing value (or writes it for the first time). GetCacheItem() returns a DataCacheItem object, which wraps the value together with other metadata properties, such as the following: CacheName: This is the named cache where the object is stored. Key: This is the key of the associated bucket. RegionName (user defined or system defined): This is the region of the cache where the object is stored. Size: This is the size of the object stored. Tags: These are the optional tags of the object, if it is located in a user-defined region. Timeout: This is the current timeout before the object would expire. Version: This is the version of the object. This is a DataCacheItemVersion object whose properties are not accessible due to their modifier. However, it is not important to access this property, as the Version object is used as a token against the Cache service to implement the optimistic concurrency. As for the timestamp value, its semantic can stay hidden from developers. The first Add() method does not specify a timeout for the object, leaving the default global expiration timeout, while the next Put() method does, as we can check in the next Get() method. We finally ask the cache about the object with the GetIfNewer() method, passing the latest version token we have. This conditional Get method returns null if the object we own is already the latest one. In step 8, we list all the keys of the first named cache, using the GetSystemRegions() method (to first list the system-defined regions), and for each region, we ask for their objects, copying them into the second named cache. In step 9, we clear all the contents of the first cache. In step 10, we call the methods added earlier, specifying the Cache endpoint to connect to and the token/password, along with the two named caches in use. Replace [named cache 1], [named cache 2], [cache endpoint], and [cache token/key] with actual values. There's more… Code-based configuration is useful when the settings stay in a different place as compared to the default config files for .NET. It is not a best practice to hardcode them, so this is the standard way to declare them in the App.config file: <configSections> <section name="dataCacheClients" type="Microsoft.ApplicationServer.Caching.DataCacheClientsSection, Microsoft.ApplicationServer.Caching.Core" allowLocation="true" allowDefinition="Everywhere" /> </configSections> The XML mentioned earlier declares a custom section, which should be as follows: <dataCacheClients> <dataCacheClient name="[name of cache]"> <autoDiscover isEnabled="true" identifier="[domain of cache]" /> <securityProperties mode="Message" sslEnabled="true"> <messageSecurity authorizationInfo="[token of endpoint]" /> </securityProperties> </dataCacheClient> </dataCacheClients> In the upcoming recipes, we will use this convention to set up the DataCache objects. ASP.NET Support With almost no effort, the Azure Cache can be used as Output Cache in ASP.NET to save the session state. To enable this, in addition to the configuration mentioned earlier, we need to include those declarations in the <system.web> section as follows: <sessionState mode="Custom" customProvider="AFCacheSessionStateProvider"> <providers> <add name="AFCacheSessionStateProvider" type="Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider, Microsoft.Web.DistributedCache" cacheName="[named cache]" dataCacheClientName="[name of cache]" applicationName="AFCacheSessionState"/> </providers> </sessionState> <caching> <outputCache defaultProvider="AFCacheOutputCacheProvider"> <providers> <add name="AFCacheOutputCacheProvider" type="Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider, Microsoft.Web.DistributedCache" cacheName="[named cache]" dataCacheClientName="[name of cache]" applicationName="AFCacheOutputCache" /> </providers> </outputCache> </caching> The difference between [name of cache] and [named cache] is as follows: The [name of cache] part is a friendly name of the cache client declared above an alias. The [named cache] part is the named cache created into the Azure Cache service. Connecting to the Azure Storage service In an Azure Cloud Service, the storage account name and access key are stored in the service configuration file. By convention, the account name and access key for data access are provided in a setting named DataConnectionString. The account name and access key needed for Azure Diagnostics must be provided in a setting named Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString. The DataConnectionString setting must be declared in the ConfigurationSettings section of the service definition file. However, unlike other settings, the connection string setting for Azure Diagnostics is implicitly defined when the Diagnostics module is specified in the Imports section of the service definition file. Consequently, it must not be specified in the ConfigurationSettings section. A best practice is to use different storage accounts for application data and diagnostic data. This reduces the possibility of application data access being throttled by competition for concurrent writes from the diagnostics monitor. What is Throttling? In shared services, where the same resources are shared between tenants, limiting the concurrent access to them is critical to provide service availability. If a client misuses the service or, better, generates a huge amount of traffic, other tenants pointing to the same shared resource could experience unavailability. Throttling (also known as Traffic Control plus Request Cutting) is one of the most adopted solutions that is solving this issue. It also provides a security boundary between application data and diagnostics data, as diagnostics data might be accessed by individuals who should have no access to application data. In the Azure Storage library, access to the storage service is through one of the Client classes. There is one Client class for each Blob service, Queue service, and Table service; they are CloudBlobClient, CloudQueueClient, and CloudTableClient, respectively. Instances of these classes store the pertinent endpoint as well as the account name and access key. The CloudBlobClient class provides methods to access containers, list their contents, and get references to containers and blobs. The CloudQueueClient class provides methods to list queues and get a reference to the CloudQueue instance that is used as an entry point to the Queue service functionality. The CloudTableClient class provides methods to manage tables and get the TableServiceContext instance that is used to access the WCF Data Services functionality while accessing the Table service. Note that the CloudBlobClient, CloudQueueClient, and CloudTableClient instances are not thread safe, so distinct instances should be used when accessing these services concurrently. The client classes must be initialized with the account name, access key, as well as the appropriate storage service endpoint. The Microsoft.WindowsAzure namespace has several helper classes. The StorageCredential class initializes an instance from an account name and access key or from a shared access signature. In this recipe, we'll learn how to use the CloudBlobClient, CloudQueueClient, and CloudTableClient instances to connect to the storage service. Getting ready This recipe assumes that the application's configuration file contains the following: <appSettings> <add key="DataConnectionString" value="DefaultEndpointsProtocol=https;AccountName={ACCOUNT_NAME};AccountKey={ACCOUNT_KEY}"/> <add key="AccountName" value="{ACCOUNT_NAME}"/> <add key="AccountKey" value="{ACCOUNT_KEY}"/> </appSettings> We must replace {ACCOUNT_NAME} and {ACCOUNT_KEY} with appropriate values for the storage account name and access key, respectively. We are not working in a Cloud Service but in a simple console application. Storage services, like many other building blocks of Azure, can also be used separately from on-premise environments. How to do it... We are going to connect to the Table service, the Blob service, and the Queue service, and perform a simple operation on each. We will do this using the following steps: Add a new class named ConnectingToStorageExample to the project. Add the following using statements to the top of the class file: using Microsoft.WindowsAzure.Storage; using Microsoft.WindowsAzure.Storage.Blob; using Microsoft.WindowsAzure.Storage.Queue; using Microsoft.WindowsAzure.Storage.Table; using Microsoft.WindowsAzure.Storage.Auth; using System.Configuration; The System.Configuration assembly should be added via the Add Reference action onto the project, as it is not included in most of the project templates of Visual Studio. Add the following method, connecting the blob service, to the class: private static void UseCloudStorageAccountExtensions() { CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse( ConfigurationManager.AppSettings[ "DataConnectionString"]); CloudBlobClient cloudBlobClient = cloudStorageAccount.CreateCloudBlobClient(); CloudBlobContainer cloudBlobContainer = cloudBlobClient.GetContainerReference( "{NAME}"); cloudBlobContainer.CreateIfNotExists(); } Add the following method, connecting the Table service, to the class: private static void UseCredentials() { string accountName = ConfigurationManager.AppSettings[ "AccountName"]; string accountKey = ConfigurationManager.AppSettings[ "AccountKey"]; StorageCredentials storageCredentials = new StorageCredentials( accountName, accountKey); CloudStorageAccount cloudStorageAccount = new CloudStorageAccount(storageCredentials, true); CloudTableClient tableClient = new CloudTableClient( cloudStorageAccount.TableEndpoint, storageCredentials); CloudTable table = tableClient.GetTableReference("{NAME}"); table.CreateIfNotExists(); } Add the following method, connecting the Queue service, to the class: private static void UseCredentialsWithUri() { string accountName = ConfigurationManager.AppSettings[ "AccountName"]; string accountKey = ConfigurationManager.AppSettings[ "AccountKey"]; StorageCredentials storageCredentials = new StorageCredentials( accountName, accountKey); StorageUri baseUri = new StorageUri(new Uri(string.Format( "https://{0}.queue.core.windows.net/", accountName))); CloudQueueClient cloudQueueClient = new CloudQueueClient(baseUri, storageCredentials); CloudQueue cloudQueue = cloudQueueClient.GetQueueReference("{NAME}"); cloudQueue.CreateIfNotExists(); } Add the following method, using the other methods, to the class: public static void UseConnectionToStorageExample() { UseCloudStorageAccountExtensions(); UseCredentials(); UseCredentialsWithUri(); } How it works... In steps 1 and 2, we set up the class. In step 3, we implement the standard way to access the storage service using the Storage Client library. We use the static CloudStorageAccount.Parse() method to create a CloudStorageAccount instance from the value of the connection string stored in the configuration file. We then use this instance with the CreateCloudBlobClient() extension method of the CloudStorageAccount class to get the CloudBlobClient instance that we use to connect to the Blob service. We can also use this technique with the Table service and the Queue service, using the relevant extension methods, CreateCloudTableClient() and CreateCloudQueueClient(), respectively, for them. We complete this example using the CloudBlobClient instance to get a CloudBlobContainer reference to a container and then create it if it does not exist We need to replace {NAME} with the name for a container. In step 4, we create a StorageCredentials instance directly from the account name and access key. We then use this to construct a CloudStorageAccount instance, specifying that any connection should use HTTPS. Using this technique, we need to provide the Table service endpoint explicitly when creating the CloudTableClient instance. We then use this to create the table. We need to replace {NAME} with the name of a table. We can use the same technique with the Blob service and Queue service using the relevant CloudBlobClient or CloudQueueClient constructor. In step 5, we use a similar technique, except that we avoid the intermediate step of using a CloudStorageAccount instance and explicitly provide the endpoint for the Queue service. We use the CloudQueueClient instance created in this step to create the queue. We need to replace {NAME} with the name of a queue. Note that we hardcoded the endpoint for the Queue service. Though this last method is officially supported, it is not a best practice to bind our code to hardcoded strings with endpoint URIs. So, it is preferable to use one of the previous methods that hides the complexity of the URI generation at the library level. In step 6, we add a method that invokes the methods added in the earlier steps. There's more… With the general availability of the .NET Framework Version 4.5, many libraries of the CLR have been added with the support of asynchronous methods with the Async/Await pattern. Latest versions of the Azure Storage Library also have these overloads, which are useful while developing mobile applications, and fast web APIs. They are generally useful when it is needed to combine the task execution model into our applications. Almost each long-running method of the library has its corresponding methodAsync() method to be called as follows: await cloudQueue.CreateIfNotExistsAsync(); In the rest of the book, we will continue to use the standard, synchronous pattern. Adding messages to a Storage queue The CloudQueue class in the Azure Storage library provides both synchronous and asynchronous methods to add a message to a queue. A message comprises up to 64 KB bytes of data (48 KB if encoded in Base64). By default, the Storage library Base64 encodes message content to ensure that the request payload containing the message is valid XML. This encoding adds overhead that reduces the actual maximum size of a message. A message for a queue should not be intended to transport a big payload, since the purpose of a Queue is just messaging and not storing. If required, a user can store the payload in a Blob and use a Queue message to point to that, letting the receiver fetch the message along with the Blob from its remote location. Each message added to a queue has a time-to-live property after which it is deleted automatically. The maximum and default time-to-live value is 7 days. In this recipe, we'll learn how to add messages to a queue. Getting ready This recipe assumes the following code is in the application configuration file: <appSettings> <add key="DataConnectionString" value="DefaultEndpointsProtocol=https;AccountName={ACCOUNT_NAME};AccountKey={ACCOUNT_KEY}"/> </appSettings> We must replace {ACCOUNT_NAME} and {ACCOUNT_KEY} with appropriate values of the account name and access key. How to do it... We are going to create a queue and add some messages to it. We do this as follows: Add a new class named AddMessagesOnStorageExample to the project. Install the WindowsAzure.Storage NuGet package and add the following assembly references to the project: System.Configuration Add the following using statements to the top of the class file: using Microsoft.WindowsAzure.Storage; using Microsoft.WindowsAzure.Storage.Queue; using System.Configuration; Add the following private member to the class: private CloudQueue cloudQueueClient; Add the following constructor to the class: public AddMessagesOnStorageExample(String queueName) { CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse( ConfigurationManager.AppSettings[ "DataConnectionString"]); CloudQueueClient cloudQueueClient = cloudStorageAccount.CreateCloudQueueClient(); cloudQueue = cloudQueueClient.GetQueueReference(queueName); cloudQueue.CreateIfNotExists(); } Add the following method to the class, adding two messages: public void AddMessages() { String content1 = "Do something"; CloudQueueMessage message1 = new CloudQueueMessage(content1); cloudQueue.AddMessage(message1); String content2 = "Do something that expires in 1 day"; CloudQueueMessage message2 = new CloudQueueMessage(content2); cloudQueue.AddMessage(message2, TimeSpan.FromDays(1.0)); String content3 = "Do something that expires in 2 hours,"+ " starting in 1 hour from now"; CloudQueueMessage message3 = new CloudQueueMessage(content3); cloudQueue.AddMessage(message2, TimeSpan.FromHours(2),TimeSpan.FromHours(1)); } Add the following method, that uses the AddMessage() method, to the class: public static void UseAddMessagesExample() { String queueName = "{QUEUE_NAME}"; AddMessagesOnStorageExample example = new AddMessagesOnStorageExample (queueName); example.AddMessages(); } How it works... In steps 1 through 3, we set up the class. In step 4, we add a private member to store the CloudQueue object used to interact with the Queue service. We initialize this in the constructor we add in step 5 where we also create the queue. In step 6, we add a method that adds three messages to a queue. We create three CloudQueueMessage objects. We add the first message to the queue with the default time-to-live of seven days, the second is added specifying an expiration of 1 day, and the third will become visible after 1 hour since its entrance in the queue, with an absolute expiration of 2 hours. Note that a client (library) exception is thrown if we specify a visibility delay higher than the absolute TTL of the message. This is naturally obvious and it is enforced at the client side, instead making a (failing) server call. In step 7, we add a method that invokes the methods we added earlier. We need to replace {QUEUE_NAME} with an appropriate name for a queue. There's more… To clear the queue from the messages we added in this recipe, we can proceed by calling the Clear() method in the CloudQueue class as follows: public void ClearQueue() { cloudQueue.Clear(); } Summary In this article, we have learned some of the recipes in order to build a complete overview of the software infrastructure that we need to set up on the cloud. Resources for Article: Further resources on this subject: Backups in the VMware View Infrastructure [Article] vCloud Networks [Article] Setting Up a Test Infrastructure [Article]

  Joomla! 1.6 First Look A concise guide to everything that's new in Joomla! 1.6.         Read more about this book       (For more resources on Joomla!, see here.) Question: What are the server requirements for installing Joomla! 1.6 on a web hosting account? Answer: The following system requirements have remained the same since the 1.5 release: Apache 1.3.x or higher. Apache is the web server software that processes the PHP instructions for how to pull in contents from the database and display a web page. XML and Zlib support. Your host's PHP installation should support XML and Zlib functionality. But the PHP and MySQL requirements have changed. To enable you to run a Joomla! 1.6 powered website, your web hosting account should support: PHP 5.2 or higher. PHP is the scripting language that Joomla! is written in. MySQL 5.0.4 or higher. The MySQL database is where Joomla! stores its data (the contents of your site).   Question: What are the changes for templates in Joomla! 1.6? Answer: Templates created for version 1.5 can't be used in Joomla! 1.6. The new release uses clean, semantic HTML code, without using tables for layout purposes. This is good news, as template developers are no longer required to add so-called template overrides in order to achieve a semantic design. However, it is one of the reasons that developers will have to upgrade their existing code to move a 1.5 template to version 1.6. Joomla! 1.6 also introduces some other nice template features, such as the ability to use ‘subtemplates’ (called Template Styles in Joomla!). This new feature allows you to easily create individual styling for parts of your site.   Question: What’s new about categorizing content in Joomla! 1.6? Answer: The main thing that a content management system should help you in doing is of course to publish content and to manage existing content with minimal effort. Joomla! 1.6 allows you to organize content exactly as you want. Up to Joomla! 1.5, you could only classify your content in three levels: sections would contain categories, and categories would hold articles. Although this didn't pose problems for most sites, it was nevertheless a strange restriction. That's why Joomla! 1.6 introduces a more flexible system of classification. Categories can now hold an unlimited number of subcategories. This means that you can have a hierarchy. A category can hold as many subcategories as you need. This concept is called "unlimited nested categories". In most cases you won't need more than two or three subcategories, but if you do, there's nothing to stop you. You can check the content category hierarchy in the Category Manager. Child categories are displayed indented, with small gray lines indicating the sublevel: The above screenshot shows the nested categories contained in the sample data that comes with Joomla! 1.6. As you can see, all article content is stored in subcategories of the main category Sample Data-Articles.   Question: What's the difference between Root user, Super Administrator, Super User, Admin? Answer: When you install Joomla!, there's always one root user, allowed to do anything in the administration area. In Joomla! 1.5, this root user was called Super Administrator. In Joomla! 1.6, this name has been changed to Super User. Another point to note is that the root user is always a Super User—but there can be more Super Users who aren't root users. The root user is a unique Super User who can assign new users to the Super User group. But there's always just one root user created when installing Joomla! Only this root user can change another Super User's details. (You can always identify the root user in the User Manager by his fixed ID number, which in Joomla! 1.6 is always 42).   Question: What's the use of the new User Permissions tab? Answer: When browsing the Global Configuration screen, you'll notice that there's a new tab called Permissions. It's where you set all site-wide user permissions. In Joomla! 1.6, you have much more control over user permissions. You can create new user groups and set permissions on different levels- not just site-wide, as was previously the case in Joomla! 1.5.   Question: What are the changes in the article editor? Answer: Creating or editing an article in Joomla! 1.6 will seem very familiar to 1.5 users. Go to Content | Article Manager | New or Edit to display the article editor screen: (Move the mouse over the image to enlarge.) On closer inspection, you'll notice some items have been renamed or rearranged: In the New Article section, you can set the article to be Featured. This is just another word for what was previously called 'Front Page' or 'Show on Front Page'. In the Article Options, references to sections have gone. The Show Category option allows you to show the Category Title of the current category. The Show Parent option allows you to also show the parent category title among the article details. In the Article Options, references to sections have gone. The Show Category option allows you to show the Category Title of the current category. The Show Parent option allows you to also show the parent category title among the article details. In the New Article section, there's a Permissions button that jumps to the separate Article Permissions section at the bottom of the screen. As you can see, the new user permissions options are available on many levels in the Joomla! 1.6 interface. Here you can set user permissions to delete or edit the current article.   Question: What's the Options button In the Menu Manager: Menus screen about? Answer: In the Menu Manager: Menus screen, there's a new button called Options: Clicking on it will open a pop up screen allowing you set all default User Permissions for all menus.   Question: What's new about the Access Control Levels system? Answer:In Joomla! 1.5, a fixed set of user groups was available, ranging from "Public" users (anyone with access to the frontend of the site) to "Super Administrators", allowed to log in to the backend and do anything. The ACL system in Joomla! 1.6 is much more flexible: Instead of fixed user groups with fixed sets of permissions, you can create as many groups as you want and grant the people in those groups any combination of permissions. ACL enables you to control anything users can do on the site: log in, create, edit, delete, publish, unpublish, trash, archive, manage, or administer things. Users are no longer limited to only one group: a user can belong to different groups at the same time. This allows you to give particular users both the set of permissions for one group and another group without having to create a third, combined set of permissions from the ground up. Permissions no longer apply to the whole site as they did in Joomla! 1.5. You can now set permissions for specific parts of the site. Permissions apply to either the whole site, or to specific components, categories, or items (such as a single article).   Question: How can we set access levels for users? Answer: By default, three Viewing Access Levels are available: Public, Registered, and Special. Go to Users | User Manager and click on the Viewing Access Levels tab to see these levels: This is what the three default Viewing Access Levels mean: Public means that there are no special viewing permissions involved. It's the set of permissions for the Public user group, who are only allowed access to the public site. Registered is the set of permissions for Registered Users. These by default are allowed to log in to the site to view the site parts that are set to the Registered access level. Special is the set of viewing permissions for all users that can log in to the backend (Manager, Administrator, Super User)   Question: How can I customize the logo on the site? Answer: You can customize the logo just by changing the Template Style settings. Let's find out how this works: Navigate to Extensions | Template Manager. Click on the Styles tab and then click on the link Beez2 - Default. The Template Manager: Edit Style screen is displayed. In the Advanced Options panel, locate the Logo option and click on Select. A pop-up screen appears. In the Upload files section, click on Browse to select a logo file from your computer. For best results, use a PNG file with a transparent background applied, with a maximum height of about 65 pixels. Select the image file and click on Start Upload. The message Upload Complete is displayed: Click on the logo image to select it and then click on Insert in the top right corner. In the Edit Style screen, click on Save and then click on View Site. The new logo image is displayed and replaces the Joomla! logo: To further customize the logo and header area, enter a short description of your site in the Advanced Options | Site Description box. This will replace the site tag line, just below the logo.   Question: What is the Redirect Manager? Answer: A new addition in 1.6 is the Redirect Manager, which you can find in the Components menu. This application can be quite useful, especially if you're migrating a 1.5 site to 1.6. When changing to a new site, many URLs from your old site are bound to change. This can result in lots of broken links from other sites that still point to the old URLs. The Redirect Manager helps you to direct visitors who come to your site through outdated links. In the Redirect Manager, just enter the old links and tell Joomla! what new pages it should show instead:   Question: What are the new module features in Joomla! 1.6? Answer: In Joomla! 1.6, using modules is more flexible: You can schedule the time during which a module should display. In previous versions of Joomla!, you could set a start date and an end date for publishing articles. Now this is also possible for modules. Modules are always assigned to one or more menu items. However, when editing a menu in Joomla! 1.5, there was no way to find out or change which modules were assigned to that menu item. You had to leave the menu item edit screen, navigate to the particular module's settings in the Module Manager, and check the module's menu assignment there. In Joomla! 1.6, you can set what modules are assigned to a menu link directly when you're editing a menu link.   Summary In this article we covered some of the most frequently asked questions on Joomla! 1.6. Further resources on this subject: Installing and Configuring Joomla! 1.5 [Article] Search Engine Optimization in Joomla! [Article] Joomla! 1.6: Organizing and Managing Content [Article] Joomla! 1.6: Managing Site Users with Access Control [Article] Mastering Joomla! 1.5 Extension and Framework Development Second Edition [Book]