How-To Tutorials

In this article by Antonio Esposito, author of Learning .NET High Performance Programming, we will learn about low-pass audio filtering implemented using .NET, and also learn about MVVM and XAML. Model-View-ViewModel and XAML The MVVM pattern is another descendant of the MVC pattern. Born from an extensive update to the MVP pattern, it is at the base of all eXtensible Application Markup Language (XAML) language-based frameworks, such as Windows presentation foundation (WPF), Silverlight, Windows Phone applications, and Store Apps (formerly known as Metro-style apps). MVVM is different from MVC, which is used by Microsoft in its main web development framework in that it is used for desktop or device class applications. The first and (still) the most powerful application framework using MVVM in Microsoft is WPF, a desktop class framework that can use the full .NET 4.5.3 environment. Future versions within Visual Studio 2015 will support built-in .NET 4.6. On the other hand, all other frameworks by Microsoft that use the XAML language supporting MVVM patterns are based on a smaller edition of .NET. This happens with Silverlight, Windows Store Apps, Universal Apps, or Windows Phone Apps. This is why Microsoft made the Portable Library project within Visual Studio, which allows us to create shared code bases compatible with all frameworks. While a Controller in MVC pattern is sort of a router for requests to catch any request and parsing input/output Models, the MVVM lies behind any View with a full two-way data binding that is always linked to a View's controls and together at Model's properties. Actually, multiple ViewModels may run the same View and many Views can use the same single/multiple instance of a given ViewModel. A simple MVC/MVVM design comparative We could assert that the experience offered by MVVM is like a film, while the experience offered by MVC is like photography, because while a Controller always makes one-shot elaborations regarding the application user requests in MVC, in MVVM, the ViewModel is definitely the view! Not only does a ViewModel lie behind a View, but we could also say that if a VM is a body, then a View is its dress. While the concrete View is the graphical representation, the ViewModel is the virtual view, the un-concrete view, but still the View. In MVC, the View contains the user state (the value of all items showed in the UI) until a GET/POST invocation is sent to the web server. Once sent, in the MVC framework, the View simply binds one-way reading data from a Model. In MVVM, behaviors, interaction logic, and user state actually live within the ViewModel. Moreover, it is again in the ViewModel that any access to the underlying Model, domain, and any persistence provider actually flows. Between a ViewModel and View, a data connection called data binding is established. This is a declarative association between a source and target property, such as Person.Name with TextBox.Text. Although it is possible to configure data binding by imperative coding (while declarative means decorating or setting the property association in XAML), in frameworks such as WPF and other XAML-based frameworks, this is usually avoided because of the more decoupled result made by the declarative choice. The most powerful technology feature provided by any XAML-based language is actually the data binding, other than the simpler one that was available in Windows Forms. XAML allows one-way binding (also reverted to the source) and two-way binding. Such data binding supports any source or target as a property from a Model or ViewModel or any other control's dependency property. This binding subsystem is so powerful in XAML-based languages that events are handled in specific objects named Command, and this can be data-bound to specific controls, such as buttons. In the .NET framework, an event is an implementation of the Observer pattern that lies within a delegate object, allowing a 1-N association between the only source of the event (the owner of the event) and more observers that can handle the event with some specific code. The only object that can raise the event is the owner itself. In XAML-based languages, a Command is an object that targets a specific event (in the meaning of something that can happen) that can be bound to different controls/classes, and all of those can register handlers or raise the signaling of all handlers. An MVVM performance map analysis Performance concerns Regarding performance, MVVM behaves very well in several scenarios in terms of data retrieval (latency-driven) and data entry (throughput- and scalability-driven). The ability to have an impressive abstraction of the view in the VM without having to rely on the pipelines of MVC (the actions) makes the programming very pleasurable and give the developer the choice to use different designs and optimization techniques. Data binding itself is done by implementing specific .NET interfaces that can be easily centralized. Talking about latency, it is slightly different from previous examples based on web request-response time, unavailable in MVVM. Theoretically speaking, in the design pattern of MVVM, there is no latency at all. In a concrete implementation within XAML-based languages, latency can refer to two different kinds of timings. During data binding, latency is the time between when a VM makes new data available and a View actually renders it. Instead, during a command execution, latency is the time between when a command is invoked and all relative handlers complete their execution. We usually use the first definition until differently specified. Although the nominal latency is near zero (some milliseconds because of the dictionary-based configuration of data binding), specific implementation concerns about latency actually exist. In any Model or ViewModel, an updated data notification is made by triggering the View with the INotifyPropertyChanged interface. The .NET interface causes the View to read the underlying data again. Because all notifications are made by a single .NET event, this can easily become a bottleneck because of the serialized approach used by any delegate or event handlers in the .NET world. On the contrary, when dealing with data that flows from the View to the Model, such an inverse binding is usually configured declaratively within the {Binding …} keyword, which supports specifying binding directions and trigger timing (to choose from the control's lost focus CLR event or anytime the property value changes). The XAML data binding does not add any measurable time during its execution. Although this, as said, such binding may link multiple properties or the control's dependency properties together. Linking this interaction logic could increase latency time heavily, adding some annoying delay at the View level. One fact upon all, is the added latency by any validation logic. It is even worse if such validation is other than formal, such as validating some ID or CODE against a database value. Talking about scalability, MVVM patterns does some work here, while we can make some concrete analysis concerning the XAML implementation. It is easy to say that scaling out is impossible because MVVM is a desktop class layered architecture that cannot scale. Instead, we can say that in a multiuser scenario with multiple client systems connected in a 2-tier or 3-tier system architecture, simple MVVM and XAML-based frameworks will never act as bottlenecks. The ability to use the full .NET stack in WPF gives us the chance to use all synchronization techniques available, in order to use a directly connected DBMS or middleware tier. Instead of scaling up by moving the application to an increased CPU clock system, the XAML-based application would benefit more from an increased CPU core count system. Obviously, to profit from many CPU cores, mastering parallel techniques is mandatory. About the resource usage, MVVM-powered architectures require only a simple POCO class as a Model and ViewModel. The only additional requirement is the implementation of the INotifyPropertyChanged interface that costs next to nothing. Talking about the pattern, unlike MVC, which has a specific elaboration workflow, MVVM does not offer this functionality. Multiple commands with multiple logic can process their respective logic (together with asynchronous invocation) with the local VM data or by going down to the persistence layer to grab missing information. We have all the choices here. Although MVVM does not cost anything in terms of graphical rendering, XAML-based frameworks make massive use of hardware-accelerated user controls. Talking about an extreme choice, Windows Forms with Graphics Device Interface (GDI)-based rendering require a lot less resources and can give a higher frame rate on highly updatable data. Thus, if a very high FPS is needed, the choice of still rendering a WPF area in GDI is available. For other XAML languages, the choice is not so easy to obtain. Obviously, this does not mean that XAML is slow in rendering with its DirectX based engine. Simply consider that WPF animations need a good Graphics Processing Unit (GPU), while a basic GDI animation will execute on any system, although it is obsolete. Talking about availability, MVVM-based architectures usually lead programmers to good programming. As MVC allows it, MVVM designs can be tested because of the great modularity. While a Controller uses a pipelined workflow to process any requests, a ViewModel is more flexible and can be tested with multiple initialization conditions. This makes it more powerful but also less predictable than a Controller, and hence is tricky to use. In terms of design, the Controller acts as a transaction script, while the ViewModel acts in a more realistic, object-oriented approach. Finally, yet importantly, throughput and efficiency are simply unaffected by MVVM-based architectures. However, because of the flexibility the solution gives to the developer, any interaction and business logic design may be used inside a ViewModel and their underlying Models. Therefore, any success or failure regarding those performance aspects are usually related to programmer work. In XAML frameworks, throughput is achieved by an intensive use of asynchronous and parallel programming assisted by a built-in thread synchronization subsystem, based on the Dispatcher class that deals with UI updates. Low-pass filtering for Audio Low-pass filtering has been available since 2008 in the native .NET code. NAudio is a powerful library helping any CLR programmer to create, manipulate, or analyze audio data in any format. Available through NuGet Package Manager, NAudio offers a simple and .NET-like programming framework, with specific classes and stream-reader for audio data files. Let's see how to apply the low-pass digital filter in a real audio uncompressed file in WAVE format. For this test, we will use the Windows start-up default sound file. The chart is still made in a legacy Windows Forms application with an empty Form1 file, as shown in the previous example: private async void Form1_Load(object sender, EventArgs e) {    //stereo wave file channels    var channels = await Task.Factory.StartNew(() =>        {            //the wave stream-like reader            using (var reader = new WaveFileReader("startup.wav"))            {                var leftChannel = new List<float>();              var rightChannel = new List<float>();                  //let's read all frames as normalized floats                while (reader.Position < reader.Length)                {                    var frame = reader.ReadNextSampleFrame();                   leftChannel.Add(frame[0]);                    rightChannel.Add(frame[1]);                }                  return new                {                    Left = leftChannel.ToArray(),                    Right = rightChannel.ToArray(),                };            }        });      //make a low-pass digital filter on floating point data    //at 200hz    var leftLowpassTask = Task.Factory.StartNew(() => LowPass(channels.Left, 200).ToArray());    var rightLowpassTask = Task.Factory.StartNew(() => LowPass(channels.Right, 200).ToArray());      //this let the two tasks work together in task-parallelism    var leftChannelLP = await leftLowpassTask;    var rightChannelLP = await rightLowpassTask;      //create and databind a chart    var chart1 = CreateChart();      chart1.DataSource = Enumerable.Range(0, channels.Left.Length).Select(i => new        {            Index = i,            Left = channels.Left[i],            Right = channels.Right[i],            LeftLP = leftChannelLP[i],            RightLP = rightChannelLP[i],        }).ToArray();      chart1.DataBind();      //add the chart to the form    this.Controls.Add(chart1); }   private static Chart CreateChart() {    //creates a chart    //namespace System.Windows.Forms.DataVisualization.Charting      var chart1 = new Chart();      //shows chart in fullscreen    chart1.Dock = DockStyle.Fill;      //create a default area    chart1.ChartAreas.Add(new ChartArea());      //left and right channel series    chart1.Series.Add(new Series    {        XValueMember = "Index",        XValueType = ChartValueType.Auto,        YValueMembers = "Left",        ChartType = SeriesChartType.Line,    });    chart1.Series.Add(new Series    {        XValueMember = "Index",        XValueType = ChartValueType.Auto,        YValueMembers = "Right",        ChartType = SeriesChartType.Line,    });      //left and right channel low-pass (bass) series    chart1.Series.Add(new Series    {        XValueMember = "Index",        XValueType = ChartValueType.Auto,        YValueMembers = "LeftLP",        ChartType = SeriesChartType.Line,        BorderWidth = 2,    });    chart1.Series.Add(new Series    {        XValueMember = "Index",        XValueType = ChartValueType.Auto,        YValueMembers = "RightLP",        ChartType = SeriesChartType.Line,        BorderWidth = 2,    });      return chart1; } Let's see the graphical result: The Windows start-up sound waveform. In bolt, the bass waveform with a low-pass filter at 200hz. The usage of parallelism in elaborations such as this is mandatory. Audio elaboration is a canonical example of engineering data computation because it works on a huge dataset of floating points values. A simple file, such as the preceding one that contains less than 2 seconds of audio sampled at (only) 22,050 Hz, produces an array greater than 40,000 floating points per channel (stereo = 2 channels). Just to have an idea of how hard processing audio files is, note that an uncompressed CD quality song of 4 minutes sampled at 44,100 samples per second * 60 (seconds) * 4 (minutes) will create an array greater than 10 million floating-point items per channel. Because of the FFT intrinsic logic, any low-pass filtering run must run in a single thread. This means that the only optimization we can apply when running FFT based low-pass filtering is parallelizing in a per channel basis. For most cases, this choice can only bring a 2X throughput improvement, regardless of the processor count of the underlying system. Summary In this article we got introduced to the applications of .NET high-performance performance. We learned how MVVM and XAML play their roles in .NET to create applications for various platforms, also we learned about its performance characteristics. Next we learned how high-performance .NET had applications in engineering aspects through a practical example of low-pass audio filtering. It showed you how versatile it is to apply high-performance programming to specific engineering applications. Resources for Article: Further resources on this subject: Windows Phone 8 Applications [article] Core .NET Recipes [article] Parallel Programming Patterns [article]

Historically, a conventional Ruby on Rails application leverages server-side business logic, a relational database, and a RESTful architecture to serve dynamically-generated HTML. However, JavaScript-intensive applications and widespread use of external web APIs somewhat challenge this architecture. In many cases, Rails is tasked with performing as an orchestration layer, collecting data from various backend services and serving re-formatted JSON or XML to clients. In such instances, how is Rails' model-view-controller architecture still relevant? In the second part of this series, we'll continue with the creation of Noterizer that we started on in Part 1, creating a simple Rails backend that makes requests to an external XML-based web service and serves JSON. We'll use RSpec for tests and Jbuilder for view rendering. Building out the Noterizer backend Currently, NotesController serves an empty JSON document, yet the goal is to serve a JSON array representing the XML data served by the aforementioned NotesXmlService endpoints: http://NotesXmlService.herokuapp.com/note-onehttp://NotesXmlService.herokuapp.com/note-two Creating a Note model First, create a model to represent the note data returned by each of the NotesXmlService endpoints. In a more traditional Rails application, such a model represents data from a database. In Noterizer, the Note model provides a Ruby interface to each NotesXmlService XML document. Create a Note model: $ touch app/models/note.rb On initialization, Note should perform an HTTP request to the URL it's passed on instantiation and expose relevant XML values via its public methods. The Note class will use Nokogiri to parse the XML. Require the nokogiri gem by adding the following to your Gemfile: gem 'nokogiri' Install nokogiri: $ bundle install Let’s also store the NotesXmlService base URL in a config property. This allows us to easily change its value throughout the application should it ever change. Add the following to config/application.rb: config.notes_xml_service_root = ‘http://notesxmlservice.herokuapp.com’ Add the following to app/models/note.rb: require 'nokogiri'class Notedef initialize(path = nil)   @uri = URI.parse(“#{Rails.config.notes_xml_service_root}/#{path}”)   @xml = get_and_parse_response    create_methodsend private def get_and_parse_response   Nokogiri::XML(get_response)end def get_response   http = Net::HTTP.new(@uri.host, @uri.port)    http.request(request).bodyend def request   Net::HTTP::Get.new(@uri.request_uri)end def create_methods   available_methods.each do |method|     self.class.send(:define_method, method) { fetch_value method.to_s }   endend def available_methods   [     :to,     :from,     :heading,     :body   ]end def fetch_value(value)   @xml.xpath("//add[@key='#{value}']/@value").textendend ``` The Note class now works like this: It performs an HTTP request to the URL it's passed on initialization. It leverages Nokogiri to parse the resulting XML. It uses Nokogiri's support of XPATH expressions to dynamically create to, from, heading, and body methods based on the corresponding values in the XML. Testing the Note model Let's test Note by creating a corresponding model spec file: $ rails g rspec:model note First, test the Note#to method by adding the following to the newly created spec/models/note_spec.rb: require 'rails_helper'RSpec.describe Note, :type => :model dobefore do   @note = Note.new(‘note-one')end describe '#to' do   it 'returns the correct "to" value from the XML' do     expect(@note.to).to eq 'Samantha'   endendend Running rake spec reveals that the test passes, though it performs a real HTTP request. This is not ideal: such requests generate unwelcome traffic on NotesServiceDmo, makes hard-coded assumptions about the XML returned by the /note-one endpoint, and relies upon an Internet connection to pass. Let's configure RSpec to use WebMock to fake HTTP requests. Add webmock to the Gemfile; specify that it’s part of the :test group: gem 'webmock', group: :test Install webmock: $ bundle install Add the following to spec/spec_helper.rb’s RSpec.configure block to disable network requests during RSpec runs: require 'webmock/rspec'RSpec.configure do |config|WebMock.disable_net_connect!end Use WebMock to stub the NotesXmlService request/response in spec/models/note_spec.rb by editing its before block to the following: before :each dopath = 'note-one'stub_request(:get, ”#{Rails.application.config.notes_xml_service_root}/#{path}”).to_return(   body: [     '<?xml version="1.0" encoding="UTF-8"?>',     '<note type="work">',       '<add key="to" value="Samantha"/>',       '<add key="from" value="David"/>',       '<add key="heading" value="Our Meeting"/>',       '<add key="body" value="Are you available to get started at 1pm?"/>',     '</note>'   ].join(''))@note = Note.new(path)end Running rake spec should now run the full test suite, including the Note model spec, without performing real HTTP requests. Similar tests can be authored for Note's from, heading, and body methods. The complete examples are viewable in Noterizer's master branch. Building up the controller Now that we have a Note model, our Notes#index controller should create an instance variable, inside of which lives an array of Note models representing the data served by each NotesXmlService endpoint. Add the following to app/controllers/notes_controller.rb's index method: def index@notes = [   Note.new(‘note-one’),   Note.new(‘note-two’)]end Testing the NotesController Let's test the modifications to NotesController. Creating a spec helper To test the controller, we'll need to stub the NotesXmlService requests, just as was done in spec/models/note_spec.rb. Rather than repeat the stub_request code, let's abstract it into a helper that can be used throughout the specs. Create a spec/support/helpers.rb file: $ mkdir spec/support && touch spec/support/helpers.rb Define the helper method by adding the following to the newly created spec/support/helpers.rb file: module Helpersdef stub_note_request(path)   base_url = Rails.application.config.notes_xml_service_root   stub_request(:get, "#{base_url}/#{path}").to_return(     body: [       '<?xml version="1.0" encoding="UTF-8"?>',        '<note type="work">',         '<add key="to" value="Samantha"/>',         '<add key="from" value="David"/>',         '<add key="heading" value="Our Meeting"/>',         '<add key="body" value="Are you available to get started at 1pm?"/>',       '</note>'     ].join('')   )endend Tweak the RSpec configuration such that it can be used by adding the following to the configure block in spec/rails_helper.rb: config.include Helpers Edit the spec/models/note_spec.rb's before block to use the #stub_note_request helper: before :each dopath = ‘note-one’stub_note_request(path)@note = Note.new(path)end Confirm that all tests continue passing by running rake spec. Writing the new NotesController tests Let's make use of the stub_note_request helper by changing spec/controllers/notes_controller_spec.rb's before block to the following: before :each dostub_note_request('note-one')stub_note_request('note-two')get :indexend Add the following to its #index tests to test the new functionality: context 'the @notes it assigns' doit 'is an array containing 2 items' do   expect(assigns(:notes).length).to eq 2endit 'is an array of Note models' do   assigns(:notes).each do |note|    expect(note).to be_a Note   endendend rake spec should now confirm that all tests pass. See Noterizer’s master branch for the complete example code. The Jbuilder view With a fully functional Note model and NotesController, Noterizer now needs a view to render the proper JSON. Much like ERB offers a Ruby HTML templating solution, Jbuilder offers a JSON templating solution. Writing the Jbuilder view templates Add the following to app/views/notes/index.json.jbuilder: json.array! @notes, partial: 'note' as: :note What does this do? This instructs Jbuilder to build an array of objects with the @notes array and render each one via a partial named _note. Create the app/views/notes/_note.json.jbuilder partial file: $ touch app/views/notes/_note.json.jbuilder Add the following: json.toField     note.tojson.fromField note.fromjson.heading   note.headingjson.body         note.body Now, http://localhost:3000/notes renders the following JSON: [{"toField": "Samantha","fromField": "David","heading": "Our Meeting","body": "Are you available to get started at 1pm?"},{"toField": "Melissa","fromField": "Chris","heading": "Saturday","body": "Are you still interested in going to the beach?"}] Testing the Jbuilder view templates First, let's test the app/views/notes/_note.json.jbuilder template. Create a spec file: $ mkdir spec/views/notes && touch spec/views/notes/_note.json.jbuilder_spec.rb Add the following to the newly created _note spec: require 'spec_helper'describe 'notes/_note' dolet(:note) do   double('Note',     to: 'Mike',     from: 'Sam',     heading: 'Tomorrow',     body: 'Call me after 3pm.',   )endbefore :each do   assign(:note, note)   render '/notes/note', note: noteendcontext 'verifying the JSON values it renders' do   subject { JSON.parse(rendered) }   describe "['toField']" do     subject { super()['toField'] }     it { is_expected.to eq 'Mike' }   endendend rake spec should confirm that all tests pass. You can write additional _note tests for fromField, heading, and body. Conclusion You have now completed this two part blog series, and you’ve built Noterizer, a basic example of a Rails application that fronts an external API. The application preserves an MVC architecture that separates concerns across the Note model, the NotesController, and JSON view templates. Noterizer also offers a simple example of using RSpec to test such an application and Jbuilder as a JSON templating language. Mike Ball is a Philadelphia-based software developer specializing in Ruby on Rails and JavaScript. He works for Comcast Interactive Media where he helps build web-based TV and video consumption applications.

Client API implementations for Sphinx Sphinx comes with a number of native searchd client API implementations. Some third-party open source implementations for Perl, Ruby, and C++ are also available. All APIs provide the same set of methods and they implement the same network protocol. As a result, they more or less all work in a similar fashion, they all work in a similar fashion. All examples in this article are for PHP implementation of the Sphinx API. However, you can just as easily use other programming languages. Sphinx is used with PHP more widely than any other language. Search using client API Let's see how we can use native PHP implementation of Sphinx API to search. We will add a configuration related to searchd and then create a PHP file to search the index using the Sphinx client API implementation for PHP. Time for action – creating a basic search script Add the searchd config section to /usr/local/sphinx/etc/sphinx-blog.conf: source blog { # source options } index posts { # index options } indexer { # indexer options } # searchd options (used by search daemon) searchd { listen = 9312 log = /usr/local/sphinx/var/log/searchd.log query_log = /usr/local/sphinx/var/log/query.log max_children = 30 pid_file = /usr/local/sphinx/var/log/searchd.pid } Start the searchd daemon (as root user): $ sudo /usr/local/sphinx/bin/searchd -c /usr/local/sphinx/etc/ sphinx-blog.conf Copy the sphinxapi.php file (the class with PHP implementation of Sphinx API) from the sphinx source directory to your working directory: $ mkdir /path/to/your/webroot/sphinx $ cd /path/to/your/webroot/sphinx $ cp /path/to/sphinx-0.9.9/api/sphinxapi.php ./ Create a simple_search.php script that uses the PHP client API class to search the Sphinx-blog index, and execute it in the browser: <?php require_once('sphinxapi.php'); // Instantiate the sphinx client $client = new SphinxClient(); // Set search options $client->SetServer('localhost', 9312); $client->SetConnectTimeout(1); $client->SetArrayResult(true); // Query the index $results = $client->Query('php'); // Output the matched results in raw format print_r($results['matches']); The output of the given code, as seen in a browser, will be similar to what's shown in the following screenshot: What just happened? Firstly, we added the searchd configuration section to our sphinx-blog.conf file. The following options were added to searchd section: listen: This options specifies the IP address and port that searchd will listen on. It can also specify the Unix-domain socket path. This options was introduced in v0.9.9 and should be used instead of the port (deprecated) option. If the port part is omitted, then the default port used is 9312.Examples: listen = localhost listen = 9312 listen = localhost:9898 listen = 192.168.1.25:4000 listen = /var/run/sphinx.s log: Name of the file where all searchd runtime events will be logged. This is an optional setting and the default value is "searchd.log". query_log: Name of the file where all search queries will be logged. This is an optional setting and the default value is empty, that is, do not log queries. max_children: The maximum number of concurrent searches to run in parallel. This is an optional setting and the default value is 0 (unlimited). pid_file: Filename of the searchd process ID. This is a mandatory setting. The file is created on startup and it contains the head daemon process ID while the daemon is running. The pid_file becomes unlinked when the daemon is stopped. Once we were done with adding searchd configuration options, we started the searchd daemon with root user. We passed the path of the configuration file as an argument to searchd. The default configuration file used is /usr/local/sphinx/etc/sphinx.conf. After a successful startup, searchd listens on all network interfaces, including all the configured network cards on the server, at port 9312. If we want searchd to listen on a specific interface then we can specify the hostname or IP address in the value of the listen option: listen = 192.168.1.25:9312 The listen setting defined in the configuration file can be overridden in the command line while starting searchd by using the -l command line argument. There are other (optional) arguments that can be passed to searchd as seen in the following screenshot: searchd needs to be running all the time when we are using the client API. The first thing you should always check is whether searchd is running or not, and start it if it is not running. We then created a PHP script to search the sphinx-blog index. To search the Sphinx index, we need to use the Sphinx client API. As we are working with a PHP script, we copied the PHP client implementation class, (sphinxapi.php) which comes along with Sphinx source, to our working directory so that we can include it in our script. However, you can keep this file anywhere on the file system as long as you can include it in your PHP script. Throughout this article we will be using /path/to/webroot/sphinx as the working directory and we will create all PHP scripts in that directory. We will refer to this directory simply as webroot. We initialized the SphinxClient class and then used the following class methods to set upthe Sphinx client API: SphinxClient::SetServer($host, $port)—This method sets the searchd hostname and port. All subsequent requests use these settings unless this method is called again with some different parameters. The default host is localhost and port is 9312. SphinxClient::SetConnectTimeout($timeout)—This is the maximum time allowed to spend trying to connect to the server before giving up. SphinxClient::SetArrayResult($arrayresult)—This is a PHP client APIspecific method. It specifies whether the matches should be returned as an array or a hash. The Default value is false, which means that matches will be returned in a PHP hash format, where document IDs will be the keys, and other information (attributes, weight) will be the values. If $arrayresult is true, then the matches will be returned in plain arrays with complete per-match information. After that, the actual querying of index was pretty straightforward using the SphinxClient::Query($query) method. It returned an array with matched results, as well as other information such as error, fields in index, attributes in index, total records found, time taken for search, and so on. The actual results are in the $results['matches'] variable. We can run a loop on the results, and it is a straightforward job to get the actual document's content from the document ID and display it. Matching modes When a full-text search is performed on the Sphinx index, different matching modes can be used by Sphinx to find the results. The following matching modes are supported by Sphinx: SPH_MATCH_ALL—This is the default mode and it matches all query words, that is, only records that match all of the queried words will be returned. SPH_MATCH_ANY—This matches any of the query words. SPH_MATCH_PHRASE—This matches query as a phrase and requires a perfect match. SPH_MATCH_BOOLEAN—This matches query as a Boolean expression. SPH_MATCH_EXTENDED—This matches query as an expression in Sphinx internal query language. SPH_MATCH_EXTENDED2—This matches query using the second version of Extended matching mode. This supersedes SPH_MATCH_EXTENDED as of v0.9.9. SPH_MATCH_FULLSCAN—In this mode the query terms are ignored and no text-matching is done, but filters and grouping are still applied. Time for action – searching with different matching modes Create a PHP script display_results.php in your webroot with the following code: <?php // Database connection credentials $dsn ='mysql:dbname=myblog;host=localhost'; $user = 'root'; $pass = ''; // Instantiate the PDO (PHP 5 specific) class try { $dbh = new PDO($dsn, $user, $pass); } catch (PDOException $e){ echo'Connection failed: '.$e->getMessage(); } // PDO statement to fetch the post data $query = "SELECT p.*, a.name FROM posts AS p " . "LEFT JOIN authors AS a ON p.author_id = a.id " . "WHERE p.id = :post_id"; $post_stmt = $dbh->prepare($query); // PDO statement to fetch the post's categories $query = "SELECT c.name FROM posts_categories AS pc ". "LEFT JOIN categories AS c ON pc.category_id = c.id " . "WHERE pc.post_id = :post_id"; $cat_stmt = $dbh->prepare($query); // Function to display the results in a nice format function display_results($results, $message = null) { global $post_stmt, $cat_stmt; if ($message) { print "<h3>$message</h3>"; } if (!isset($results['matches'])) { print "No results found<hr />"; return; } foreach ($results['matches'] as $result) { // Get the data for this document (post) from db $post_stmt->bindParam(':post_id', $result['id'], PDO::PARAM_INT); $post_stmt->execute(); $post = $post_stmt->fetch(PDO::FETCH_ASSOC); // Get the categories of this post $cat_stmt->bindParam(':post_id', $result['id'], PDO::PARAM_INT); $cat_stmt->execute(); $categories = $cat_stmt->fetchAll(PDO::FETCH_ASSOC); // Output title, author and categories print "Id: {$posmt['id']}<br />" . "Title: {$post['title']}<br />" . "Author: {$post['name']}"; $cats = array(); foreach ($categories as $category) { $cats[] = $category['name']; } if (count($cats)) { print "<br />Categories: " . implode(', ', $cats); } print "<hr />"; } } Create a PHP script search_matching_modes.php in your webroot with the following code: <?php // Include the api class Require('sphinxapi.php'); // Include the file which contains the function to display results require_once('display_results.php'); $client = new SphinxClient(); // Set search options $client->SetServer('localhost', 9312); $client->SetConnectTimeout(1); $client->SetArrayResult(true); // SPH_MATCH_ALL mode will be used by default // and we need not set it explicitly display_results( $client->Query('php'), '"php" with SPH_MATCH_ALL'); display_results( $client->Query('programming'), '"programming" with SPH_MATCH_ALL'); display_results( $client->Query('php programming'), '"php programming" with SPH_MATCH_ALL'); // Set the mode to SPH_MATCH_ANY $client->SetMatchMode(SPH_MATCH_ANY); display_results( $client->Query('php programming'), '"php programming" with SPH_MATCH_ANY'); // Set the mode to SPH_MATCH_PHRASE $client->SetMatchMode(SPH_MATCH_PHRASE); display_results( $client->Query('php programming'), '"php programming" with SPH_MATCH_PHRASE'); display_results( $client->Query('scripting language'), '"scripting language" with SPH_MATCH_PHRASE'); // Set the mode to SPH_MATCH_FULLSCAN $client->SetMatchMode(SPH_MATCH_FULLSCAN); display_results( $client->Query('php'), '"php programming" with SPH_MATCH_FULLSCAN'); Execute search_matching_modes.php in a browser (http://localhost/sphinx/search_matching_modes.php).

  (For more resources on jQuery, see here.)   Each container has a heading element associated with it that is used to open the container and display the content. When you click on a heading, its content will slide into view (with an animation) below it. The currently visible content is hidden, while the new content is shown whenever we click on an accordion heading. The accordion widget is a robust and highly configurable widget that allows you to save space on your web pages, by displaying only a single panel of content at any time. The following screenshot shows an example of an accordion widget: It's easy for our visitors to use and easy for us to implement. It has a range of configurable options that can be used to customize its appearance and behavior, and exposes a series of methods that allow you to control it programmatically. It also comes with a rich set of interaction events that we can use to hook into key interactions between our visitors and the widget. The height of the accordion's container element will be set automatically, so that there is room to show the tallest content panel in addition to the headers. Also, by default, the size of the widget will remain fixed, so that it won't push other elements on the page around it out of the way, when content panels open or close.   Accordion's structure Let's take a moment to familiarize ourselves with the underlying markup that an accordion is made of. Within the outer container is a series of links. These links are the headings within the accordion and each heading will have a corresponding content panel that opens when the header is clicked. It's worth remembering that only one content panel can be open at a time when using the accordion widget. In a blank page in your text editor, create the following page: <!DOCTYPE html> <html> <head> <meta charset="utf-8"> <link rel="stylesheet" href="css/smoothness/jquery-ui-1.8.9.custom.css"> <title>Accordion </title> </head> <body> <div id="myAccordion"> <h2> <a href="#">Header 1 </a> </h2> <div>Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean sollicitudin. Sed interdum pulvinar justo. Nam iaculis volutpat ligula. Integer vitae felis quis diam laoreet ullamcorper. </div> <h2> <a href="#">Header 2 </a> </h2> <div>Etiam tincidunt est vitae est. Ut posuere, mauris at sodales rutrum, turpis tellus fermentum metus, ut bibendum velit enim eu lectus. Suspendisse potenti. </div> <h2> <a href="#">Header 3 </a> </h2> <div>Donec at dolor ac metus pharetra aliquam. Suspendisse purus. Fusce tempor ultrices libero. Sed quis nunc. Pellentesque tincidunt viverra felis. Integer elit mauris, egestas ultricies, gravida vitae, feugiat a, tellus. </div> </div> <script src="development-bundle/jquery-1.3.2.js"> </script> <script src="development-bundle/ui/ui.core.js"> </script> <script src="development-bundle/ui/ui.accordion.js"> </script> <script> $(function() { $("#myAccordion").accordion(); }); </script> </body> </html> Save the file as accordion1.html in the jqueryui folder, and try it out in a browser. The widget should appear as it did in the screenshot at the start of the article, fully skinned and ready for action. The following list shows the required dependencies of the widget: jquery.ui.all.css jquery-x.x.x.js jquery.ui.core.js jquery.ui.accordion.js Each widget also has its own source file, and may depend on other components as well. For example, we could include some effect files to use non-standard animations on the opening accordion panels. The order in which these files appear is important. The jQuery library must always appear first, followed by the jquery.ui.core.js file. After these files, any other files that the widget depends upon, should appear before the widget's own script file. The library components will not function as expected if files are not loaded in the correct order. The underlying markup required for the accordion is flexible, and the widget can be constructed from a variety of different structures. In this example, the accordion headings are formed from links wrapped in <h2> elements, and the content panels are simple <div> elements. For the accordion to function correctly, each content panel should appear directly after its corresponding header. All of the elements for the widget are enclosed within a container <div> that is targeted with the accordion() widget method. After the required script dependencies from the library, we use a custom <script> block to transform the underlying markup into the accordion. To initialize the widget, we use a simple id selector $("#myAccordion"), to specify the element that contains the markup for the widget, and then chain the accordion() widget method after the selector to create the accordion. In this example, we used an empty fragment (#) as the value of the href attribute in our tab heading elements, such as: <h2><a href="#">Header 1</a></h2> You should note that any URL supplied for accordion headers will not be followed, when the header is clicked in the default implementation. Similar to the tabs widget, the underlying markup that is transformed into the accordion has a series of classnames added to it, when the widget is initialized. A number of different elements that make up the widget are given role and aria- attributes. ARIA stands for Accessible Rich Internet Applications and is a W3C recommendation for ensuring that rich-internet applications remain accessible to assisted technologies. The accordion panels that are initially hidden from view are given the attribute aria-expanded="false" to ensure that screen readers don't discard or cannot access content that is hidden using display:none. This makes the accordion widget highly accessible.   Styling the accordion ThemeRoller is the recommended tool for choosing or creating the theme of the accordion widget, but there may be times when we want to considerably change the look and style of the widget beyond what is possible with ThemeRoller. In that case, we can just style our own accordion. In a new page in your text editor add the following code: #myAccordion { width:400px; border:1px solid #636363; padding-bottom:1px; } .ui-accordion-header { border:1px solid #fff; font-family:Georgia; background:#e2e2e2 none; } .ui-widget-content { font-size:70%; border:none; } .ui-corner-bottom { -moz-border-radius:4px 4px 0 0; -webkit-border-radius:4px 4px 0 0; border-radius:4px 4px 0 0; } .ui-corner-all { -moz-border-radius:0; -webkit-border-radius:0; border-radius:0; } .ui-accordion .ui-accordion-header { margin:0 0 -1px; } #myAccordion .ui-state-active, #myAccordion .ui-widget-content .ui-state-active { background:#fff; } .ui-state-hover, .ui-widget-content .ui-state-hover { background:#d2d2d2; } Save this file as accordionTheme.css in the css folder, and link to it after the jQuery UI style sheet in the <head> of accordion1.html: <link rel="stylesheet" href="css/accordionTheme.css"> Save the new file as accordion2.html in the jqueryui folder and view it in a browser. It should appear something like this: As you can see from this screenshot, we've disabled the built-in rounded corners that are added by the theme file and have set alternative fonts, background colors, and border colors. We haven't changed the widget much, but we haven't used many style rules. It would be easy to continue overriding rules in this way to build a much more complex custom theme.   Configuring an accordion The accordion has a range of configurable options that allow us to change the default behavior of the widget. The following table lists the available options, their default values, and gives a brief description of their usage: Option Default value Used to... active first child (the first panel is open) Set the active heading on page load. animated slide Animate the opening of content panels. autoHeight true Automatically set height according to the biggest drawer. clearStyle false Clear the height and overflow styles after a panel opens. collapsible false Allow all of the content panels to be closed at the same time. disabled false Disable the widget. event click Specify the event on headers that trigger drawers to open. fillSpace false Allow the accordion to fill the height of its container instead of sizing itself according to the content within it. header > li >:first-child,>:not(li):even Specify the selector for header elements. Although it looks complex, this is a standard jQuery selector that simply targets the first-child within every odd <li> element. icons 'header': 'ui-icon-triangle-1-e', 'headerSelected': 'ui-icon-triangle-1-s' Set the icons for the header elements and the selected state. navigation false Enable navigation mode for accordion. navigationFilter location.href Change the function used to obtain the ID of the content panel that should be open when the widget is initialized. Changing the trigger event Most of the options are self-explanatory, and the values they accept are usually Booleans, strings, or element selectors. Let's put some of them to use, so that we can explore their functionality. Change the final <script> element in accordion2.html so that it appears as follows: <script> (function($) { var accOpts = { event:"mouseover" } $("#myAccordion").accordion(accOpts); })(jQuery); </script> Save these changes as accordion3.html. First, we create a new object literal called accOpts that contains the key event and the value mouseover, which is the event we wish to use to trigger the opening of an accordion panel. We pass this object to the accordion() method as an argument and it overrides the default option of the widget, which is click. The mouseover event is commonly used as an alternative trigger event. Other events can also be used, for example, we can set keydown as the event, but in order for this to work, the accordion panel that we wish to open, must already be focused. You should note that you can also set options using an inline object within the widget method, without creating a separate object. Using the following code would be equally as effective, and would often be the preferred way of coding: <script> (function($) { $("#myAccordion").accordion({ event:"mouseover" }); })(jQuery); </script>  

In today's world, many server-side applications make use of XML to structure data because XML is a standard way of representing structured information. It is easy to work with, and people can easily read, write, and understand XML without the need of any specialized skills. The XML standard is widely accepted and used in server communications such as Simple Object Access Protocol (SOAP) based web services. XML stands for eXtensible Markup Language. The XML standard specification is available at http://www.w3.org/XML/. Adobe Flex provides a standardized ECMAScript-based set of API classes and functionality for working with XML data. This collection of classes and functionality provided by Flex are known as E4X. You can use these classes provided by Flex to build sophisticated Rich Internet Applications using XML data. XML basics XML is a standard way to represent categorized data into a tree structure similar to HTML documents. XML is written in plain-text format, and hence it is very easy to read, write, and manipulate its data. A typical XML document looks like this: <book>    <title>Flex 3 with Java</title>    <author>Satish Kore</author>    <publisher>Packt Publishing</publisher>    <pages>300</pages> </book> Generally, XML data is known as XML documents and it is represented by tags wrapped in angle brackets (< >). These tags are also known as XML elements. Every XML document starts with a single top-level element known as the root element. Each element is distinguished by a set of tags known as the opening tag and the closing tag. In the previous XML document, <book> is the opening tag and </book> is the closing tag. If an element contains no content, it can be written as an empty statement (also called self-closing statement). For example, <book/> is as good as writing <book></book>. XML documents can also be more complex with nested tags and attributes, as shown in the following example: <book ISBN="978-1-847195-34-0">   <title>Flex 3 with Java</title>   <author country="India" numberOfBooks="1">    <firstName>Satish</firstName>    <lastName>Kore</lastName> </author>   <publisher country="United Kingdom">Packt Publishing</publisher>   <pages>300</pages> </book> Notice that the above XML document contains nested tags such as <firstName> and <lastName> under the <author> tag. ISBN, country, and numberOfBooks, which you can see inside the tags, are called XML attributes. To learn more about XML, visit the W3Schools' XML Tutorial at http://w3schools.com/xml/. Understanding E4X Flex provides a set of API classes and functionality based on the ECMAScript for XML (E4X) standards in order to work with XML data. The E4X approach provides a simple and straightforward way to work with XML structured data, and it also reduces the complexity of parsing XML documents. Earlier versions of Flex did not have a direct way of working with XML data. The E4X provides an alternative to DOM (Document Object Model) interface that uses a simpler syntax for reading and querying XML documents. More information about other E4X implementations can be found at http://en.wikipedia.org/wiki/E4X. The key features of E4X include: It is based on standard scripting language specifications known as ECMAScript for XML. Flex implements these specifications in the form of API classes and functionality for simplifying the XML data processing. It provides easy and well-known operators, such as the dot (.) and @, to work with XML objects. The @ and dot (.) operators can be used not only to read data, but also to assign data to XML nodes, attributes, and so on. The E4X functionality is much easier and more intuitive than working with the DOM documents to access XML data. ActionScript 3.0 includes the following E4X classes: XML, XMLList, QName, and Namespace. These classes are designed to simplify XML data processing into Flex applications. Let's see one quick example: Define a variable of type XML and create a sample XML document. In this example, we will assign it as a literal. However, in the real world, your application might load XML data from external sources, such as a web service or an RSS feed. private var myBooks:XML =   <books publisher="Packt Pub">    <book title="Book1" price="99.99">    <author>Author1</author>    </book>    <book title="Book2" price="59.99">    <author>Author2</author>    </book>    <book title="Book3" price="49.99">    <author>Author3</author>    </book> </books>; Now, we will see some of the E4X approaches to read and parse the above XML in our application. The E4X uses many operators to simplify accessing XML nodes and attributes, such as dot (.) and attribute identifier (@), for accessing properties and attributes. private function traceXML():void {    trace(myBooks.book.(@price < 50.99).@title); //Output: Book3    trace(myBooks.book[1].author); //Output: Author2    trace(myBooks.@publisher); //Output: Packt Pub    //Following for loop outputs prices of all books    for each(var price in myBooks..@price) {    trace(price);    } } In the code above, we are using a conditional expression to extract the title of the book(s) whose price is set below 50.99$ in the first trace statement. If we have to do this manually, imagine how much code would have been needed to parse the XML. In the second trace, we are accessing a book node using index and printing its author node's value. And in the third trace, we are simply printing the root node's publisher attribute value and finally, we are using a for loop to traverse through prices of all the books and printing each price. The following is a list of XML operators: Operator Name Description    @   attribute identifier Identifies attributes of an XML or XMLList object.     { }     braces(XML) Evaluates an expression that is used in an XML or XMLList initializer.   [ ]     brackets(XML) Accesses a property or attribute of an XML or XMLList object, for example myBooks.book["@title"].     + concatenation(XMLList) Concatenates (combines) XML or XMLList values into an XMLList object.     += concatenation assignment (XMLList) Assigns expression1 The XML object An XML class represents an XML element, attribute, comment, processing instruction, or a text element. We have used the XML class in our example above to initialize the myBooks variable with an XML literal. The XML class is included into an ActionScript 3.0 core class, so you don't need to import a package to use it. The XML class provides many properties and methods to simplify XML processing, such as ignoreWhitespace and ignoreComments properties, used for ignoring whitespaces and comments in XML documents respectively. You can use the prependChild() and appendChild() methods to prepend and append XML nodes to existing XML documents. Methods such as toString() and toXMLString() allow you to convert XML to a string. An example of an XML object: private var myBooks:XML = <books publisher="Packt Pub"> <book title="Book1" price="99.99"> <author>Author1</author> </book> <book title="Book2" price="120.00"> <author>Author2</author> </book> </books>;   In the above example, we have created an XML object by assigning an XML literal to it. You can also create an XML object from a string that contains XML data, as shown in the following example: private var str:String = "<books publisher="Packt Pub"> <book title="Book1" price="99.99"> <author>Author1</author> </book> <book title="Book2" price="59.99"> <author>Author2</author> </book> </books>"; private var myBooks:XML = new XML(str); trace(myBooks.toXMLString()); //outputs formatted xml as string If the XML data in string is not well-formed (for example, a closing tag is missing), then you will see a runtime error. You can also use binding expressions in the XML text to extract contents from a variable data. For example, you could bind a node's name attribute to a variable value, as in the following line: private var title:String = "Book1" var aBook:XML = <book title="{title}">; To read more about XML class methods and properties, go through Flex 3 LiveDocs at http://livedocs.adobe.com/flex/3/langref/XML.html. The XMLList object As the class name indicates, XMLList contains one or more XML objects. It can contain full XML documents, XML fragments, or the results of an XML query. You can typically use all of the XML class's methods and properties on the objects from XMLList. To access these objects from the XMLList collection, iterate over it using a for each… statement. The XMLList provides you with the following methods to work with its objects: child(): Returns a specified child of every XML object children(): Returns specified children of every XML object descendants(): Returns all descendants of an XML object elements(): Calls the elements() method of each XML object in the XMLList. Returns all elements of the XML object parent(): Returns the parent of the XMLList object if all items in the XMLList object have the same parent attribute(attributeName): Calls the attribute() method of each XML object and returns an XMLList object of the results. The results match the given attributeName parameter attributes(): Calls the attributes() method of each XML object and returns an XMLList object of attributes for each XML object contains(): Checks if the specified XML object is present in the XMLList copy(): Returns a copy of the given XMLList object length(): Returns the number of properties in the XMLList object valueOf(): Returns the XMLList object For details on these methods, see the ActionScript 3.0 Language Reference. Let's return to the example of the XMLList: var xmlList:XMLList = myBooks.book.(@price == 99.99); var item:XML; for each(item in xmlList) { trace("item:"+item.toXMLString()); } Output: item:<book title="Book1" price="99.99"> <author>Author1</author> </book> In the example above, we have used XMLList to store the result of the myBooks.book.(@price == 99.99); statement. This statement returns an XMLList containing XML node(s) whose price is 99.99$. Working with XML objects The XML class provides many useful methods to work with XML objects, such as the appendChild() and prependChild() methods to add an XML element to the beginning or end of an XML object, as shown in the following example: var node1:XML = <middleInitial>B</middleInitial> var node2:XML = <lastName>Kore</lastName> var root:XML = <personalInfo></personalInfo> root = root.appendChild(node1); root = root.appendChild(node2); root = root.prependChild(<firstName>Satish</firstName>); The output is as follows: <personalInfo> <firstName>Satish</firstName> <middleInitial>B</middleInitial> <lastName>Kore</lastName> </personalInfo> You can use the insertChildBefore() or insertChildAfter() method to add a property before or after a specified property, as shown in the following example: var x:XML = <count> <one>1</one> <three>3</three> <four>4</four> </count>; x = x.insertChildBefore(x.three, "<two>2</two>"); x = x.insertChildAfter(x.four, "<five>5</five>"); trace(x.toXMLString()); The output of the above code is as follows: <count> <one>1</one> <two>2</two> <three>3</three> <four>4</four> <five>5</five> </count>

Basic Dojo layout facts The Layout widgets in Dojo are varied in nature, but their most common use is as 'windows' or areas which organize and present other widgets or information. Several use the same kind of child elements the ContentPane. The ContentPane is a widget which can contain other widgets and plain HTML, reload content using Ajax and so on. The ContentPane can also be used stand-alone in a page, but is more usable inside a layout container of some sort. And what is a layout container? Well, it's a widget which contains ContentPanes, of course. A layout container can often contain other widgets as well, but most containers work very well with a different configuration of ContentPanes, which properly insulates the further contents. Take the TabContainer, for example. It is used to organize two or more ContentPanes, where each gets its own tab. When a user clicks on one of the tabs, the ContentPane inside it is shown and all others are hidden. Using BorderManager can bring the necessary CSS styling down to a minimum, while giving a simple interface for managing dynamic changes of child widgets and elements. ContentPane A ContentPane can look like anything of course, so it doesn't really help putting a screen-dump of one on the page. However, the interface is very good to know. The following arguments are detected by ContentPane and can be used when creating one either programmatically or by markup: // href: String //The href of the content that displays now. //Set this at construction if you want to load data externally //when the pane is shown.(Set preload=true to load it immediately.) //Changing href after creation doesn't have any effect; //see setHref(); href: "", //extractContent: Boolean //Extract visible content from inside of <body> .... </body> extractContent: false, //parseOnLoad: Boolean //parse content and create the widgets, if any parseOnLoad:true, //preventCache: Boolean //Cache content retreived externally preventCache:false, //preload: Boolean //Force load of data even if pane is hidden. preload: false, //refreshOnShow: Boolean //Refresh (re-download) content when pane goes from hidden to shown refreshOnShow: false, //loadingMessage: String //Message that shows while downloading loadingMessage: "<span class='dijitContentPaneLoading'>$ {loadingState}</span>", //errorMessage: String //Message that shows if an error occurs errorMessage: "<span class='dijitContentPaneError'>${errorState}</span>", You don't need any of those, of course. A simple way to create a ContentPane would be: var pane = new dojo.layout.ContentPane({}); And a more common example would be the following: var panediv = dojo.byId('panediv');var pane = new dojo.layout.ContentPane({ href: "/foo/content.html", preload: true}, panediv); where we would have an element already in the page with the id 'panediv'. As you see, there are also a couple of properties that manage caching and parsing of contents. At times, you want your ContentPane to parse and render any content inside it (if it contains other widgets), whereas other times you might not (if it contains a source code listing, for instance). You will see additional properties being passed in the creation of a ContentPane which are not part of the ContentPane itself, but are properties that give information specific to the surrounding Container. For example, the TabContainer wants to know which tab this is, and so on. Container functions All container widgets arrange other widgets, and so have a lot of common functionality defined in the dijit._Container class. The following functions are provided for all Container widgets: addChild: Adds a child widget to the container. removeChild: Removes a child widget from the container. destroyDescendants: Iterates over all children, calling destroy on each. getChildren: Returns an array containing references to all children. hasChildren: Returns a boolean. LayoutContainer The LayoutContainer is a widget which lays out children widgets according to one of five alignments: right, left, top, bottom, or client. Client means "whatever is left", basically. The widgets being organized need not be ContentPanes, but this is normally the case. Each widget then gets to set a layoutAlign property, like this: layoutAlign = "left". The normal way to use LayoutContainer is to define it using markup in the page, and then define the widgets to be laid out inside it. LayoutContainer has been superceeded by BorderContainer, and will be removed in Dojo version 2.0. SplitContainer The SplitContainer creates a horizontal or vertical split bar between two or more child widgets. A markup declaration of a SplitContainer can look like this: <div dojoType="dijit.layout.SplitContainer" orientation="vertical" sizerWidth="7" activeSizing="false" style="border: 1px solid #bfbfbf; float: left; margin-right: 30px; width: 400px; height: 300px;"> The SplitContainer must have a defined height and width. The orientation property is self-explanatory, as is sizerWidth. The property activeSizing means, if set to true, that the child widgets will be continually resized when the user changes the position of the sizer. This can be bad if the child widgets are complex or access remote information to render themselves, in which case the setting can be set to false, as in the above example. Then the resize event will only be sent to the child widgets when the user stops. Each child widget needs to define the sizeMin and sizeShare attributes. The sizeMin attribute defines the minimum size for the widget in pixels, but the sizeShare attribute is a relative value for the share of space this widget takes in relation to the other widget's sizeShare values. If we have three widgets inside the SplitPane with sizeShare values of 10, 40 and 50, they will have the same ratios in size as if the values had been 1:4:5. StackContainer The StackContainer hides all children widgets but only one at any given time, and is one of the base classes for both the Accordion and TabContainers. StackContainer exists as a separate widget to allow you to define how and when the child widgets are shown. Maybe you would like to define a special kind of control for changing between child widget views, or maybe you want other events in your application to make the Container show specific widgets. Either way, the StackContainer is one of the most versatile Containers, along with the BorderContainer. The following functions are provided for interacting with the StackContainer: back - Selects and shows the previous child widget. forward - Selects and shows the next child widget. getNextSibling - Returns a reference to the next child widget. getPreviousSibling - Returns a reference to the previous child widget. selectChild - Takes a reference to the child widget to select and show.   Here is the slightly abbreviated markup for the test shown above (test_StackContainer.html): <div id="myStackContainer" dojoType="dijit.layout.StackContainer" style="width: 90%; border: 1px solid #9b9b9b; height: 20em; margin: 0.5em 0 0.5em 0; padding: 0.5em;"> <p id="page1" dojoType="dijit.layout.ContentPane" title= "page 1">IT WAS the best of times, ....</p> <p id="page2" dojoType="dijit.layout.ContentPane" title= "page 2">There were a king with a large jaw ...</p> <p id="page3" dojoType="dijit.layout.ContentPane" title= "page 3">It was the year of Our Lord one thousand seven hundred and seventy- five. .../p></div> The StackContainer also publishes topics on certain events which can be caught using the messaging system. The topics are: [widgetId]-addChild [widgetId]-removeChild [widgetId]-selectChild [widgetId]-containerKeyPress Where [widgetId] is the id of this widget. So if you had a StackContainer defined in the following manner: <div id="myStackContainer" dojoType="dijit.layout.StackContainer">...</div> You can use the following code to listen to events from your StackContainer: dojo.subscribe("myStackContainer-addChild", this, function(arg){ var child = arg[0]; var index = arg[1];}); Compare with the following code from the StackContainer class itself: addChild: function(/*Widget*/ child, /*Integer?*/ insertIndex) { // summary: Adds a widget to the stack this.inherited(arguments); if(this._started) { // in case the tab titles have overflowed from one line // to two lines this.layout(); dojo.publish(this.id+"-addChild", [child, insertIndex]); // if this is the first child, then select it if(!this.selectedChildWidget) { this.selectChild(child); } } }, Also declared in the class file for the StackContainer is the dijit.layout.StackController. This is a sample implementation of a separate widget which presents user controls for stepping forward, backward, and so on in the widget stack. What differentiates this widget from the Tabs in the TabContainer, for example, is that the widget is completely separate and uses the message bus to listen to events from the StackContainer. You can use it as-is, or subclass it as a base for you own controllers. But naturally, you can build whatever you want and connect the events to the forward() and back() function on the StackContainer. It's interesting to note that at the end of the files that define StackContainer, the _Widget base class for all widgets is extended in the following way: //These arguments can be specified for the children of a//StackContainer.//Since any widget can be specified as a StackContainer child,//mix them into the base widget class. (This is a hack, but it's//effective.)dojo.extend(dijit._Widget, {//title: String//Title of this widget.Used by TabContainer to the name the tab, etc.title: "",//selected: Boolean//Is this child currently selected?selected: false,//closable: Boolean//True if user can close (destroy) this child, such as//(for example) clicking the X on the tab.closable: false, //true if user can close this tab paneonClose: function(){//summary: Callback if someone tries to close the child, child//will be closed if func returns true return true; }}); This means that all child widgets inside a StackContainer (or Tab or AccordionContainer) can define the above properties, which will be respected and used accordingly. However, since the properties are applied to the _Widget superclass they are of course now generic to all widgets, even those not used inside any containers at all. The most commonly used property is the closable property, which adds a close icon to the widget and title, which defines a title for the tab. A lot of Dijits respond to keypress events, according to WAI rules. Let's look at the code that is responsible for managing key events in StackContainer and all its descendants: onkeypress: function(/*Event*/ e){ //summary: //Handle keystrokes on the page list, for advancing to next/previous button //and closing the current page if the page is closable. if(this.disabled || e.altKey ){ return; } var forward = null; if(e.ctrlKey || !e._djpage){ var k = dojo.keys; switch(e.charOrCode){ case k.LEFT_ARROW: case k.UP_ARROW: if(!e._djpage){ forward = false; } break; case k.PAGE_UP: if(e.ctrlKey){ forward = false; } break; case k.RIGHT_ARROW: case k.DOWN_ARROW: if(!e._djpage){ forward = true; } break; case k.PAGE_DOWN: if(e.ctrlKey){ forward = true; } break; case k.DELETE: if(this._currentChild.closable) { this.onCloseButtonClick(this._currentChild); } dojo.stopEvent(e); break; default: if(e.ctrlKey){ if(e.charOrCode == k.TAB) { this.adjacent(!.shiftKey).onClick(); dojo.stopEvent(e); } else if(e.charOrCode == "w") { if(this._currentChild.closable) { this.onCloseButtonClick(this._currentChild); } dojo.stopEvent(e); // avoid browser tab closing. } } } // handle page navigation if(forward !== null) { this.adjacent(forward).onClick(); dojo.stopEvent(e); } }}, The code is a very good example on how to handle key press events in Dojo in its own right, but for our purposes we can summarize in the following way: If UP, LEFT, or SHIFT+TAB is pressed, forward is set to false, and the last block of code will use that as an argument to the adjacent function which returns the prior child widget if false and the next child widget if true. In this case, the former. If DOWN, RIGHT, or TAB is pressed, forward will be set to true, which will declare the next child widget to be activated and shown. If DELETE or w is pressed and the current child widget is closable, it will be destroyed. TabContainer The TabContainer, which derives from StackContainer, organizes all its children into tabs, which are shown one at a time. As you can see in the picture below, the TabContainer can also manage hierarchical versions of itself. The TabContainer takes an argument property called tabPosition, which controls where the tab icons are displayed for each tab. Possible values are "top", "bottom", "left-h", "right-h", with "top" as default. There are no special functions provided for TabContainer, which adds very little logic to that provided from the StackContainer superclass. AccordionContainer The AccorionContainer shows a horizontal bar for each added child widget, which represents its collapsed state. The bar acts as a tab and also holds the title defined for the child widget. When the bar is clicked, an animation hides the current widget, and also animates in the widget whose bar was clicked. The abbreviated code for the test case above (test_accordionContainer.html) is here: <div dojoType="dijit.layout.AccordionContainer" style="width: 400px; height: 300px; overflow: hidden"> <div dojoType="dijit.layout.AccordionPane" title="a"> Hello World </div> <div dojoType="dijit.layout.AccordionPane" title="b"> <p> Nunc consequat nisi ... </p> <p> Sed arcu magna... </p> </div> <div dojoType="dijit.layout.AccordionPane" title="c"> <p>The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog.</p> </div></div>  A funny thing about the AccordionContainer is that it requires not any old widget as a child node, but its own AccordionPane, as you see in the code above. However, the AccordionPane has ContentPane as superclass, and defines itself slightly differently due to the special looks of an accordion. Also, the AccordionPane does not currently support nested Layout widgets, even though single-level widgets are supported. BorderContainer The BorderContainer has replaced the functionality of both LayoutContainer and SplitContainer. Note that the outermost BorderContainer widget does not carry any layout information. This is instead delegated to each individual widget. As each child gets added to the BorderContainer, the layout is recalculated. Using the BorderContainer is a very good alternative to using CSS-based "tableless tables". For using the BorderContainer, you don't need any other rules, and the Container recalculates positioning automatically, without the need for additional CSS rules (except for the height/width case below) each time you add an element or widget to the area. Since BorderContainer widget replaces both SplitContainer and LayoutContainer, it both lets its child widgets declare where they are in relation to each other. Optionally, add resizing splitter between children. Also, instead of optionally declaring one child as "client", one child must now always be declared as "center". For some reason, the child widget now use region, instead of layoutAlign, so a child widget which would have been defined like this in LayoutContainer: <div dojoType="dijit.layout.ContentPane" layoutAlign="top">...</div> is now defined like this instead: <div dojoType="dijit.layout.ContentPane" region="top">...</div> All "side" widgets must define a width, in style, by CSS class or otherwise, and the same applies for top/bottom widgets, but with height. Center widgets must not declare either height or width, since they use whatever is left over from the other widgets. You can also use leading and trailing instead of right and left. The only difference is that when you change locale to a region that has text going from right to left (like Arabic and many others), this will arrange the widgets appropriate to the locale. The BorderContainer also takes an optional design property, which defines if the BorderContainer is a headline or sidebar. The headline is the default and looks like the picture below. headline means that the top and bottom widgets extend the full length of the container, whereas sidebar means the the right and left (or leading and trailing) widgets extend top to bottom. The sizeShare attribute for the ContentPanes used in the SplitContainer is deprecated in BorderContainer. All ContentPanes sizes are defined using regular techniques (direct stylin, classes, and so on). From the BorderContainer test located in dijit/tests/layout/test_BorderContainer_nested.html, we find the following layout: The (abbreviated) source code for the example is here: <div dojoType="dijit.layout.BorderContainer" style="border: 2px solid black; width: 90%; height: 500px; padding: 10px;"> <div dojoType="dijit.layout.ContentPane" region="left" style="background-color: #acb386; width: 100px;"> left </div> <div dojoType="dijit.layout.ContentPane" region="right" style="background-color: #acb386; width: 100px;"> right </div> <div dojoType="dijit.layout.ContentPane" region="top" style="background-color: #b39b86; height: 100px;"> top bar </div> <div dojoType="dijit.layout.ContentPane" region="bottom" style="background-color: #b39b86; height: 100px;"> bottom bar </div> <div dojoType="dijit.layout.ContentPane" region="center" style="background-color: #f5ffbf; padding: 0px;"> <div dojoType="dijit.layout.BorderContainer" design="sidebar" style="border: 2px solid black; height: 300px;"> <div dojoType="dijit.layout.ContentPane" region="left" style="background-color: #acb386; width: 100px;"> left </div> <div dojoType="dijit.layout.ContentPane" region="right" style="background-color: #acb386; width: 100px;"> right </div> <div dojoType="dijit.layout.ContentPane" region="top" style="background-color: #b39b86; height: 100px;"> top bar </div> <div dojoType="dijit.layout.ContentPane" region="bottom" style="background-color: #b39b86; height: 100px;"> bottom bar </div> <div dojoType="dijit.layout.ContentPane" region="center" style="background-color: #f5ffbf; padding: 10px;"> main panel with <a href="http://www.dojotoolkit.org/"> a link</a>.<br /> (to check we're copying children around properly). <br /> <select dojoType="dijit.form.FilteringSelect"> <option value="1">foo</option> <option value="2">bar</option> <option value="3">baz</option> </select> Here's some text that comes AFTER the combo box. </div> </div> </div></div> You see here the recurring theme of using ContentPanes inside Containers. Also, the innermost "center" ContentPane wraps a new BorderContainer which has its own internal top/left layout widgets. Depending on what kind of application you are building, the BorderContainer might be a good starting point. Since you already know that you can change and reload the contents of individual ContentPanes, you are left with a layout in which each element can function as a lightweight Iframe with none of the negative side effects. DragPane The DragPane is a very simple idea. You have a very large area of elements to display, and want to let the user 'drag' the underlying pane around using the mouse. The DragPane can be used in instances where you have a lot of pictures to present. It can also be used to present text or other widgets that are too numerous to fit in your current designated area of screen real estate. The only property of DragPane is invert, which if set to true, inverts the axis of the drag of the mouse. Example: <div class="hugetext" id="container" invert="false" dojoType="dojox.layout.DragPane"> <p style="color:#666; padding:8px; margin:0;"> Lorem ipsum dolor sit amet, consectetuer adipiscing elit. In porta. Etiam mattis libero nec ante. Nam porta lacus eu ligula. Cras mauris. Suspendisse vel augue. Vivamus aliquam orci ut eros. Nunc eleifend sagittis turpis. purus purus in nibh. Phasellus in nunc. </p></div>

ZBrush 4 Sculpting for Games: Beginner's Guide Sculpt machines, environments, and creatures for your game development projects         Read more about this book       (For more resources on the subject, see here.) You can download the finished creature by clicking here. Retopologizing for games Okay, we've got our high-poly mesh ready. But in order to bring it into a game engine, we need to build a low-polygon mesh from it. Because we used ZSketch in the beginning, we now create the low-poly mesh at the end of the process. This step is called retopologizing, or in most cases just retopo, because it creates a mesh with a new topology and projects the details from the high-poly onto it. In this way, we end up with a mesh with a clean and optimized topology and all of the high resolution details that we sculpted in earlier. This process can also be done in other major 3D applications such as Blender, 3ds Max, and so on. Honestly, I would prefer the retopologizing in Blender over that of ZBrush, as I find it a bit more robust. Nonetheless, the results are the same. So let's see how we can do this in ZBrush Before retopologizing, we should be clear about these points: Polycount: Think of how many polygons the character should have; this always seems tricky, but after some time you get used to it. Ask yourself how many polygons your platform and engine can handle for a character and how big will it be seen onscreen. Let's say our Brute will be used on a PC and console title, he's a few meters high and is visible fullscreen. So we're heading roughly for something around 5000 polygons. This is a very rough estimate and if we need less polygons for it, that's fine too. The fewer the better. Animation: How will the character be animated? If there's elaborate facial animation going on, the character should have enough polygons in his face, so the animator can move something. Ideally, the polygon-flow supports the animation. Quads: Always try to establish as many quads as possible. Sometimes, we just need to add triangles, but try to go for quads wherever you can. With these three points in mind, let's start retopologizing Time for action – creating an in-game mesh with retopologize First, lower the subdivision levels of all the subtools to get increased performance. Append a ZSphere as a subtool and make it the active one. Name it something meaningful like Retopo. Activate Transparency and place the ZSphere inside the mesh so that it is hidden, as shown in the next screenshot. Appending a ZSphere to the model gives us the advantage to easily hide subtools or adjust their subdivision level as we need it: Deactivate Transparency and go to Tool Topology| and click on Edit Topology. Activate Symmetry. Pick the SkinShade4 material and choose a darker color, so that we can see the orange lines of the retopologizing better. The SkinShade4 material offers very low contrast, which is what we need here. Begin drawing edges by left-clicking on the mesh. They will automatically snap to the underlying high-poly surface. Start out by laying in two circles around the eyes, as explained in the next image Retopologize commands When in retopologize mode, left-clicking adds a new vertex and an edge connecting it to the last selected vertex. If we want to start out from a different vertex, we can select a new starting point by Ctrl + left-clicking. Deleting points can be done by Alt + left-clicking. When you miss the point you wanted to delete, the last selected one gets deleted instead, so be careful about that.

In this article by Richard M Reese, author of the book Natural Language Processing with Java, we will see how to use NLP APIs. Using NLP APIs We will demonstrate the NER process using OpenNLP, Stanford API, and LingPipe. Each of these provide alternate techniques that can often do a good job of identifying entities in the text. The following declaration will serve as the sample text to demonstrate the APIs: String sentences[] = {"Joe was the last person to see Fred. ", "He saw him in Boston at McKenzie's pub at 3:00 where he " + " paid $2.45 for an ale. ", "Joe wanted to go to Vermont for the day to visit a cousin who " + "works at IBM, but Sally and he had to look for Fred"}; Using OpenNLP for NER We will demonstrate the use of the TokenNameFinderModel class to perform NLP using the OpenNLP API. Additionally, we will demonstrate how to determine the probability that the entity identified is correct. The general approach is to convert the text into a series of tokenized sentences, create an instance of the TokenNameFinderModel class using an appropriate model, and then use the find method to identify the entities in the text. The following example demonstrates the use of the TokenNameFinderModel class. We will use a simple sentence initially and then use multiple sentences. The sentence is defined here: String sentence = "He was the last person to see Fred."; We will use the models found in the en-token.bin and en-ner-person.bin files for the tokenizer and name finder models, respectively. The InputStream object for these files is opened using a try-with-resources block, as shown here: try (InputStream tokenStream = new FileInputStream(        new File(getModelDir(), "en-token.bin"));        InputStream modelStream = new FileInputStream(            new File(getModelDir(), "en-ner-person.bin"));) {    ...   } catch (Exception ex) {    // Handle exceptions } Within the try block, the TokenizerModel and Tokenizer objects are created:    TokenizerModel tokenModel = new TokenizerModel(tokenStream);    Tokenizer tokenizer = new TokenizerME(tokenModel); Next, an instance of the NameFinderME class is created using the person model: TokenNameFinderModel entityModel =    new TokenNameFinderModel(modelStream); NameFinderME nameFinder = new NameFinderME(entityModel); We can now use the tokenize method to tokenize the text and the find method to identify the person in the text. The find method will use the tokenized String array as input and return an array of Span objects, as shown: String tokens[] = tokenizer.tokenize(sentence); Span nameSpans[] = nameFinder.find(tokens); The Span class holds positional information about the entities found. The actual string entities are still in the tokens array: The following for statement displays the person found in the sentence. Its positional information and the person are displayed on separate lines: for (int i = 0; i < nameSpans.length; i++) {    System.out.println("Span: " + nameSpans[i].toString());    System.out.println("Entity: "        + tokens[nameSpans[i].getStart()]); } The output is as follows: Span: [7..9) person Entity: Fred We will often work with multiple sentences. To demonstrate this, we will use the previously defined sentences string array. The previous for statement is replaced with the following sequence. The tokenize method is invoked against each sentence and then the entity information is displayed as earlier: for (String sentence : sentences) {    String tokens[] = tokenizer.tokenize(sentence);    Span nameSpans[] = nameFinder.find(tokens);    for (int i = 0; i < nameSpans.length; i++) {        System.out.println("Span: " + nameSpans[i].toString());        System.out.println("Entity: "            + tokens[nameSpans[i].getStart()]);    }    System.out.println(); } The output is as follows. There is an extra blank line between the two people detected because the second sentence did not contain a person: Span: [0..1) person Entity: Joe Span: [7..9) person Entity: Fred     Span: [0..1) person Entity: Joe Span: [19..20) person Entity: Sally Span: [26..27) person Entity: Fred Determining the accuracy of the entity When the TokenNameFinderModel identifies entities in text, it computes a probability for that entity. We can access this information using the probs method as shown in the following line of code. This method returns an array of doubles, which corresponds to the elements of the nameSpans array: double[] spanProbs = nameFinder.probs(nameSpans); Add this statement to the previous example immediately after the use of the find method. Then add the next statement at the end of the nested for statement: System.out.println("Probability: " + spanProbs[i]); When the example is executed, you will get the following output. The probability fields reflect the confidence level of the entity assignment. For the first entity, the model is 80.529 percent confident that "Joe" is a person: Span: [0..1) person Entity: Joe Probability: 0.8052914774025202 Span: [7..9) person Entity: Fred Probability: 0.9042160889302772   Span: [0..1) person Entity: Joe Probability: 0.9620970782763985 Span: [19..20) person Entity: Sally Probability: 0.964568603518126 Span: [26..27) person Entity: Fred Probability: 0.990383039618594 Using other entity types OpenNLP supports different libraries as listed in the following table. These models can be downloaded from http://opennlp.sourceforge.net/models-1.5/. The prefix, en, specifies English as the language and ner indicates that the model is for NER. English finder models Filename Location name finder model en-ner-location.bin Money name finder model en-ner-money.bin Organization name finder model en-ner-organization.bin Percentage name finder model en-ner-percentage.bin Person name finder model en-ner-person.bin Time name finder model en-ner-time.bin If we modify the statement to use a different model file, we can see how they work against the sample sentences: InputStream modelStream = new FileInputStream(    new File(getModelDir(), "en-ner-time.bin"));) { When the en-ner-money.bin model is used, the index in the tokens array in the earlier code sequence has to be increased by one. Otherwise, all that is returned is the dollar sign. The various outputs are shown in the following table. Model Output en-ner-location.bin Span: [4..5) location Entity: Boston Probability: 0.8656908776583051 Span: [5..6) location Entity: Vermont Probability: 0.9732488014011262 en-ner-money.bin Span: [14..16) money Entity: 2.45 Probability: 0.7200919701507937 en-ner-organization.bin Span: [16..17) organization Entity: IBM Probability: 0.9256970736336729 en-ner-time.bin The model was not able to detect time in this text sequence The model failed to find the time entities in the sample text. This illustrates that the model did not have enough confidence that it found any time entities in the text. Processing multiple entity types We can also handle multiple entity types at the same time. This involves creating instances of the NameFinderME class based on each model within a loop and applying the model against each sentence, keeping track of the entities as they are found. We will illustrate this process with the following example. It requires rewriting the previous try block to create the InputStream instance within the block, as shown here: try {    InputStream tokenStream = new FileInputStream(        new File(getModelDir(), "en-token.bin"));    TokenizerModel tokenModel = new TokenizerModel(tokenStream);    Tokenizer tokenizer = new TokenizerME(tokenModel);    ... } catch (Exception ex) {    // Handle exceptions } Within the try block, we will define a string array to hold the names of the model files. As shown here, we will use models for people, locations, and organizations: String modelNames[] = {"en-ner-person.bin",    "en-ner-location.bin", "en-ner-organization.bin"}; An ArrayList instance is created to hold the entities as they are discovered: ArrayList<String> list = new ArrayList(); A for-each statement is used to load one model at a time and then to create an instance of the NameFinderME class: for(String name : modelNames) {    TokenNameFinderModel entityModel = new TokenNameFinderModel(        new FileInputStream(new File(getModelDir(), name)));    NameFinderME nameFinder = new NameFinderME(entityModel);    ... } Previously, we did not try to identify which sentences the entities were found in. This is not hard to do but we need to use a simple for statement instead of a for-each statement to keep track of the sentence indexes. This is shown in the following example, where the previous example has been modified to use the integer variable index to keep the sentences. Otherwise, the code works the same way as earlier: for (int index = 0; index < sentences.length; index++) {    String tokens[] = tokenizer.tokenize(sentences[index]);    Span nameSpans[] = nameFinder.find(tokens);    for(Span span : nameSpans) {        list.add("Sentence: " + index            + " Span: " + span.toString() + " Entity: "            + tokens[span.getStart()]);    } } The entities discovered are then displayed: for(String element : list) {    System.out.println(element); } The output is as follows: Sentence: 0 Span: [0..1) person Entity: Joe Sentence: 0 Span: [7..9) person Entity: Fred Sentence: 2 Span: [0..1) person Entity: Joe Sentence: 2 Span: [19..20) person Entity: Sally Sentence: 2 Span: [26..27) person Entity: Fred Sentence: 1 Span: [4..5) location Entity: Boston Sentence: 2 Span: [5..6) location Entity: Vermont Sentence: 2 Span: [16..17) organization Entity: IBM Using the Stanford API for NER We will demonstrate the CRFClassifier class as used to perform NER. This class implements what is known as a linear chain Conditional Random Field (CRF) sequence model. To demonstrate the use of the CRFClassifier class, we will start with a declaration of the classifier file string, as shown here: String model = getModelDir() +    "\english.conll.4class.distsim.crf.ser.gz"; The classifier is then created using the model: CRFClassifier<CoreLabel> classifier =    CRFClassifier.getClassifierNoExceptions(model); The classify method takes a single string representing the text to be processed. To use the sentences text, we need to convert it to a simple string: String sentence = ""; for (String element : sentences) {    sentence += element; } The classify method is then applied to the text. List<List<CoreLabel>> entityList = classifier.classify(sentence); A List instance of List instances of CoreLabel objects is returned. The object returned is a list that contains another list. The contained list is a List instance of CoreLabel objects. The CoreLabel class represents a word with additional information attached to it. The "internal" list contains a list of these words. In the outer for-each statement in the following code sequence, the reference variable, internalList, represents one sentence of the text. In the inner for-each statement, each word in that inner list is displayed. The word method returns the word and the get method returns the type of the word. The words and their types are then displayed: for (List<CoreLabel> internalList: entityList) {    for (CoreLabel coreLabel : internalList) {        String word = coreLabel.word();        String category = coreLabel.get(            CoreAnnotations.AnswerAnnotation.class);        System.out.println(word + ":" + category);    } } Part of the output follows. It has been truncated because every word is displayed. The O represents the "Other" category: Joe:PERSON was:O the:O last:O person:O to:O see:O Fred:PERSON .:O He:O ... look:O for:O Fred:PERSON To filter out the words that are not relevant, replace the println statement with the following statements. This will eliminate the other categories: if (!"O".equals(category)) {    System.out.println(word + ":" + category); } The output is simpler now: Joe:PERSON Fred:PERSON Boston:LOCATION McKenzie:PERSON Joe:PERSON Vermont:LOCATION IBM:ORGANIZATION Sally:PERSON Fred:PERSON Using LingPipe for NER We will demonstrate how name entity models and the ExactDictionaryChunker class are used to perform NER analysis. Using LingPipe's name entity models LingPipe has a few named entity models that we can use with chunking. These files consist of a serialized object that can be read from a file and then applied to text. These objects implement the Chunker interface. The chunking process results in a series of Chunking objects that identify the entities of interest. A list of the NER models is found in the following table. These models can be downloaded from http://alias-i.com/lingpipe/web/models.html: Genre Corpus File English News MUC-6 ne-en-news-muc6.AbstractCharLmRescoringChunker English Genes GeneTag ne-en-bio-genetag.HmmChunker English Genomics GENIA ne-en-bio-genia.TokenShapeChunker We will use the model found in the ne-en-news-muc6.AbstractCharLmRescoringChunker file to demonstrate how this class is used. We start with a try-catch block to deal with exceptions as shown in the following example. The file is opened and used with the AbstractExternalizable class' static readObject method to create an instance of a Chunker class. This method will read in the serialized model: try {    File modelFile = new File(getModelDir(),        "ne-en-news-muc6.AbstractCharLmRescoringChunker");      Chunker chunker = (Chunker)        AbstractExternalizable.readObject(modelFile);    ... } catch (IOException | ClassNotFoundException ex) {    // Handle exception } The Chunker and Chunking interfaces provide methods that work with a set of chunks of text. Its chunk method returns an object that implements the Chunking instance. The following sequence displays the chunks found in each sentence of the text, as shown here: for (int i = 0; i < sentences.length; ++i) {    Chunking chunking = chunker.chunk(sentences[i]);    System.out.println("Chunking=" + chunking); } The output of this sequence is as follows: Chunking=Joe was the last person to see Fred. : [0-3:PERSON@-Infinity, 31-35:ORGANIZATION@-Infinity] Chunking=He saw him in Boston at McKenzie's pub at 3:00 where he paid $2.45 for an ale. : [14-20:LOCATION@-Infinity, 24-32:PERSON@-Infinity] Chunking=Joe wanted to go to Vermont for the day to visit a cousin who works at IBM, but Sally and he had to look for Fred : [0-3:PERSON@-Infinity, 20-27:ORGANIZATION@-Infinity, 71-74:ORGANIZATION@-Infinity, 109-113:ORGANIZATION@-Infinity] Instead, we can use methods of the Chunk class to extract specific pieces of information as illustrated here. We will replace the previous for statement with the following for-each statement. This calls a displayChunkSet method: for (String sentence : sentences) {    displayChunkSet(chunker, sentence); } The output that follows shows the result. However, it does not always match the entity type correctly. Type: PERSON Entity: [Joe] Score: -Infinity Type: ORGANIZATION Entity: [Fred] Score: -Infinity Type: LOCATION Entity: [Boston] Score: -Infinity Type: PERSON Entity: [McKenzie] Score: -Infinity Type: PERSON Entity: [Joe] Score: -Infinity Type: ORGANIZATION Entity: [Vermont] Score: -Infinity Type: ORGANIZATION Entity: [IBM] Score: -Infinity Type: ORGANIZATION Entity: [Fred] Score: -Infinity Using the ExactDictionaryChunker class The ExactDictionaryChunker class provides an easy way to create a dictionary of entities and their types, which can be used to find them later in text. It uses a MapDictionary object to store entries and then the ExactDictionaryChunker class is used to extract chunks based on the dictionary. The AbstractDictionary interface supports basic operations for entities, categories, and scores. The score is used in the matching process. The MapDictionary and TrieDictionary classes implement the AbstractDictionary interface. The TrieDictionary class stores information using a character trie structure. This approach uses less memory when it is a concern. We will use the MapDictionary class for our example. To illustrate this approach, we start with a declaration of the MapDictionary class: private MapDictionary<String> dictionary; The dictionary will contain the entities that we are interested in finding. We need to initialize the model as performed in the following initializeDictionary method. The DictionaryEntry constructor used here accepts three arguments: String: The name of the entity String: The category of the entity Double: Represent a score for the entity The score is used when determining matches. A few entities are declared and added to the dictionary. private static void initializeDictionary() {    dictionary = new MapDictionary<String>();    dictionary.addEntry(        new DictionaryEntry<String>("Joe","PERSON",1.0));    dictionary.addEntry(        new DictionaryEntry<String>("Fred","PERSON",1.0));    dictionary.addEntry(        new DictionaryEntry<String>("Boston","PLACE",1.0));    dictionary.addEntry(        new DictionaryEntry<String>("pub","PLACE",1.0));    dictionary.addEntry(        new DictionaryEntry<String>("Vermont","PLACE",1.0));    dictionary.addEntry(        new DictionaryEntry<String>("IBM","ORGANIZATION",1.0));    dictionary.addEntry(        new DictionaryEntry<String>("Sally","PERSON",1.0)); } An ExactDictionaryChunker instance will use this dictionary. The arguments of the ExactDictionaryChunker class are detailed here: Dictionary<String>: It is a dictionary containing the entities TokenizerFactory: It is a tokenizer used by the chunker boolean: If it is true, the chunker should return all matches boolean: If it is true, matches are case sensitive Matches can be overlapping. For example, in the phrase "The First National Bank", the entity "bank" could be used by itself or in conjunction with the rest of the phrase. The third parameter determines if all of the matches are returned. In the following sequence, the dictionary is initialized. We then create an instance of the ExactDictionaryChunker class using the Indo-European tokenizer, where we return all matches and ignore the case of the tokens: initializeDictionary(); ExactDictionaryChunker dictionaryChunker    = new ExactDictionaryChunker(dictionary,        IndoEuropeanTokenizerFactory.INSTANCE, true, false); The dictionaryChunker object is used with each sentence, as shown in the following code sequence. We will use the displayChunkSet method: for (String sentence : sentences) {    System.out.println("nTEXT=" + sentence);    displayChunkSet(dictionaryChunker, sentence); } On execution, we get the following output: TEXT=Joe was the last person to see Fred. Type: PERSON Entity: [Joe] Score: 1.0 Type: PERSON Entity: [Fred] Score: 1.0   TEXT=He saw him in Boston at McKenzie's pub at 3:00 where he paid $2.45 for an ale. Type: PLACE Entity: [Boston] Score: 1.0 Type: PLACE Entity: [pub] Score: 1.0   TEXT=Joe wanted to go to Vermont for the day to visit a cousin who works at IBM, but Sally and he had to look for Fred Type: PERSON Entity: [Joe] Score: 1.0 Type: PLACE Entity: [Vermont] Score: 1.0 Type: ORGANIZATION Entity: [IBM] Score: 1.0 Type: PERSON Entity: [Sally] Score: 1.0 Type: PERSON Entity: [Fred] Score: 1.0 This does a pretty good job but it requires a lot of effort to create the dictionary for a large vocabulary. Training a model We will use OpenNLP to demonstrate how a model is trained. The training file used must: Contain marks to demarcate the entities Have one sentence per line We will use the following model file named en-ner-person.train: <START:person> Joe <END> was the last person to see <START:person> Fred <END>. He saw him in Boston at McKenzie's pub at 3:00 where he paid $2.45 for an ale. <START:person> Joe <END> wanted to go to Vermont for the day to visit a cousin who works at IBM, but <START:person> Sally <END> and he had to look for <START:person> Fred <END>. Several methods of this example are capable of throwing exceptions. These statements will be placed in a try-with-resource block as shown here, where the model's output stream is created: try (OutputStream modelOutputStream = new BufferedOutputStream(        new FileOutputStream(new File("modelFile")));) {    ... } catch (IOException ex) {    // Handle exception } Within the block, we create an OutputStream<String> object using the PlainTextByLineStream class. This class' constructor takes a FileInputStream instance and returns each line as a String object. The en-ner-person.train file is used as the input file, as shown here. The UTF-8 string refers to the encoding sequence used: ObjectStream<String> lineStream = new PlainTextByLineStream(    new FileInputStream("en-ner-person.train"), "UTF-8"); The lineStream object contains streams that are annotated with tags delineating the entities in the text. These need to be converted to the NameSample objects so that the model can be trained. This conversion is performed by the NameSampleDataStream class as shown here. A NameSample object holds the names of the entities found in the text: ObjectStream<NameSample> sampleStream =    new NameSampleDataStream(lineStream); The train method can now be executed as follows: TokenNameFinderModel model = NameFinderME.train(    "en", "person", sampleStream,    Collections.<String, Object>emptyMap(), 100, 5); The arguments of the method are as detailed in the following table: Parameter Meaning "en" Language Code "person" Entity type sampleStream Sample data null Resources 100 The number of iterations 5 The cutoff The model is then serialized to an output file: model.serialize(modelOutputStream); The output of this sequence is as follows. It has been shortened to conserve space. Basic information about the model creation is detailed: Indexing events using cutoff of 5   Computing event counts... done. 53 events Indexing... done. Sorting and merging events... done. Reduced 53 events to 46. Done indexing. Incorporating indexed data for training... Number of Event Tokens: 46      Number of Outcomes: 2    Number of Predicates: 34 ...done. Computing model parameters ... Performing 100 iterations. 1: ... loglikelihood=-36.73680056967707 0.05660377358490566 2: ... loglikelihood=-17.499660626361216 0.9433962264150944 3: ... loglikelihood=-13.216835449617108 0.9433962264150944 4: ... loglikelihood=-11.461783667999262 0.9433962264150944 5: ... loglikelihood=-10.380239416084963 0.9433962264150944 6: ... loglikelihood=-9.570622475692486 0.9433962264150944 7: ... loglikelihood=-8.919945779143012 0.9433962264150944 ... 99: ... loglikelihood=-3.513810438211968 0.9622641509433962 100: ... loglikelihood=-3.507213816708068 0.9622641509433962 Evaluating a model The model can be evaluated using the TokenNameFinderEvaluator class. The evaluation process uses marked up sample text to perform the evaluation. For this simple example, a file called en-ner-person.eval was created that contained the following text: <START:person> Bill <END> went to the farm to see <START:person> Sally <END>. Unable to find <START:person> Sally <END> he went to town. There he saw <START:person> Fred <END> who had seen <START:person> Sally <END> at the book store with <START:person> Mary <END>. The following code is used to perform the evaluation. The previous model is used as the argument of the TokenNameFinderEvaluator constructor. A NameSampleDataStream instance is created based on the evaluation file. The TokenNameFinderEvaluator class' evaluate method performs the evaluation: TokenNameFinderEvaluator evaluator =    new TokenNameFinderEvaluator(new NameFinderME(model));   lineStream = new PlainTextByLineStream(    new FileInputStream("en-ner-person.eval"), "UTF-8"); sampleStream = new NameSampleDataStream(lineStream); evaluator.evaluate(sampleStream); To determine how well the model worked with the evaluation data, the getFMeasure method is executed. The results are then displayed: FMeasure result = evaluator.getFMeasure(); System.out.println(result.toString()); The following output displays the precision, recall, and F-measure. It indicates that 50 percent of the entities found exactly match the evaluation data. The recall is the percentage of entities defined in the corpus that were found in the same location. The performance measure is the harmonic mean and is defined as: F1 = 2 * Precision * Recall / (Recall + Precision) Precision: 0.5 Recall: 0.25 F-Measure: 0.3333333333333333 The data and evaluation sets should be much larger to create a better model. The intent here was to demonstrate the basic approach used to train and evaluate a POS model. Summary We investigated several techniques for performing NER. Regular expressions is one approach that is supported by both core Java classes and NLP APIs. This technique is useful for many applications and there are a large number of regular expression libraries available. Dictionary-based approaches are also possible and work well for some applications. However, they require considerable effort to populate at times. We used LingPipe's MapDictionary class to illustrate this approach. Resources for Article: Further resources on this subject: Tuning Solr JVM and Container [article] Model-View-ViewModel [article] AngularJS Performance [article]

(For more resources on Plone, see here.) Faculty/Staff Directory is practically a departmental web site in a box, but it doesn't stop there. The product is general enough to serve in non-academic settings as well. It is also not limited to people; it has been repurposed to such diverse applications as cataloguing tea leaves. It really is a general-purpose taxonomy engine—albeit with an academic bent in many of its naming choices. This article is an exploration of a product that will likely provide the framework for much of your site. We'll tour its features, get ideas on how they're commonly used in a school setting, and, finally, get a sneak peek into the future of the product. Install the product Faculty/Staff Directory depends on several other products: membrane, Relations, and archetypes.schemaextender. Thus, the easiest way to get it, as documented in its readme, is by adding Products.FacultyStaffDirectory to your buildout.cfg, like so: [buildout]eggs = ...(other eggs)... Products.FacultyStaffDirectory Then, shut down Zope, and run buildout. After starting up Zope again, install FacultyStaffDirectory in the Add-on Products control panel, and you're ready to go. Test drive Faculty/Staff Directory Faculty/Staff Directory (FSD) is tremendously flexible; you can use it as a simple list of phone numbers or as the foundation supporting the rest of your site. In this section, we take FSD for a spin, with ample pit stops along the way to discuss alternative design possibilities. Keep in mind that FSD's features are given to creative repurposing. We'll not only see their typical uses but some more inventive ones as well. Create a directory and choose global roles Faculty/Staff Directory centers around the idea of collecting people in a central folder called, fittingly, a faculty/staff directory. Our first step is to create that folder. Once you've installed FSD in the Add-on Products control panel, choose a place in your site where you would like to place a personnel directory, and add a faculty/staff directory there. You'll be prompted for two things: Title (which is usually something like People) and a selection of Roles. These roles, part of FSD's optional user-and-group integration, are granted site-wide to all people in the directory. Thus, as with the similarly scoped roles in the Users and Groups control panel, it's best to use them sparingly: choose only Member or, if you don't intend the people represented in your Directory to log in, no roles at all. Most roles should be assigned on a per-folder basis using the Sharing tab, and FSD provides facilities for that as well, as we discuss later in the Integrate users and groups section. Add people Now that you have a place to keep them, it's time to add actual personnel. FSD represents people through the aptly named Person type. A fully tricked-out person might look like this: People, without exception, live directly inside the faculty/staff directory—though they often appear to be elsewhere, as we will soon see. Each person can track the following information: Basic Information   Access Account ID Doubling as the object ID (or "short name") of the person, this field should contain the person's institution-wide login name. If you choose to take advantage of FSD's user-and-group integration, this is how logged-in users are matched with their Person objects. The name of this field and its validation requirements can be changed (and should be, in most deployments) in the Faculty/Staff Directory control panel under Site Setup. Name First, middle, and last names, as well as a suffix such as Ph. D or Jr. Image A picture of the person in GIF, JPEG, or PNG format, for use on their page and in some listings. It will be automatically scaled as necessary. Classifications Classifications are FSD's most prominent way to group people. This is a convenient place to assign one or more when creating a person. Departments Another way of grouping people. See Group People, below, for an in-depth comparison of all the various types of groupings. Password If the Person objects provide user passwords option in the FSD control panel is on, one can log into the Plone site using the Access Account ID as a username and the contents of this field as a password. Personal Assistant(s) Other people who should have access to edit this person's information (excluding password and the fields under User Settings). Contact Information   Email This email address is run through Plone's spam armoring, as yet not widely cracked, before being displayed to visitors. An alternative armoring method ("somebody AT here DOT edu") is available from the FSD control panel under Site Setup. Street Address City State Postal Code Phone The required format of the Office Phone field, along with its example text, can be set in the FSD control panel-a must for installations outside the United States and Canada. Professional Information   Job Titles As many job titles as you care to provide, one per line. These are all listed on the person's page. Biography A rich text field for the person's biographical information. Since this field can contain all the formatting you can muster-headings, images, and all-it is a wonderful thing to abuse. Fill it with lists of published journal articles, current projects, and anything else appropriate to show on the person's page. Education A plain text field where every line represents a conferred degree, certification, or such. Web Sites A list of URLs, one per line, to display on the person's page. There is no option to provide link text for these URLs, so you may wish to embed links in the Biography field instead. Committees Specialties The committees and specialties to which this person belongs. See the full discussion of FSD's various methods of grouping people under Group People, below. User Settings   Language Content Editor If you leave FSD's user-and-group integration enabled for the Person type, these duplications of the standard Plone options will apply. Integration can be disabled on a type-by-type basis in the FSD control panel under Site Setup. That's a lot of information, but you can fill out only what you'll use. Fields left blank will be omitted from directory pages, labels and all. For more thorough customization, you can write a Faculty/Staff Directory extender product to hide inapplicable fields from your content providers or to collect information that FSD doesn't ordinarily track. Here's how to add people to the directory: From within the directory, pull down the Add new… menu, and choose Person. Fill out some of the person's attributes. Be sure to assign at least one classification to each person, but don't worry about departments, specialties, or committees yet, as we're about to cover them in detail.

In this article by Greg L. Turnquist, the author of the book, Learning Spring Boot, we will discuss modernizing our Spring Boot app with JavaScript and adding production-ready support features. (For more resources related to this topic, see here.) Modernizing our app with JavaScript We just saw that, with a single @Grab statement, Spring Boot automatically configured the Thymeleaf template engine and some specialized view resolvers. We took advantage of Spring MVC's ability to pass attributes to the template through ModelAndView. Instead of figuring out the details of view resolvers, we instead channeled our efforts into building a handy template to render data fetched from the server. We didn't have to dig through reference docs, Google, and Stack Overflow to figure out how to configure and integrate Spring MVC with Thymeleaf. We let Spring Boot do the heavy lifting. But that's not enough, right? Any real application is going to also have some JavaScript. Love it or hate it, JavaScript is the engine for frontend web development. See how the following code lets us make things more modern by creating modern.groovy: @Grab("org.webjars:jquery:2.1.1")@Grab("thymeleaf-spring4")@Controllerclass ModernApp {def chapters = ["Quick Start With Groovy","Quick Start With Java","Debugging and Managing Your App","Data Access with Spring Boot","Securing Your App"]@RequestMapping("/")def home(@RequestParam(value="name", defaultValue="World")String n) {new ModelAndView("modern").addObject("name", n).addObject("chapters", chapters)}} A single @Grab statement pulls in jQuery 2.1.1. The rest of our server-side Groovy code is the same as before. There are multiple ways to use JavaScript libraries. For Java developers, it's especially convenient to use the WebJars project (http://webjars.org), where lots of handy JavaScript libraries are wrapped up with Maven coordinates. Every library is found on the /webjars/<library>/<version>/<module> path. To top it off, Spring Boot comes with prebuilt support. Perhaps you noticed this buried in earlier console outputs: ...2014-05-20 08:33:09.062 ... : Mapped URL path [/webjars/**] onto handlerof [...... With jQuery added to our application, we can amp up our template (templates/modern.html) like this: <html><head><title>Learning Spring Boot - Chapter 1</title><script src="webjars/jquery/2.1.1/jquery.min.js"></script><script>$(document).ready(function() {$('p').animate({fontSize: '48px',}, "slow");});</script></head><body><p th_text="'Hello, ' + ${name}"></p><ol><li th_each="chapter : ${chapters}"th:text="${chapter}"></li></ol></body></html> What's different between this template and the previous one? It has a couple extra <script> tags in the head section: The first one loads jQuery from /webjars/jquery/2.1.1/jquery.min.js (implying that we can also grab jquery.js if we want to debug jQuery) The second script looks for the <p> element containing our Hello, world! message and then performs an animation that increases the font size to 48 pixels after the DOM is fully loaded into the browser If we run spring run modern.groovy and visit http://localhost:8080, then we can see this simple but stylish animation. It shows us that all of jQuery is available for us to work with on our application. Using Bower instead of WebJars WebJars isn't the only option when it comes to adding JavaScript to our app. More sophisticated UI developers might use Bower (http://bower.io), a popular JavaScript library management tool. WebJars are useful for Java developers, but not every library has been bundled as a WebJar. There is also a huge community of frontend developers more familiar with Bower and NodeJS that will probably prefer using their standard tool chain to do their jobs. We'll see how to plug that into our app. First, it's important to know some basic options. Spring Boot supports serving up static web resources from the following paths: /META-INF/resources/ /resources/ /static/ /public/ To craft a Bower-based app with Spring Boot, we first need to craft a .bowerrc file in the same folder we plan to create our Spring Boot CLI application. Let's pick public/ as the folder of choice for JavaScript modules and put it in this file, as shown in the following code: {"directory": "public/"} Do I have to use public? No. Again, you can pick any of the folders listed previously and Spring Boot will serve up the code. It's a matter of taste and semantics. Our first step towards a Bower-based app is to define our project by answering a series of questions (this only has to be done once): $ bower init[?] name: app_with_bower[?] version: 0.1.0[?] description: Learning Spring Boot - bower sample[?] main file:[?] what types of modules does this package expose? amd[?] keywords:[?] authors: Greg Turnquist <gturnquist@pivotal.io>[?] license: ASL[?] homepage: http://blog.greglturnquist.com/category/learning-springboot[?] set currently installed components as dependencies? No[?] add commonly ignored files to ignore list? Yes[?] would you like to mark this package as private which prevents it frombeing accidentally published to the registry? Yes...[?] Looks good? Yes Now that we have set our project, let's do something simple such as install jQuery with the following command: $ bower install jquery --savebower jquery#* cached git://github.com/jquery/jquery.git#2.1.1bower jquery#* validate 2.1.1 against git://github.com/jquery/jquery.git#* These two commands will have created the following bower.json file: {"name": "app_with_bower","version": "0.1.0","authors": ["Greg Turnquist <gturnquist@pivotal.io>"],"description": "Learning Spring Boot - bower sample","license": "ASL","homepage": "http://blog.greglturnquist.com/category/learningspring-boot","private": true,"ignore": ["**/.*","node_modules","bower_components","public/","test","tests"],"dependencies": {"jquery": "~2.1.1"}} It will also have installed jQuery 2.1.1 into our app with the following directory structure: public└── jquery├── MIT-LICENSE.txt├── bower.json└── dist├── jquery.js└── jquery.min.js We must include --save (two dashes) whenever we install a module. This ensures that our bower.json file is updated at the same time, allowing us to rebuild things if needed. The altered version of our app with WebJars removed should now look like this: @Grab("thymeleaf-spring4")@Controllerclass ModernApp {def chapters = ["Quick Start With Groovy","Quick Start With Java","Debugging and Managing Your App","Data Access with Spring Boot","Securing Your App"]@RequestMapping("/")def home(@RequestParam(value="name", defaultValue="World")String n) {new ModelAndView("modern_with_bower").addObject("name", n).addObject("chapters", chapters)}} The view name has been changed to modern_with_bower, so it doesn't collide with the previous template if found in the same folder. This version of the template, templates/modern_with_bower.html, should look like this: <html><head><title>Learning Spring Boot - Chapter 1</title><script src="jquery/dist/jquery.min.js"></script><script>$(document).ready(function() {$('p').animate({fontSize: '48px',}, "slow");});</script></head><body><p th_text="'Hello, ' + ${name}"></p><ol><li th_each="chapter : ${chapters}"th:text="${chapter}"></li></ol></body></html> The path to jquery is now jquery/dist/jquery.min.js. The rest is the same as the WebJars example. We just launch the app with spring run modern_with_bower.groovy and navigate to http://localhost:8080. (Might need to refresh the page to ensure loading of the latest HTML.) The animation should work just the same. The options shown in this section can quickly give us a taste of how easy it is to use popular JavaScript tools with Spring Boot. We don't have to fiddle with messy tool chains to achieve a smooth integration. Instead, we can use them the way they are meant to be used. What about an app that is all frontend with no backend? Perhaps we're building an app that gets all its data from a remote backend. In this age of RESTful backends, it's not uncommon to build a single page frontend that is fed data updates via AJAX. Spring Boot's Groovy support provides the perfect and arguably smallest way to get started. We do so by creating pure_javascript.groovy, as shown in the following code: @Controllerclass JsApp { } That doesn't look like much, but it accomplishes a lot. Let's see what this tiny fragment of code actually does for us: The @Controller annotation, like @RestController, causes Spring Boot to auto-configure Spring MVC. Spring Boot will launch an embedded Apache Tomcat server. Spring Boot will serve up static content from resources, static, and public. Since there are no Spring MVC routes in this tiny fragment of code, things will fall to resource resolution. Next, we can create a static/index.html page as follows: <html>Greetings from pure HTML which can, in turn, load JavaScript!</html> Run spring run pure_javascript.groovy and navigate to http://localhost:8080. We will see the preceding plain text shown in our browser as expected. There is nothing here but pure HTML being served up by our embedded Apache Tomcat server. This is arguably the lightest way to serve up static content. Use spring jar and it's possible to easily bundle up our client-side app to be installed anywhere. Spring Boot's support for static HTML, JavaScript, and CSS opens the door to many options. We can add WebJar annotations to JsApp or use Bower to introduce third-party JavaScript libraries in addition to any custom client-side code. We might just manually download the JavaScript and CSS. No matter what option we choose, Spring Boot CLI certainly provides a super simple way to add rich-client power for app development. To top it off, RESTful backends that are decoupled from the frontend can have different iteration cycles as well as different development teams. You might need to configure CORS (http://spring.io/understanding/CORS) to properly handle making remote calls that don't go back to the original server. Adding production-ready support features So far, we have created a Spring MVC app with minimal code. We added views and JavaScript. We are on the verge of a production release. Before deploying our rapidly built and modernized web application, we might want to think about potential issues that might arise in production: What do we do when the system administrator wants to configure his monitoring software to ping our app to see if it's up? What happens when our manager wants to know the metrics of people hitting our app? What are we going to do when the Ops center supervisor calls us at 2:00 a.m. and we have to figure out what went wrong? The last feature we are going to introduce in this article is Spring Boot's Actuator module and CRaSH remote shell support (http://www.crashub.org). These two modules provide some super slick, Ops-oriented features that are incredibly valuable in a production environment. We first need to update our previous code (we'll call it ops.groovy), as shown in the following code: @Grab("spring-boot-actuator")@Grab("spring-boot-starter-remote-shell")@Grab("org.webjars:jquery:2.1.1")@Grab("thymeleaf-spring4")@Controllerclass OpsReadyApp {@RequestMapping("/")def home(@RequestParam(value="name", defaultValue="World")String n) {new ModelAndView("modern").addObject("name", n)}} This app is exactly like the WebJars example with two key differences: it adds @Grab("spring-boot-actuator") and @Grab("spring-boot-starter-remote-shell"). When you run this version of our app, the same business functionality is available that we saw earlier, but there are additional HTTP endpoints available: Actuator endpoint Description /autoconfig This reports what Spring Boot did and didn't auto-configure and why /beans This reports all the beans configured in the application context (including ours as well as the ones auto-configured by Boot) /configprops This exposes all configuration properties /dump This creates a thread dump report /env This reports on the current system environment /health This is a simple endpoint to check life of the app /info This serves up custom content from the app /metrics This shows counters and gauges on web usage /mappings This gives us details about all Spring MVC routes /trace This shows details about past requests Pinging our app for general health Each of these endpoints can be visited using our browser or using other tools such as curl. For example, let's assume we ran spring run ops.groovy and then opened up another shell. From the second shell, let's run the following curl command: $ curl localhost:8080/health{"status":"UP"} This immediately solves our first need listed previously. We can inform the system administrator that he or she can write a management script to interrogate our app's health. Gathering metrics Be warned that each of these endpoints serves up a compact JSON document. Generally speaking, command-line curl probably isn't the best option. While it's convenient on *nix and Mac systems, the content is dense and hard to read. It's more practical to have: A JSON plugin installed in our browser (such as JSONView at http://jsonview.com) A script that uses a JSON parsing library if we're writing a management script (such as Groovy's JsonSlurper at http://groovy.codehaus.org/gapi/groovy/json/JsonSlurper.html or JSONPath at https://code.google.com/p/json-path) Assuming we have JSONView installed, the following screenshot shows a listing of metrics: It lists counters for each HTTP endpoint. According to this, /metrics has been visited four times with a successful 200 status code. Someone tried to access /foo, but it failed with a 404 error code. The report also lists gauges for each endpoint, reporting the last response time. In this case, /metrics took 2 milliseconds. Also included are some memory stats as well as the total CPUs available. It's important to realize that the metrics start at 0. To generate some numbers, you might want to first click on some links before visiting /metrics. The following screenshot shows a trace report: It shows the entire web request and response for curl localhost:8080/health. This provides a basic framework of metrics to satisfy our manager's needs. It's important to understand that metrics gathered by Spring Boot Actuator aren't persistent across application restarts. So to gather long-term data, we have to gather them and then write them elsewhere. With these options, we can perform the following: Write a script that gathers metrics every hour and appends them to a running spreadsheet somewhere else in the filesystem, such as a shared drive. This might be simple, but probably also crude. To step it up, we can dump the data into a Hadoop filesystem for raw collection and configure Spring XD (http://projects.spring.io/spring-xd/) to consume it. Spring XD stands for Spring eXtreme Data. It is an open source product that makes it incredibly easy to chain together sources and sinks comprised of many components, such as HTTP endpoints, Hadoop filesystems, Redis metrics, and RabbitMQ messaging. Unfortunately, there is no space to dive into this subject. With any monitoring, it's important to check that we aren't taxing the system too heavily. The same container responding to business-related web requests is also serving metrics data, so it will be wise to engage profilers periodically to ensure that the whole system is performing as expected. Detailed management with CRaSH So what can we do when we receive that 2:00 a.m. phone call from the Ops center? After either coming in or logging in remotely, we can access the convenient CRaSH shell we configured. Every time the app launches, it generates a random password for SSH access and prints this to the local console: 2014-06-11 23:00:18.822 ... : Configuring property ssh.port=2000 fromproperties2014-06-11 23:00:18.823 ... : Configuring property ssh.authtimeout=600000 fro...2014-06-11 23:00:18.824 ... : Configuring property ssh.idletimeout=600000 fro...2014-06-11 23:00:18.824 ... : Configuring property auth=simple fromproperties2014-06-11 23:00:18.824 ... : Configuring property auth.simple.username=user f...2014-06-11 23:00:18.824 ... : Configuring property auth.simple.password=bdbe4a... We can easily see that there's SSH access on port 2000 via a user if we use this information to log in: $ ssh -p 2000 user@localhostPassword authenticationPassword:. ____ _ __ _ _/\ / ___'_ __ _ _(_)_ __ __ _ ( ( )___ | '_ | '_| | '_ / _' | \/ ___)| |_)| | | | | || (_| | ) ) ) )' |____| .__|_| |_|_| |___, | / / / /=========|_|==============|___/=/_/_/_/:: Spring Boot :: (v1.1.6.RELEASE) on retina> There's a fistful of commands: help: This gets a listing of available commands dashboard: This gets a graphic, text-based display of all the threads, environment properties, memory, and other things autoconfig: This prints out a report of which Spring Boot auto-configuration rules were applied and which were skipped (and why) All of the previous commands have man pages: > man autoconfigNAMEautoconfig - Display auto configuration report fromApplicationContextSYNOPSISautoconfig [-h | --help]STREAMautoconfig <java.lang.Void, java.lang.Object>PARAMETERS[-h | --help]Display this help message... There are many commands available to help manage our application. More details are available at http://www.crashub.org/1.3/reference.html. Summary In this article, we learned about modernizing our Spring Boot app with JavaScript and adding production-ready support features. We plugged in Spring Boot's Actuator module as well as the CRaSH remote shell, configuring it with metrics, health, and management features so that we can monitor it in production by merely adding two lines of extra code. Resources for Article: Further resources on this subject: Getting Started with Spring Security [Article] Spring Roo 1.1: Working with Roo-generated Web Applications [Article] Spring Security 3: Tips and Tricks [Article]

(For more resources related to this topic, see here.) System requirements Node runs on POSIX-like operating systems, the various UNIX derivatives (Solaris, and so on), or workalikes (Linux, Mac OS X, and so on), as well as on Microsoft Windows, thanks to the extensive assistance from Microsoft. Indeed, many of the Node built-in functions are direct corollaries to POSIX system calls. It can run on machines both large and small, including the tiny ARM devices such as the Raspberry Pi microscale embeddable computer for DIY software/hardware projects. Node is now available via package management systems, limiting the need to compile and install from source. Installing from source requires having a C compiler (such as GCC), and Python 2.7 (or later). If you plan to use encryption in your networking code you will also need the OpenSSL cryptographic library. The modern UNIX derivatives almost certainly come with these, and Node's configure script (see later when we download and configure the source) will detect their presence. If you should have to install them, Python is available at http://python.org and OpenSSL is available at http://openssl.org. Installing Node using package managers The preferred method for installing Node, now, is to use the versions available in package managers such as apt-get, or MacPorts. Package managers simplify your life by helping to maintain the current version of the software on your computer and ensuring to update dependent packages as necessary, all by typing a simple command such as apt-get update. Let's go over this first. Installing on Mac OS X with MacPorts The MacPorts project (http://www.macports.org/) has for years been packaging a long list of open source software packages for Mac OS X, and they have packaged Node. After you have installed MacPorts using the installer on their website, installing Node is pretty much this simple: $ sudo port search nodejs nodejs @0.10.6 (devel, net) Evented I/O for V8 JavaScript nodejs-devel @0.11.2 (devel, net) Evented I/O for V8 JavaScript Found 2 ports. -- npm @1.2.21 (devel) node package manager $ sudo port install nodejs npm .. long log of downloading and installing prerequisites and Node Installing on Mac OS X with Homebrew Homebrew is another open source software package manager for Mac OS X, which some say is the perfect replacement for MacPorts. It is available through their home page at http://mxcl.github.com/homebrew/. After installing Homebrew using the instructions on their website, using it to install Node is as simple as this: $ brew search node leafnode node $ brew install node ==> Downloading http://nodejs.org/dist/v0.10.7/node-v0.10.7.tar.gz ######################################################################## 100.0% ==> ./configure –prefix=/usr/local/Cellar/node/0.10.7 ==> make install ==> Caveats Homebrew installed npm. We recommend prepending the following path to your PATH environment variable to have npm-installed binaries picked up: /usr/local/share/npm/bin ==> Summary /usr/local/Cellar/node/0.10.7: 870 files, 16M, built in 21.9 minutes Installing on Linux from package management systems While it's still premature for Linux distributions or other operating systems to prepackage Node with their OS, that doesn't mean you cannot install it using the package managers. Instructions on the Node wiki currently list packaged versions of Node for Debian, Ubuntu, OpenSUSE, and Arch Linux. See: https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager For example, on Debian sid (unstable): # apt-get update # apt-get install nodejs # Documentation is great. And on Ubuntu: # sudo apt-get install python-software-properties # sudo add-apt-repository ppa:chris-lea/node.js # sudo apt-get update # sudo apt-get install nodejs npm We can expect in due course that the Linux distros and other operating systems will routinely bundle Node into the OS like they do with other languages today. Installing the Node distribution from nodejs.org The nodejs.org website offers prebuilt binaries for Windows, Mac OS X, Linux, and Solaris. You simply go to the website, click on the Install button, and run the installer. For systems with package managers, such as the ones we've just discussed, it's preferable to use that installation method. That's because you'll find it easier to stay up-to-date with the latest version. However, on Windows this method may be preferred. For Mac OS X, the installer is a PKG file giving the typical installation process. For Windows, the installer simply takes you through the typical install wizard process. Once finished with the installer, you have a command line tool with which to run Node programs. The pre-packaged installers are the simplest ways to install Node, for those systems for which they're available. Installing Node on Windows using Chocolatey Gallery Chocolatey Gallery is a package management system, built on top of NuGet. Using it requires a Windows machine modern enough to support the Powershell and the .NET Framework 4.0. Once you have Chocolatey Gallery (http://chocolatey.org/), installing Node is as simple as this: C:> cinst install nodejs Installing the StrongLoop Node distribution StrongLoop (http://strongloop.com) has put together a supported version of Node that is prepackaged with several useful tools. This is a Node distribution in the same sense in which Fedora or Ubuntu are Linux distributions. StrongLoop brings together several useful packages, some of which were written by StrongLoop. StrongLoop tests the packages together, and distributes installable bundles through their website. The packages in the distribution include Express, Passport, Mongoose, Socket.IO, Engine.IO, Async, and Request. We will use all of those modules in this book. To install, navigate to the company home page and click on the Products link. They offer downloads of precompiled packages for both RPM and Debian Linux systems, as well as Mac OS X and Windows. Simply download the appropriate bundle for your system. For the RPM bundle, type the following: $ sudo rpm -i bundle-file-name For the Debian bundle, type the following: $ sudo dpkg -i bundle-file-name The Windows or Mac bundles are the usual sort of installable packages for each system. Simply double-click on the installer bundle, and follow the instructions in the install wizard. Once StrongLoop Node is installed, it provides not only the nodeand npmcommands (we'll go over these in a few pages), but also the slnodecommand. That command offers a superset of the npmcommands, such as boilerplate code for modules, web applications, or command-line applications. Installing from source on POSIX-like systems Installing the pre-packaged Node distributions is currently the preferred installation method. However, installing Node from source is desirable in a few situations: It could let you optimize the compiler settings as desired It could let you cross-compile, say for an embedded ARM system You might need to keep multiple Node builds for testing You might be working on Node itself Now that you have the high-level view, let's get our hands dirty mucking around in some build scripts. The general process follows the usual configure, make, and makeinstallroutine that you may already have performed with other open source software packages. If not, don't worry, we'll guide you through the process. The official installation instructions are in the Node wiki at https://github.com/joyent/node/wiki/Installation. Installing prerequisites As noted a minute ago, there are three prerequisites, a C compiler, Python, and the OpenSSL libraries. The Node installation process checks for their presence and will fail if the C compiler or Python is not present. The specific method of installing these is dependent on your operating system. These commands will check for their presence: $ cc --version i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3) Copyright (C) 2007 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. $ python Python 2.6.6 (r266:84292, Feb 15 2011, 01:35:25) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> Installing developer tools on Mac OS X The developer tools (such as GCC) are an optional installation on Mac OS X. There are two ways to get those tools, both of which are free. On the OS X installation DVD is a directory labeled Optional Installs, in which there is a package installer for—among other things—the developer tools, including Xcode. The other method is to download the latest copy of Xcode (for free) from http://developer.apple.com/xcode/. Most other POSIX-like systems, such as Linux, include a C compiler with the base system. Installing from source for all POSIX-like systems First, download the source from http://nodejs.org/download. One way to do this is with your browser, and another way is as follows: $ mkdir src $ cd src $ wget http://nodejs.org/dist/v0.10.7/node-v0.10.7.tar.gz $ tar xvfz node-v0.10.7.tar.gz $ cd node-v0.10.7 The next step is to configure the source so that it can be built. It is done with the typical sort of configure script and you can see its long list of options by running the following: $ ./configure –help. To cause the installation to land in your home directory, run it this way: $ ./configure –prefix=$HOME/node/0.10.7 ..output from configure If you want to install Node in a system-wide directory simply leave off the -prefixoption, and it will default to installing in /usr/local. After a moment it'll stop and more likely configure the source tree for installation in your chosen directory. If this doesn't succeed it will print a message about something that needs to be fixed. Once the configure script is satisfied, you can go on to the next step. With the configure script satisfied, compile the software: $ make .. a long log of compiler output is printed $ make install If you are installing into a system-wide directory do the last step this way instead: $ make $ sudo make install Once installed you should make sure to add the installation directory to your PATHvariable as follows: $ echo 'export PATH=$HOME/node/0.10.7/bin:${PATH}' >>~/.bashrc $ . ~/.bashrc For cshusers, use this syntax to make an exported environment variable: $ echo 'setenv PATH $HOME/node/0.10.7/bin:${PATH}' >>~/.cshrc $ source ~/.cshrc This should result in some directories like this: $ ls ~/node/0.10.7/ bin include lib share $ ls ~/node/0.10.7/bin node node-waf npm Maintaining multiple Node installs simultaneously Normally you won't have multiple versions of Node installed, and doing so adds complexity to your system. But if you are hacking on Node itself, or are testing against different Node releases, or any of several similar situations, you may want to have multiple Node installations. The method to do so is a simple variation on what we've already discussed. If you noticed during the instructions discussed earlier, the –prefixoption was used in a way that directly supports installing several Node versions side-by-side in the same directory: $ ./configure –prefix=$HOME/node/0.10.7 And: $ ./configure –prefix=/usr/local/node/0.10.7 This initial step determines the install directory. Clearly when Version 0.10.7, Version 0.12.15, or whichever version is released, you can change the install prefix to have the new version installed side-by-side with the previous versions. To switch between Node versions is simply a matter of changing the PATHvariable (on POSIX systems), as follows: $ export PATH=/usr/local/node/0.10.7/bin:${PATH} It starts to be a little tedious to maintain this after a while. For each release, you have to set up Node, npm, and any third-party modules you desire in your Node install; also the command shown to change your PATHis not quite optimal. Inventive programmers have created several version managers to make this easier by automatically setting up not only Node, but npmalso, and providing commands to change your PATHthe smart way: Node version manager: https://github.com/visionmedia/n Nodefront, aids in rapid frontend development: http://karthikv.github.io/nodefront/

Introduction This article guides you through the process of creating tilemaps and changing them at runtime in the Duality game engine. A Few Words about the Duality Game Engine Duality is an open source, Windows-only 2D game engine based on OpenGL and written entirely in C#. Despite being a non-commercial project, it is a rather mature product because it has been around since 2011. It has also proved to be capable of delivering professional games. Obviously, the feature bucket is somewhat limited compared to the major proprietary engines. On the other hand, this fact makes Duality an excellent learning tool, because the user often needs to implement logic on a lower level and also has the opportunity to inspect the source code of the engine. Thus I would recommend this engine to game developers with an intermediate skill level, who'd like to have a take on open source and are not afraid of learning by doing. This article does not describe the inner workings of Duality in detail because these are well-explained on the GitHub wiki of the project. If you are not familiar with the engine, please consider reviewing some of the key concepts described there. The repository containing the finished project is available on GitHub. Required tools First things first: Duality editor only supports Windows at the moment. You will need a C# compiler and a text editor. Visual Studio 2013 or higher is recommended, but Xamarin Studio/MonoDevelop also works. Duality itself is written in C# 4, but this tutorial uses language features introduced in C# 6. Creating a new project and initializing the Tilemap plugin Downloading the Duality binaries The latest binary package of the engine is available on its main site. Unlike most game engines, it does not need a setup process. In fact, every single project is a self-contained application, including the editor. This might seem strange at first, but this architecture makes version control, migration, and having different engine versions for different projects very easy. On top of this, it is also a self-building application. When you unzip the downloaded file, it contains only one executable: DualityEditor.exe. At its first run, it downloads the required packages from the main repository. It is based on the industry-standard NuGet format, and the packages are actually stored on NuGet.org. After accepting the license agreement and proceeding through the automated package install, the following screen should show up: Dualitor The usage of the editor application named Dualitor is intuitive; however, if you are not familiar with it yet, the quick start guide might be worth checking out. Adding the Tilemaps plugin to the project Duality has a modular architecture. Thus, every piece of distinct functionality is organized in modules, or, using the official terminology, plugins. Practically, every plugin is a .NET assembly that Duality loads and uses. There are two types of plugins: core plugins provide in-game functionality and services and are distributed with the game. On the other hand, editor plugins are used by Dualitor, and usually provide some sort of editing or content management functionality which aids the developer in creating the game. Now let's install the Tilemaps plugin. It's done via the Package Management window, which is available from the File menu. The following steps describe the installation process: Select the 'Online Repository' item from the 'View' dropdown. Select the Tilemaps (Editor) entry from the list. Since it depends on the Tilemaps (Core) package, the latter will be installed as well. Click on 'Install'. This downloads the two packages from the repository. Click on 'Apply'. Dualitor restarts and loads the plugins. Installing the Tilemaps plugin Tileset resource and Tilemap GameObject Creating the Tileset resource Resources are the asset holder objects in Duality. They can be simple imported types such as bitmap information, but some, like the Tileset resource we are going to use now, are generated by the editor. It contains information about rendering, collision, and autotiling properties, as well as a reference to the actual pixel data to be rendered. This pixel data is contained in another resource: the Pixmap. Let's start with a very simple tileset that consists of two solid color tiles, sized 32 x 32. Tileset Download this .png file, and drop it in the Project View. A new Pixmap is created. Next, create a new Tileset from the context menu of the Project View: Add Tileset At this point, the Tileset Resource does not know which Pixmap it should use. First, open the Tileset Editor window from the View menu on the menu bar. The same behavior is invoked when the Tileset is double-clicked in the Project View. The following steps describe how to establish this link: Select the newly created Tileset in the Tileset Editor. Add a new Visual Layer by clicking on the little green plus button. Select the 'Main Texture' layer, if it's not already selected. Drag and drop the Pixmap named 'tileset' into the 'SourceData' area. Since the default value of the tile size is already 32 x 32, there is no need to modify that. Click on 'Apply' to recompile the Resource. A quicker way to do this is by just selecting the 'Create Tileset' option from the Pixmap's context menu, but it is always nice to know what happens in the background. Linking the Pixmap and the Tileset resource Instantiating a Tilemap Duality, similar to many engines, uses a Scene-GameObject-Component model of object hierarchy. To display a tilemap in the scene, a GameObject is needed, with two components attached to it: a Tilemap and a TilemapRenderer. The former contains the actual tilemap information, that is, which tile is located at a specific position. Creating this GameObject is very easy; just drag and drop the Tileset Resource from the Project View to the Scene View. Switch back to the Camera #0 window from the Tileset Editor, and you should see a large green rectangle, which is the tilemap itself. Notice that a new GameObject called 'TilemapRenderer' appears in the Scene View, and it has 3 components attached: a Transform, a Tilemap, and a TilemapRenderer. To edit the Tilemap, the Camera #0 window mode has to be set to 'Tilemap Editor'. After doing that, a new window named 'Tile Palette' appears. The Tileset can be edited with multiple tools—feel free to experiment with them! Editing the Tilemap To be continued In the second part of this tutorial, some C# coding is involved. We will create a custom component to change the tilemap dynamically. Author Lőrinc Serfőző is a software engineer at Graphisoft, the company behind the BIM solution ArchiCAD. He is studying mechatronics engineering at the Budapest University of Technology and Economics, an interdisciplinary field between the more traditional mechanical engineering, electrical engineering, and informatics, and has quickly grown a passion toward software development. He is a supporter of open source and contributes to the C# and OpenGL-based Duality game engine, creating free plugins and tools for users.

Microsoft SQL Server 2008 High Availability Minimize downtime, speed up recovery, and achieve the highest level of availability and reliability for SQL server applications by mastering the concepts of database mirroring,log shipping,clustering, and replication  Install various SQL Server High Availability options in a step-by-step manner  A guide to SQL Server High Availability for DBA aspirants, proficient developers and system administrators  Learn the pre and post installation concepts and common issues you come across while working on SQL Server High Availability  Tips to enhance performance with SQL Server High Availability  External references for further study Introduction Mr. Young, who is the Principal DBA in XY Incorporation, a large manufacturing company, was asked to come up with a solution that could serve his company's needs to make a database server highly available, without a manual or minimal human intervention. He was also asked to keep in mind the limited budget the company has for the financial year. After careful research, he has come up with an idea to go with Database Mirroring as it provides an option of Automatic Failover—a cost effective solution. He has prepared a technical document for the management and peers, in order to make them understand how it works, based on tests he performed on virtual test servers. What is Database Mirroring Database Mirroring is an option that can be used to cater to the business need, in order to increase the availability of SQL Server database as standby, for it to be used as an alternate production server in the case of any emergency. As its name suggests, mirroring stands for making an exact copy of the data. Mirroring can be done onto a disk, website, or somewhere else. Similarly, Microsoft has introduced Database Mirroring with the launch of SQL Server 2005 post SP1, which performs the same function—making an exact copy of the database between two physically separate database servers. As Mirroring is a database-wide feature, it can be implemented per database instead of implementing it server wide. Disk Mirroring is a technology wherein data is stored on physically separate but identical hard disks at the same time called hardware array disk 1 or RAID 1. Different components of the Database Mirroring To install Database Mirroring, there are three components that are required. They are as follows: Principal Server: This is the database server that will send out the data to the participant server (we'll call it secondary/standby server) in the form of transactions. Secondary Server: This is the database server that receives all the transactions that are sent by the Principal Server in order to keep the database identical. Witness Server (optional): This server will continuously monitor the Principal and Secondary Server and will enable automatic failover. This is an optional server. To have the automatic failover feature, the Principal Server should be in the synchronous mode. How Database Mirroring works In Database Mirroring, every server is a known partner and they complement each other as Principal and Mirror. There will be only one Principal and only one Mirror at any given time. In reality, DML operations that are performed on the Principal Server are all re-performed at the Mirror server. As we all know, the data is written into the Log Buffer before it is written into data pages. Database Mirroring sends data that is written into Principal Server's Log Buffer simultaneously to the Mirror database. All these transactions are sent in a sequential manner and as quickly as possible. There are two different operating modes at which Database Mirroring operates—asynchronous and synchronous. Asynchronous a.k.a. High Performance mode The transactions are sent to the Secondary Server as soon as they are written into the Log Buffer. In this mode of operation, the data is first committed at the Principal Server before it actually is written into the Log Buffer of the Secondary Server. Hence this mode is called High Performance mode, but at the same time, it introduces a chance of data loss. Synchronous a.k.a. High Safety mode The transactions are sent to the Secondary Server as soon as they are written to the Log Buffer. These transactions are then committed simultaneously at both the ends. Prerequisites Let's now have a look at the prerequisite to have Database Mirroring in place. Please be cautious with the prerequisites, as a single missed requisite would result in a failure in installation. Recovery Mode: To implement Database Mirroring, the database should be in the Full Recovery mode. Initialization of database: The database on which to install Database Mirroring should be present in the mirror database. To achieve this, we can restore the most recent backup, followed by the transaction log with the NORECOVERY option. Database name: Ensure that the database name is the same for both, the principal as well as the mirror database. Compatibility: Ensure that the partner servers are on the same edition of the SQL Server. Database Mirroring is supported by the Standard and Enterprise edition. Synchronous mode with Automatic failover is an Enterprise-only feature. Disk space: Ensure that we have ample space available for the mirror database. Service accounts: Ensure that the service accounts that we have used are domain accounts and they have the required permission, that is, CONNECT permission to the endpoints. Listener port: These are TCP ports on which a Database Mirroring session is established between the Principal and Mirror server. Endpoints: These are the objects that are dedicated to Database Mirroring and enable the SQL Server to communicate over the network. Authentication: While both the Principal and Mirror servers talk to each other, they should authenticate each other. For this, the accounts that we use—local accounts or domain accounts—should have login and send message permissions. If the accounts we use are using local logins as service accounts, we must use Certificates to authenticate a connection request.

Let's get started. Setting up Microsoft Messaging Queue Service (MSMQ) on your mobile device MSMQ is not installed by default on the Windows Mobile platform. This section will guide you on how to install MSMQ on your mobile device or device emulator. You will first need to download the Redistributable Server Components for Windows Mobile 5.0 package (which can also be used for Windows Mobile 6.0) from this location: href="http://www.microsoft.com/downloads/details.aspx?FamilyID=cdfd2bb2-fa13-4062-b8d1-4406ccddb5fd&displaylang=en After downloading and unzipping this file, you will have access to the MSMQ.arm.cab file in the following folder: Optional Windows Mobile 5.0 Server Componentsmsmq Copy this file via ActiveSync to your mobile device and run it on the device. This package contains two applications (and a bunch of other DLL components) that you will be using frequently on the device: msmqadm.exe:This is the command line tool that allows you to start and stop the MSMQ service on the mobile device and also configure MSMQ settings. It can also be invoked programmatically from code. visadm.exe: This tool does the same thing as above, but provides a visual interface. These two files will be unpacked into the Windows folder of your mobile device. The following DLL files will also be unpacked into the Windows folder: msmqd.dll msmqrt.dll Verify that these files exist. The next thing you need to do is to change the name of your device (if you haven't done so earlier). In most cases, you are probably using the Windows Mobile Emulator, which comes with an unassigned device name by default. To change your device name, navigate to Settings | System | About on your mobile device. You can change the device name in the Device ID tab. At this point, you have the files for MSMQ unpacked, but it isn't exactly installed yet. To do this, you must invoke either msmqadm.exe or visadm.exe. Launch the following application: Windowsvisadm.exe A pop-up window will appear. This window contains a text box and a Run button that allows you to type in the desired command and to execute it. The first command you need to issue is the register install command. Type in the command and click the Run button. No message will be displayed in the window. This command will install MSMQ (as a device driver) on your device. Run the following commands in the given order next (one after the other): MSMQ Command Name Purpose register You will need to run the register command one more time (without the install keyword) to create the MSMQ configuration keys in the registry. enable binary This command enables the proprietary MSMQ binary protocol to send messages to remote queues. enable srmp This command enables SRMP (SOAP Reliable Messaging Protocol), for sending messages to remote queues over HTTP. start This command starts the MSMQ service   Verify that the MSMQ service has been installed successfully by clicking on the Shortcuts button and then clicking the Verify button in the ensuing pop-up window. You will be presented with a pop-up dialog as shown in the following screenshot: MSMQ log information If you scroll down in this same window above, you will find the Base Dir path, which contains the MSMQ auto-generated log file. This log file, named MQLOGFILE by default, contains useful MSMQ related information and error messages. After you've done the preceding steps, you will need to do a soft-reset of your device. The MSMQ service will automatically start upon boot up.

(For more resources related to this topic, see here.) A typical configuration might look something as follows: To use the Avro Source, you specify the type property with a value of avro. You need to provide a bind address and port number to listen on: collector.sources=av1collector.sources.av1.type=avrocollector.sources.av1.bind=0.0.0.0collector.sources.av1.port=42424collector.sources.av1.channels=ch1collector.channels=ch1collector.channels.ch1.type=memorycollector.sinks=k1collector.sinks.k1.type=hdfscollector.sinks.k1.channel=ch1collector.sinks.k1.hdfs.path=/path/in/hdfs Here we have configured the agent on the right that listens on port 42424, uses a memory channel, and writes to HDFS. Here I've used the memory channel for brevity of this example configuration. Also, note that I've given this agent a different name, collector, just to avoid confusion. The agents on the left—feeding the collector tier—might have a configuration similar to this. I have left the sources off this configuration for brevity: client.channels=ch1client.channels.ch1.type=memoryclient.sinks=k1client.sinks.k1.type=avroclient.sinks.k1.channel=ch1client.sinks.k1.hostname=collector.example.comclient.sinks.k1.port=42424 The hostname value, collector.example.com, has nothing to do with the agent name on that machine, it is the host name (or you can use an IP) of the target machine with the receiving Avro Source. This configuration, named client, would be applied to both agents on the left assuming both had similar source configurations. Since I don't like single points of failure, I would configure two collector agents with the preceding configuration and instead set each client agent to round robin between the two using a sink group. Again, I've left off the sources for brevity: client.channels=ch1client.channels.ch1.type=memoryclient.sinks=k1 k2client.sinks.k1.type=avroclient.sinks.k1.channel=ch1client.sinks.k1.hostname=collectorA.example.comclient.sinks.k1.port=42424client.sinks.k2.type=avroclient.sinks.k2.channel=ch1client.sinks.k2.hostname=collectorB.example.comclient.sinks.k2.port=42424client.sinkgroups=g1client.sinkgroups.g1=k1 k2client.sinkgroups.g1.processor.type=load_balanceclient.sinkgroups.g1.processor.selector=round_robinclient.sinkgroups.g1.processor.backoff=true Summary In this article, we covered tiering data flows using the Avro Source and Sink. More information on this topic can be found in the book Apache Flume: Distributed Log Collection for Hadoop. Resources for Article : Further resources on this subject: Supporting hypervisors by OpenNebula [Article] Integration with System Center Operations Manager 2012 SP1 [Article] VMware View 5 Desktop Virtualization [Article]