Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-service-chaining
Packt
29 Oct 2013
5 min read
Save for later

Service Chaining

Packt
29 Oct 2013
5 min read
(For more resources related to this topic, see here.) Service Chaining is one of the common and popular use case for an ESB to support. To cater for a request from the client, the ESB may have to call a chain of business services. This is different from the Scatter and Gather pattern we covered before. The Scatter and Gather EIP explains how to handle a scenario where the incoming request has to be handled by multiple recipients and each recipient will reply back to form an aggregated response. Service Chaining does more intelligent decision making and simply does which is very much similar to service orchestration. Getting ready Let's take TravelManagementService as an example. To cater a travel arrangement request from the client, TravelManagementService has to call AirLineReservationService with the provided dates for travelling. If that succeeds it will call RentACarService and finally it will call the HotelReservationService to reserve a hotel. How to do it... Setup AirLineReservationService, RentACarService, and HotelReservationService. We need to have these business services up and running, so WSO2 ESB can route messages to them. Download AirLineReservationService.aar, RentACarService.aar, and HotelReservationService.aar from SAMPLE-6 and copy those to wso2esb-4.8.0/samples/axis2Server/repository/services/. If there is no services directory create one. Start simple Axis2Server from wso2esb-4.8.0/samples/axis2Server. Validate whether the three business services are up and running by accessing their WSDLs. Start WSO2 ESB and login with username as admin and password also as admin. ESB will start on https://localhost:9443. Get synapse.xml from SAMPLE-6, copy the content of it, go to Main | Manage | Service Bus | Source View, and paste it there, overriding the existing and click on Update. The above will create a proxy service called TravelManagementService in the ESB, which will route messages to the three business services running in simple Axis2Server. Go to Main | Manage | Services | List. You should be able to see TravelManagementService listed there. Now let's test our set up with cURL. $ curl -d @request.xml -H "Content-Type: application/soap+xml action=arrangeMyTravel" http://localhost:8280/services/TravelManagementService You can get request.xml from SAMPLE-6. How it works... Let's have a deep look at the synapse configuration. The following explains key configuration elements: When the request hits inSequence we have to first call the AirLineReservationService and need to process its response. If the response is true, only we will proceed to call RentACarService. The same applies when we call HotelReservationService. <send receive="rentACarSeq"> <endpoint> <address uri="…/AirLineReservationService"/> </endpoint> </send> The receive attribute in the <send/> mediator will make sure the response from calling endpoint goes to the <sequence/> defined with that particular name. In this case, the response here will hit the rentACarSeq sequence. If we do not define this attribute then the response will directly go to the outSequence. The rentACarSeq sequence has to process the response from the AirLineReservationService. If it's false, then it will call the processResponse sequence to send a message back to the client, without further processing. If the response is true, then rentACarSeq will proceed to call RentACarService. <sequence name="rentACarSeq"> <log level="full"/> <switch source="//ns:reserveAirLineResponse/ns:return"> <case regex="false"> <sequence key="processResponse"/> </case> <default> <send receive="reserveHotelSeq"> <endpoint> <address uri="…/RentACarService"/> </endpoint> </send> </default> </switch> </sequence> The processResponse sequence sends the response back to the client. Here we are using the <payloadFactory/> mediator to craft the response. Also when you want to send a response back to the client using the <send/> mediator we need to set the RESPONSE property to true prior to that. <sequence name="processResponse"> <payloadFactory> <format> <res:arrangeMyTravel > <res:return>$1</res:return> </res:arrangeMyTravel> </format> <args> <arg evaluator="xml" expression="$ctx:businessServiceResponse"/> </args> </payloadFactory> <property name="RESPONSE" value="true" scope="default" type="STRING"/> <send/> </sequence> A better approach to send back the response to the client was introduced from WSO2 ESB 4.8.0 onwards. There you need to use the <respond> mediator instead of the above. Summary So now you have a brief idea about WSO2 ESB. We also learnt how to cater a request from the client using Service Chaining. We've also learnt that the ESB removes point-to-point dependencies in your system to build a highly scalable and loosely coupled solution. And lastly, we have learnt the difference between Scatter and Gather EIP and Service Chaining. Resources for Article: Further resources on this subject: BizTalk: The ESB Management Portal [Article] Aggregate Services in ServiceMix JBI ESB [Article] Getting Started with Mule [Article]
Read more
  • 0
  • 0
  • 5687

article-image-gns3-orchestra
Packt
29 Oct 2013
9 min read
Save for later

The GNS3 orchestra

Packt
29 Oct 2013
9 min read
(For more resources related to this topic, see here.) You will observe multiple TCP connections and UDP pipes being created from both the GNS3 management console and your operating system's command line. To get a closer look at how the conductor works, open GNS3 to a new blank canvas and issue the command debug 3 in the GNS3 command console : => debug 3 As you open GNS3, the conductor readies the players awaiting your instructions. The moment you drag your first Cisco router onto the workspace, GNS3 spawns an instance of Dynamips and connects to it on port 7200. You can see this in two places: To reproduce the effects shown here, use a C7200 router image with 256MB RAM. By issuing a netstat–a command in a Windows/Linux/OS X command window: C: >netstat -an ... Proto Local Address Foreign Address State TCP 127.0.0.1:7200 0.0.0.0:0 LISTENING TCP 127.0.0.1:7200 127.0.0.1:49194 ESTABLISHED TCP 127.0.0.1:49194 127.0.0.1:7200 ESTABLISHED In the output of the GNS3 management console. The following output lines are abbreviated to conserve space: Hypervisor manager: connecting on 127.0.0.1:7200 Hypervisor manager: connected to hypervisor on 127.0.0.1 port 7200 Also in the output of the GNS3 management console , note the following (trimmed) lines: Hypervisor manager: hypervisor base UDP is 10001 …<snip>… PORT TRACKER: allocate port 2101 sending to dynamips at 127.0.0.1:7200 -> vm set_con_tcp_port R1 2101 returned -> ['100-OK'] PORT TRACKER: allocate port 2501 sending to dynamips at 127.0.0.1:7200 -> vm set_aux_tcp_port R1 2501 returned -> ['100-OK'] The UDPbase port I'll deal with shortly, but notice that GNS3 has told Dynamips to prepare to open ports 2101 and 2501 for console and AUX port communications respectively, which are the base ports defined in GNS3 Preferences for the Dynamips setting under the Dynamips tab. Also note that these ports are not yet opened, in the orchestral analogy you could say they are merely being tuned up at this stage. Next, add a second router and observe (in the console output) that the console and AUX port allocations have incremented by one, but there is no change to the base UDP port. sending to dynamips at 127.0.0.1:7200 -> vm set_con_tcp_port R2 2102 sending to dynamips at 127.0.0.1:7200 -> vm set_aux_tcp_port R2 2502 You are about to connect these two devices. Configure them with FastEthernet interfaces if necessary, or use the FastEthernet Add a link tool, and connect the two routers. Watch the console output for these lines: Connect link from R1 f1/0 to R2 f1/0 new base UDP port for dynamips at 127.0.0.1:7200 is now: 10002 new base UDP port for dynamips at 127.0.0.1:7200 is now: 10003 sending to dynamips at 127.0.0.1:7200 -> nio create_udp nio_udp0 10001 127.0.0.1 10002 sending to dynamips at 127.0.0.1:7200 -> nio create_udp nio_udp1 10002 127.0.0.1 10001 sending to dynamips at 127.0.0.1:7200 -> vm slot_add_nio_binding R1 1 0 nio_udp0 sending to dynamips at 127.0.0.1:7200 -> vm slot_add_nio_binding R2 1 0 nio_udp1 And on your host computer, netstat -an reveals: C:> netstat -an | find "1000" UDP 0.0.0.0:10001 *:* UDP 0.0.0.0:10002 *:* Understanding what is going on here is the key to understanding how Dynamips achieves communication between routers. What has just happened is that a UDP tunnel has been created between these two devices. UDP tunnel concept Links between devices in GNS3 is achieved using UDP tunnels. What this means in this scenario is that whenever R1 sends a frame from interface f1/0, the entire frame, including the Source MAC address, destination MAC address and payload, gets put inside a UDP packet with a source port of 10001 and a destination IP address:destination port of 127.0.0.1:10002 which means that the frame will end up at R2's f1/0 interface because it is bound to port 10002. The return frames take the reverse path: source port 10002, destination IP:port 127.0.0.1:10001. To illustrate this, I assigned an IP addresses of 1.1.1.1 and 1.1.1.2 to interface f1/0 on R1 and R2 respectively, then captured a ping packet on the link between R1 and R2 on the host computer's loopback interface. The Wireshark capture shown in the following screenshot shows a ping packet from 1.1.1.1 on its way to 1.1.1.2, but you can see that the entire layer 2 frame (1), including the layer 2 MAC addresses of R1 and R2 is encapsulated inside a UDP packet travelling from 127.0.0.1:10001 to 127.0.0.1:10002 (2). Another thing to note is that the first UDP port used was the Base UDP port defined in GNS3 Preferences, Dynamips settings under the Dynamips tab. Now would also be a good time to issue a show run command in the GNS3 management console window, to see how GNS3 is building up your topology.net file. => show runautostart = False [127.0.0.1:7200] workingdir = C:UserschrisAppDataLocalTempGNS3_rwftbworking udp = 10001 [[7200]] image = C:UserschrisGNS3Imagesc7200-p-mz.124-10a.image ram = 256 idlepc = 0x60750000 sparsemem = True ghostios = True [[ROUTER R1]] console = 2101 aux = 2501 slot1 = PA-2FE-TX f1/0 = R2 f1/0 [[ROUTER R2]] console = 2102 aux = 2502 slot1 = PA-2FE-TX f1/0 = R1 f1/0 Note that the amount of RAM set for each of these routers is 256MiB. Also recall that in the Hypervisor Manager settings previously shown, the Memory limit per hypervisor was set to 512MiB. Now add another router, and watch the console output, and check your host computer's TCP connections again with the netstat -an command. You will see of course: Hypervisor manager: connecting on 127.0.0.1:7201 and… TCP 127.0.0.1:7201 0.0.0.0:0 LISTENING This shows that a second hypervisor instance has been created, and allocated TCP port 7201 for communication. You will also see this reflected in the configuration information if you issue another a show run command in the GNS3 management console window. ...<Output omitted>... [127.0.0.1:7201] workingdir = C:UserschrisAppDataLocalTempGNS3_rwftbworking udp = 10101 This also reveals that the base UDP port for this hypervisor is 10101, recall that the value for the UDP incrementation in the Dynamips Hypervisor Manager setting page was 100, so the base UDP port for this instance of the hypervisor is 100 greater than the general base UDP port of 10001 for Dynamips. You can probably predict what UDP port numbers will be used then if you now connect R2 to R3 with a FastEthernet link. Make the link and see if your prediction was correct: sending to dynamips at 127.0.0.1:7200 -> nio create_udp nio_udp2 10003 127.0.0.1 10101 sending to dynamips at 127.0.0.1:7201 -> nio create_udp nio_udp3 10101 127.0.0.1 10003 sending to dynamips at 127.0.0.1:7200 -> vm slot_add_nio_binding R2 1 1 nio_udp2 sending to dynamips at 127.0.0.1:7201 -> vm slot_add_nio_binding R3 1 0 nio_udp3 Did you predict that the next connection would be made from port 10003 to 10101? Well done. But what if you add a switch or a hub? Add a generic Ethernet switch to the topology, and issue a show run command in the GNS3 management console window. You will notice that there is NO reference to the switch in the output, and in fact if you saved your topology at this point and loaded it later, there would be no switch in your topology. That is because the switch doesn't get allocated to a hypervisor until it has at least one connection to another item in the topology. The question is, since our topology has two hypervisors running, which hypervisor will be allocated the switch? Connect your recently added switch to R1. Observe what happens in the GNS3 management console, and issue another show run command in the GNS3 management console window. Here is what you are looking for: Firstly, you should see the connections being created. Note that the UDP port numbers are from the range allocated to the first hypervisor that was spawned, NOT the most recent hypervisor spawned. sending to dynamips at 127.0.0.1:7200 -> nio create_udp nio_udp4 10004 127.0.0.1 10005 sending to dynamips at 127.0.0.1:7200 -> nio create_udp nio_udp5 10005 127.0.0.1 10004 Secondly, in the topology description, you can see that SW1 has been assigned to the hypervisor running on TCP port 7200 which allocated UDP ports from the 10000+ range. What actually happens is that GNS3 assigns generic devices like switches, hubs, and clouds to the hypervisor to which the device is first connected. => show run autostart = False [127.0.0.1:7200] workingdir = C:UserschrisAppDataLocalTemp udp = 10001 [[7200]] ... [[ETHSW SW1]] 1 = access 1 R1 f1/1 [[ROUTER R1]] ... And finally, if you changed your Memory usage per hypervisor setting back on page 73, don't forget to change it back. I recommend setting it to 1024MiB . By now you are probably wondering how GNS3 and Dynamips deal with the other supported emulators: Qemu and Oracle VirtualBox . Conducting Qemu and VirtualBox Recall that Dynamips is a hypervisor used to initiate the spawning of Cisco router VMs (virtual machines) instances, and configure host communication to these VMs via console AUX and virtual network interfaces. Just like Dynamips, both Qemu and Oracle VirtualBox follow a similar hypervisor model, only in this case the hypervisor is "wrapped" to give it similar functionality to Dynamips. The wrappers for the hypervisors are called qemuwrapper and vboxwrapper respectively, and these wrappers listen on ports 10525 and 11525 as shown in the configuration options in GNS3 Preferences under the Qemu and VirtualBox settings. Trivia: Port 10525 was chosen by Thomas Pani when he wrote pemuwrapper, the wrapper for the PIX 525 emulator, port 1525 was already assigned by the IANA. Pemuwrapper evolved into qemuwrapper, and when Alexey Eromenko, alias "Technologov" wrote vboxwrapper he simply added 1000 to the qemuwrapper port number. Again, just like Dynamips, you can run qemuwrapper and vboxwrapper as standalone applications, then telnet to port 127.0.0.1:10525 or 127.0.0.1:11525 to issue commands like qemu version or vbox version or the commands to spawn a virtual machine if you bothered to learn the syntax. Again, like Dynamips, qemuwrapper and vboxwrapper direct the TCP port to be used for console connections and UDP port for UDP tunnel connections, the base values for these can also be found in GNS3 Preferences under the Qemu and VirtualBox settings. But unlike Dynamips, the wrapper is NOT the hypervisor as well. The hypervisor is Qemu or VirtualBox, so these applications had to be compiled to allow communication via UDP tunnel interfaces. In the case of Qemu prior to Version 1.1, this required a specially compiled version. The GNS3 downloads page has links to the patched Version 0.11 that I used throughout this article. VirtualBox has built-in support for UDP tunnels. To see the full GNS3 orchestra playing, you can now add Qemu and VirtualBox devices to your topology and watch the GNS3 management console and check your TCP/UDP connections with the netstat -an command. As you watch the GNS3 management console you will see the hidden power of GNS3 beyond the GNS3 GUI as it conducts its orchestral sections of Dynamips, qemuwrapper, and vboxwrapper to play in harmony, and even see that they have their own sections in the GNS3 topology.net file, as can be seen by issuing a show run command in the GNS3 management console . Summary This article covers all we need to know about the GNS3 Orcehstra—the GNS3 GUI Resources for Article: Further resources on this subject: Editing attributes [Article] iPhone JavaScript: Installing Frameworks [Article] Visualizing my Social Graph with d3.js [Article]
Read more
  • 0
  • 0
  • 19040

article-image-openflow-controllers
Packt
29 Oct 2013
5 min read
Save for later

The OpenFlow Controllers

Packt
29 Oct 2013
5 min read
(For more resources related to this topic, see here.) SDN controllers The decoupled control and data plane architecture of software-defined networking ( SDN ), as depicted in the following figure, and in particular OpenFlow can be compared with an operating system and computer hardware. The OpenFlow controller (similar to the operating system) provides a programmatic interface to the OpenFlow switches (similar to the computer hardware). Using this programmatic interface, network applications, referred to as Net Apps, can be written to perform control and management tasks and offer new functionalities. The control plane in SDN and OpenFlow in particular is logically centralized and Net Apps are written as if the network is a single system. With a reactive control model, the OpenFlow switches must consult an OpenFlow controller each time a decision must be made, such as when a new packet flow reaches an OpenFlow switch (that is, Packet_in event). In the case of flow-based control granularity, there will be a small performance delay as the first packet of each new flow is forwarded to the controller for decision (for example, forward or drop), after which future traffic within that flow will be forwarded at line rate within the switching hardware. While the first-packet delay is negligible in many cases, it may be a concern if the central OpenFlow controller is geographically remote or if most flows are short-lived (for example, as single-packet flows). An alternative proactive approach is also possible in OpenFlow to push policy rules out from the controller to the switches. While this simplifies the control, management, and policy enforcement tasks, the bindings must be closely maintained between the controller and OpenFlow switches. The first important concern of this centralized control is the scalability of the system and the second one is the placement of controllers. A recent study of the several OpenFlow controller implementations (NOX-MT, Maestro, and Beacon), conducted on a large emulated network with 100,000 hosts and up to 256 switches, revealed that all OpenFlow controllers were able to handle at least 50,000 new flow requests per second in each of the experimental scenarios. Furthermore, new OpenFlow controllers under development, such as Mc-Nettle (http://haskell.cs.yale.edu/nettle/mcnettle/) target powerful multicore servers and are being designed to scale up to large data center workloads (for example, 20 million flow requests per second and up to 5,000 switches). In packet switching networks, traditionally, each packet contains the required information for a network switch to make individual routing decisions. However, most applications send data as a flow of many individual packets. The control granularity in OpenFlow is in the scale of flows, not packets. When controlling individual flows, the decision made for the first packet of the flow can be applied to all the subsequent packets of the flow within the data plane (OpenFlow switches). The overhead may be further reduced by grouping the flows together, such as all traffic between two hosts, and performing control decisions on the aggregated flows. The role of controller in SDN approach Multiple controllers may be used to reduce the latency or increase the scalability and fault tolerance of the OpenFlow (SDN) deployment. OpenFlow allows the connection of multiple controllers to a switch, which would allow backup controllers to take over in the event of a failure. Onix and HyperFlow take the idea further by attempting to maintain a logically centralized, but physically distributed control plane. This decreases the lookup overhead by enabling communication with local controllers, while still allowing applications to be written with a simplified central view of the network. The potential main downside of this approach is maintaining the consistent state in the overall distributed system. This may cause Net Apps, that believe they have an accurate view of the network, to act incorrectly due to inconsistency in the global network state. Recalling the operating system analogy, an OpenFlow controller acts as a network operating system and should implement at least two interfaces: a southbound interface that allows OpenFlow switches to communicate with the controller, and a northbound interface that presents a programmable application programming interface (API) to network control and management applications (that is, Net Apps). The existing southbound interface is OpenFlow protocol as an early SDN southbound interface implementation. External control and management systems/software or network services may wish to extract information about the underlying network or enforce policies, or control an aspect of the network behavior. Besides, a primary OpenFlow controller may need to share policy information with a backup controller, or to communicate with other controllers across multiple control domains. While the southbound interface (for example, OpenFlow or ForCES, http://datatracker.ietf.org/wg/forces/charter/) is well defined and can be considered as a de facto standard, there is no widely accepted standard for northbound interactions, and they are more likely to be implemented on a use-case basis for particular applications.
Read more
  • 0
  • 0
  • 12105

article-image-starting-instance
Packt
29 Oct 2013
6 min read
Save for later

Starting an instance

Packt
29 Oct 2013
6 min read
(For more resources related to this topic, see here.) When you add a new instance to your project, it is automatically started. After a few moments, it will be ready for logging in using SSH. In order to create and start an instance, run the following command: % gcutil –-project=<project_id> addinstance <instance_name> gcutil will interactively collect the necessary details on the command line. For our example, we create a new instance called hello-google-compute: % gcutil –project=packt-gce-starter addinstance hello-gce INFO: Waiting for insert of instance hello-gce. Sleeping for 3s. [omitted] The omitted output will show information on the zone, image, and machine type selected during interactive setup and provide a return code to indicate whether the operation was successful. If you wish to create an instance non-interactively, you will have to set a few additional parameters. The instance name must only contain lower-case letters, numbers, or dashes and it must start with a letter. Option Description --machine-type The machine type to host the instance. gcutillistmachinetypes displays a list of available machine types, and gcutil getmachinetype provides details on a specific machine type. --image The name of the image to install, from the project's images collection. gcutil listimages displays a list of available images, and gcutil getimage provides details on a specific image. A comprehensive list of options can be listed by running the following command: % gcutil help addinstance After creation, your instance will be also displayed in the Google Cloud Console. % gcutil --project=<project_id> listinstances By default, every instance has a network setup that allows the virtual machine to communicate with other machines in the same network and with the rest of the world via the Internet. Note that, however, communication is restricted by the default firewall to incoming SSH traffic; see The Firewall object section for details. Information on a newly created instance   Checking the status of an instance After having started an instance, or for routine checks during the management of your GCE infrastructure, you may want to check the status of a given virtual machine. This can be done either via the Web UI, as shown in the preceding screenshot, or by calling: % gcutil --project=<project_id> getinstance <instance_name> For our test instance, this will yield something like: % gcutil –-project=packt-gce-starter getinstance hello-gce +------------------------+-------------------------------------+ | property | value | +------------------------+-------------------------------------+ | name | hello-gce | | description | | | creation-time | 2013-05-27T11:02:54.825-07:00 | | machine | machineTypes/n1-standard-1 | | | | | status | RUNNING | | status-message | | | | | | disk | 0 | | type | EPHEMERAL | | mode | READ_WRITE | | | | | network-interface | | | network | networks/default | | ip | 10.240.17.7 | | access-configuration | External NAT | | type | ONE_TO_ONE_NAT | | external-ip | 192.158.30.140 | +------------------------+-------------------------------------+ Note the status line within the above output; it says RUNNING. This indicates that the machine is ready to be used. GCE instance states and transitions   Every instance in GCE has a defined status lifecycle, as shown in the preceding figure, and the following states are known: Status Description PROVISIONING Resources are being reserved for the instance, but the virtual machine is not running yet. STAGING Resources have been acquired for the instance, and the virtual machine is prepared for launch. RUNNING The instance is booting up or running. STOPPED The instance has been either shutdown, or it failed. Subsequently, it will either reboot (changing to PROVISIONING), or stop (changing to TERMINATED). TERMINATED The instance has been either shutdown or it failed, and rebooting the virtual machine is not an option. This status is permanent, and the instance must be deleted and recreated. Logging in to your instance As mentioned before, the default network setup allows you to connect to your instance via the SSH protocol. GCE automatically handles key management for you and your project members. To relieve you from the hassle of key handling, gcutil wraps SSH and takes care of sorting out password-less authentication correctly. To login to your instance via SSH, run the following command from your workstation computer: % gcutil --project=<project_id> ssh <instance_name> For our example, you would do the following: % gcutil --project=packt-gce-starter ssh hello-gce INFO: Zone for 'hello-gce' detected as u'europe-west1-a'. INFO: Running command line: ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -i /Users/packt/.ssh/google_compute_engine -A -p 22 packt@192.158.30.140 -- [omitted] packt@hello-gce:~$ As mentioned before, GCE takes care of SSH through the gcutil. In fact, gcutil does a lot! Most importantly, it checks whether you already have a public/private key pair and, if not, creates one for you. It also takes care of uploading your public key to the Google Cloud Console and associating it with your Google user account. In addition, it automatically injects your public key into every instance so that you can directly login, even if you did not set up a user account within the OS image. If you wish to use a different SSH client, you will have to manage usage and the upload of the correct key manually. gcutil stores the generated key pair in your home folder under the hidden directory ~/.ssh. There, you will (besides others) find two files: google_compute_engine (your private key) google_compute_engine.pub (your public key) Even if you do not wish to use gcutil for SSH'ing into your virtual machines, you should go through its setup routine at least once so that your keys are generated and uploaded to the Google Cloud Console; otherwise, you will not be able to login. One ring to rule them all During initial key ring generation, gcutil will ask you to enter and repeat a passphrase to protect your SSH key. Although you can leave the passphrase empty, we strongly recommend not doing so. If you do not protect your key with a passphrase, anyone who gets hold of your workstation computer can easily copy your key and will then have full administrative access to your whole GCE infrastructure! Summary Thus we learned how to get started with creating and running your infrastructure in the Cloud and few concepts that make up a well-performing GCE system. Resources for Article : Further resources on this subject: Blogger: Improving Your Blog with Google Analytics and Search Engine Optimization [Article] Google Earth, Google Maps and Your Photos: a Tutorial [Article] Search Engine Optimization using Sitemaps in Drupal 6 [Article]
Read more
  • 0
  • 0
  • 1497

article-image-use-iso-image-installation-windows8-virtual-machine
Packt
29 Oct 2013
5 min read
Save for later

Use Of ISO Image for Installation of Windows8 Virtual Machine

Packt
29 Oct 2013
5 min read
(For more resources related to this topic, see here.) In the past, the only way that a Windows consumer could acquire the Windows OS was to purchase the installation media on a CD-ROM, floppy disk, or physical computer accessory, which had to be ordered online or bought from a local bricks and mortar store. Now with the recent release of Windows 8, Microsoft is continuing to extend its installation platform to digital media on a large scale. Windows 8 simplifies the process by using a web platform installer called Windows 8 Upgrade Assistant, which makes it easier to download, burn physical copies, and create a backup copy of the installation media. This form of digital distribution allows Microsoft to deploy products at a faster speed to the market, and increase its capacity to meet consumer demands. Getting ready To proceed with this recipe you will need to download Windows 8, so go to http://www.windows.com/buy. How to do it... In the first part of this section we will look into downloading the Windows 8 ISO file. Skip these steps if you have already downloaded the Windows 8 ISO file in advance. Visit the Microsoft website to purchase Windows 8 and then select the option to download the Windows 8 Upgrade Assistant file. Launch the Windows 8 Upgrade Assistant file and proceed with purchasing Windows 8. After completing the transaction, wait while the Windows 8 setup files are downloaded. The estimated download time varies, based on your Internet connection.speed. In addition, you have the option to pause the download and resume it later. Once the download is complete, the Windows 8 Upgrade Assistant will verify the integrity of the download by checking for file corruption and missing files. Wait while the Windows 8 setup gets the files ready to begin the installation. You will see a prompt that says Install Windows 8. Select the Install by creating media radio button. Select ISO file, then click on the Save button. When prompted to select a location to save the ISO file, choose a folder location and type Windows 8 as the filename. Then click on the Save button. When the product key is revealed, write it down and store it somewhere secure. Then click on the Finish button. The following set of instructions explains the details of installing Windows 8 on a newly created virtual machine using VMware Player. These steps are similar to the installation procedures encountered when installing Windows 8 on a physical computer: Open the VMware application by going to Start | All Programs| VMware. Then click on the VMware Player menu item. If you are opening VMware Player for the first time, the VMware Player License Agreement prompt will be displayed. Read the terms and conditions. Select Yes, I accept the terms in the license agreement to proceed to open the VMware Player application. If you select No, I do not accept the terms in the license agreement, you will not be permitted to continue. The Welcome to the New Virtual Machine Wizard screen will be visible. Click on Create a New Virtual Machine on the right side of the window. Select the Installer disc image file (iso): radio button. Then click on the Browse button. Browse to the directory of the Windows 8 ISO image and click on Open. You will see an information icon that says Windows 8 detected. This operating system will use Easy Install. Click on the Next button to continue. Under Easy Install Information, type in the Windows product information and click on Next to continue. You now have the options to: Enter the Windows product key Select the version of Windows to install (Windows 8 or Windows 8 Pro) Enter the full name of the computer Enter a password, which is optional If you do not enter a product key, you will receive a message saying that it can be manually entered later. Enter a new virtual machine name and directory location to install the virtual machine. For example, type Windows 8 VM and then click on the Next button to continue. Enter 16 as the Maximum disk size (GB) and store the virtual disk as a single file. Then click on the Finish button. This is because Windows 8 requires a minimum of 16 GB of free hard drive space. The Windows 8 virtual machine will power on automatically for the first time. At the Ready to Create Virtual Machine prompt, click on the Finish button. Remember that Windows 8 requires a minimum of 16 GB hard disk free space for the 32 bit installation and 20 GB space for the 64 bit installation. VMware will prompt you to install VMware Tools for Windows 2000 and later, click on the Remind Me Later button. The virtual machine will automatically boot up to the Windows 8 setup wizard. Wait until the Windows installation is complete. The virtual machine will reboot several times during this process. You will see various Windows 8 pictorials during the installation; please be patient. Once the installation is complete, your virtual machine will be immediately directed to the Windows 8 home screen. Summary This article introduced you to downloading of the Windows 8 operating system as an ISO, creating a new virtual machine, and installing Window 8 as a virtual machine. Resources for Article: Further resources on this subject: VMware View 5 Desktop Virtualization [Article] Windows 8 with VMware View [Article] Cloning and Snapshots in VMware Workstation [Article]    
Read more
  • 0
  • 0
  • 5098

article-image-introducing-beagleboard
Packt
29 Oct 2013
9 min read
Save for later

Introducing BeagleBoard

Packt
29 Oct 2013
9 min read
(For more resources related to this topic, see here.) We'll first have a quick overview of the features of BeagleBoard (with focus on the latest xM version) —an open source hardware platform, borne for audio, video, and digital signal processing. Then we will introduce the concept of rapid prototyping and explain what we can do with the BeagleBoard support tools from MATLAB® and Simulink® by MathWorks®. Finally, this article ends with a summary. Different from most approaches that involve coding and compiling at a Linux PC and require intensive manual configuration in command-line manner, the rapid prototyping approach presented in this article is a Windows-based approach that features a Windows PC for embedded software development through user-friendly graphic interaction and relieves the developer from intensive coding so that you can concentrate on your application and algorithms and have the BeagleBoard run your inspiration. First of all, let's begin with a quick overview of this article. A quick overview of the BeagleBoard's functionality We can create a number of exciting projects to demonstrate how to build a prototype of an embedded audio, video, and digital signal processing system rapidly without intensive programming and coding. The main projects include: Installing Linux for BeagleBoard from a Windows PC Developing C/C++ with Eclipse on a Windows PC Automatic embedded code generation for BeagleBoard Serial communication and digital I/O application: Infrared motion detection Audio application: voice recognition Video application: motion detection These projects provide the workflow of building an embedded system. With the help of various online documents you can learn about setting up the development environment, writing software at a host PC running Microsoft Windows, and compiling the code for standalone ARM-executables at the BeagleBoard running Linux. Then you can learn the skills of rapid prototyping embedded audio and video systems via the BeagleBoard support tools from Simulink by MathWorks. The main features of these techniques include: Open source hardware A Windows-based friendly development environment Rapid prototyping and easy learning without intensive coding These features will save you from intensive coding and will also relieve the pressure on you to build an embedded audio/video processing system without learning the complicated embedded Linux. The rapid prototyping techniques presented allow you to concentrate on your brilliant concept and algorithm design, rather than being distracted by the complicated embedded system and low-level manual programming. This is beneficial for students and academics who are primarily interested in the development of audio/video processing algorithms, and want to build an embedded prototype for proof-of-concept quickly. BeagleBoard-xM BeagleBoard, the brainchild of a small group of Texas Instruments (TI) engineers and volunteers, is a pocket-sized, low-cost, fan-less, single-board computer containing TI Open Multimedia Application Platform 3 (OMAP3) System on a chip (SoC) processor, which integrates a 1 GHz ARM core and a TI's Digital Signal Processor (DSP) together. Since many consumer electronics devices nowadays run some form of embedded Linux-based environment and usually are on an ARM-based platform, the BeagleBoard was proposed as an inexpensive development kit for hobbyists, academics, and professionals for high-performance, ARM-based embedded system learning and evaluation. As an open hardware embedded computer with open source software development in mind, the BeagleBoard was created for audio, video, and digital signal processing with the purpose of meeting the demands of those who want to get involved with embedded system development and build their own embedded devices or solutions. Furthermore, by utilizing standard interfaces, the BeagleBoard comes with all of the expandability of today's desktop machines. The developers can easily bring their own peripherals and turn the pocket-sized BeagleBoard into a single-board computer with many additional features. The following figure shows the PCB layout and major components of the latest xM version of the BeagleBoard. The BeagleBoard-xM (referred to as BeagleBoard in this article unless specified otherwise) is an 8.25 x 8.25cm (3.25" x 3.25") circuit board that includes the following components: CPU: TI's DM3730 processor, which houses a 1 GHz ARM Cortex-A8 superscalar core and a TI's C64x+ DSP core. The power of the 32-bit ARM and C64+ DSP, plus a large amount of onboard DDR RAM arm the BeagleBoard with the capacity to deal with computational intensive tasks, such as audio and video processing. Memory: 512 MB MDDR SDRAM working 166MHz. The processor and the 512 MB RAM comes in a .44 mm (Package on Package) POP package, where the memory is mounted on top of the processor. microSD card slot: being provided as a means for the main nonvolatile memory. The SD cards are where we install our operating system and will act as a hard disk. The BeagleBoard is shipped with a 4GB microSD card containing factory-validated software (actually, an Angstrom distribution of embedded Linux tailored for BeagleBoard). Of course, this storage can be easily expanded by using, for example, an USB portable hard drive. USB2.0 On-The-Go (OTG) mini port: This port can be used as a communication link to a host PC and the power source deriving power from the PC over the USB cable. 4-port USB-2.0 hub: These four USB Type A connectors with full LS/FS/HS support. Each port can provide power on/off control and up to 500 mA as long as the input DC to the BeagleBoard is at least 3 A. RS232 port: A single RS232 port via UART3 of DM3730 processor is provided by a DB9 connector on BeagleBoard-xM. A USB-to-serial cable can be plugged directly into the DB9 connector. By default, when the BeagleBoard boots, system information will be sent to the RS232 port and you can log in to the BeagleBoard through it. 10/100 M Ethernet: The Ethernet port features auto-MDIX, which works for both crossover cable and straight-through cable. Stereo audio output and input: BeagleBoard has a hardware accelerated audio encoding and decoding (CODEC) chip and provides stereo in and out ports via two 3.5 mm jacks to support external audio devices, such as headphones, powered speakers, and microphones (either stereo or mono). Video interfaces: It includes S-video and Digital Visual Interface (DVI)-D output, LCD port, a Camera port. Joint Test Action Group (JTAG) connector: reset button, a user button, and many developer-friendly expansion connectors. The user button can be used as an application button. To get going, we need to power the BeagleBoard by either the USB OTG mini port, which just provides current of up to 500 mA to run the board alone, or a 5V power source to run with external peripherals. The BeagleBoard boots from the microSD card once the power is on. Various alternative software images are available on the BeagleBoard website, so we can replace the factory default images and have the BeagleBoard run with many other popular embedded operating systems (like Andria and Windows CE). The off-the-shelf expansion via standard interfaces on the BeagleBoard allows developers to choose various components and operating systems they prefer to build their own embedded solutions or a desktop-like system as shown below: BeagleBoard for rapid prototyping A rapid prototyping approach allows you to quickly create a working implementation of your proof-of-concept and verify your audio or video applications on hardware early, which overcomes barriers in the design-implementation-validation loops and helps you find the right solution for your applications. Rapid prototyping not only reduces the development time from concept to product, but also allows you to identify defects and mistakes in system and algorithm design at an early stage. Prototyping your concept and evaluating its performance on a target hardware platform gives you confidence in your design, and promotes its success in applications. The powerful BeagleBoard equipped with many standard interfaces provides a good hardware platform for rapid embedded system prototyping. On the other hand, the rapid prototyping tool, the BeagleBoard Support from Simulink package, provided by MathWorks with graphic user interface (GUI) allows developers to easily implement their concept and algorithm graphically in Simulink, and then directly run the algorithms at the BeagleBoard. In short, you design algorithms in MATLAB/Simulink and see them perform as a standalone application on the BeagleBoard. In this way, you can concentrate on your brilliant concept and algorithm design, rather than being distracted by the complicated embedded system and low-level manual programming. The prototyping tool reduces the steep learning curve of embedded systems and helps hobbyists, students, and academics who have a great idea, but have little background knowledge of embedded systems. This feature is particularly useful to those who want to build a prototype of their applications in a short time. MathWorks introduced the BeagleBoard support package for rapid prototyping in 2010. Since the release of MATLAB 2012a, support for the BeagleBoard-xM has been integrated into Simulink and is also available in the student version of MATLAB and Simulink. Your rapid prototyping starts with modeling your systems and implementing algorithms in MATLAB and Simulink. From your models, you can automatically generate algorithmic C code along with processor-specific, real-time scheduling code and peripheral drivers, and run them as standalone executables on embedded processors in real time. The following steps provide an overview of the work flow for BeagleBoard rapid prototyping in MATLAB/Simulink: Create algorithms for various applications in Simulink and MATLAB with a user-friendly GUI. The applications can be audio processing (for example, digital amplifiers), computer vision applications (for example, object tracking), control systems (for example, flight control), and so on. Verify and improve the algorithm work by simulation. With intensive simulation, it is expected that most defects, errors, and mistakes in algorithms will be identified. Then the algorithms are easily modified and updated to fix the identified issues. Run the algorithms as standalone applications on the BeagleBoard. Interactive parameter turning, signal monitoring, and performance optimization of applications running on the BeagleBoard. Summary In this article, we have familiarized ourselves with the BeagleBoard and rapid prototyping by using MATLAB/Simulink. We have also looked at some of the features of rapid prototyping and the basic steps in rapid prototyping in MATLAB/Simulink. Resources for Article: Further resources on this subject: 2-Dimensional Image Filtering [Article] Creating Interactive Graphics and Animation [Article] Advanced Matplotlib: Part 1 [Article]
Read more
  • 0
  • 0
  • 13528
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-understanding-websockets-and-server-sent-events-detail
Packt
29 Oct 2013
10 min read
Save for later

Understanding WebSockets and Server-sent Events in Detail

Packt
29 Oct 2013
10 min read
(For more resources related to this topic, see here.) Encoders and decoders in Java API for WebSockets As seen in the previous chapter, the class-level annotation @ServerEndpoint indicates that a Java class is a WebSocket endpoint at runtime. The value attribute is used to specify a URI mapping for the endpoint. Additionally the user can add encoder and decoder attributes to encode application objects into WebSocket messages and WebSocket messages into application objects. The following table summarizes the @ServerEndpoint annotation and its attributes: Annotation Attribute Description @ServerEndpoint   This class-level annotation signifies that the Java class is a WebSockets server endpoint.   value The value is the URI with a leading '/.'   encoders The encoders contains a list of Java classes that act as encoders for the endpoint. The classes must implement the Encoder interface.   decoders The decoders contains a list of Java classes that act as decoders for the endpoint. The classes must implement the Decoder interface.   configurator The configurator attribute allows the developer to plug in their implementation of ServerEndpoint.Configurator that is used when configuring the server endpoint.   subprotocols The sub protocols attribute contains a list of sub protocols that the endpoint can support. In this section we shall look at providing encoder and decoder implementations for our WebSockets endpoint. The preceding diagram shows how encoders will take an application object and convert it to a WebSockets message. Decoders will take a WebSockets message and convert to an application object. Here is a simple example where a client sends a WebSockets message to a WebSockets java endpoint that is annotated with @ServerEndpoint and decorated with encoder and decoder class. The decoder will decode the WebSockets message and send back the same message to the client. The encoder will convert the message to a WebSockets message. This sample is also included in the code bundle for the book. Here is the code to define ServerEndpoint with value for encoders and decoders: @ServerEndpoint(value="/book", encoders={MyEncoder.class}, decoders = {MyDecoder.class} ) public class BookCollection { @OnMessage public void onMessage(Book book,Session session) { try { session.getBasicRemote().sendObject(book); } catch (Exception ex) { ex.printStackTrace(); } } @OnOpen public void onOpen(Session session) { System.out.println("Opening socket" +session.getBasicRemote() ); } @OnClose public void onClose(Session session) { System.out.println("Closing socket" + session.getBasicRemote()); } } In the preceding code snippet, you can see the class BookCollection is annotated with @ServerEndpoint. The value=/book attribute provides URI mapping for the endpoint. The @ServerEndpoint also takes the encoders and decoders to be used during the WebSocket transmission. Once a WebSocket connection has been established, a session is created and the method annotated with @OnOpen will be called. When the WebSocket endpoint receives a message, the method annotated with @OnMessage will be called. In our sample the method simply sends the book object using the Session.getBasicRemote() which will get a reference to the RemoteEndpoint and send the message synchronously. Encoders can be used to convert a custom user-defined object in a text message, TextStream, BinaryStream, or BinaryMessage format. An implementation of an encoder class for text messages is as follows: public class MyEncoder implements Encoder.Text<Book> { @Override public String encode(Book book) throws EncodeException { return book.getJson().toString(); } } As shown in the preceding code, the encoder class implements Encoder.Text<Book>. There is an encode method that is overridden and which converts a book and sends it as a JSON string. (More on JSON APIs is covered in detail in the next chapter) Decoders can be used to decode WebSockets messages in custom user-defined objects. They can decode in text, TextStream, and binary or BinaryStream format. Here is a code for a decoder class: public class MyDecoder implements Decoder.Text<Book> { @Override public Book decode(String string) throws DecodeException { javax.json.JsonObject jsonObject = javax.json.Json.createReader(new StringReader(string)).readObject(); return new Book(jsonObject); } @Override public boolean willDecode(String string) { try { javax.json.Json.createReader(new StringReader(string)).readObject(); return true; } catch (Exception ex) { } return false; } In the preceding code snippet, the Decoder.Text needs two methods to be overridden. The willDecode() method checks if it can handle this object and decode it. The decode() method decodes the string into an object of type Book by using the JSON-P API javax.json.Json.createReader(). The following code snippet shows the user-defined class Book: public class Book { public Book() {} JsonObject jsonObject; public Book(JsonObject json) { this.jsonObject = json; } public JsonObject getJson() { return jsonObject; } public void setJson(JsonObject json) { this.jsonObject = json; } public Book(String message) { jsonObject = Json.createReader(new StringReader(message)).readObject(); } public String toString () { StringWriter writer = new StringWriter(); Json.createWriter(writer).write(jsonObject); return writer.toString(); } } The Book class is a user-defined class that takes the JSON object sent by the client. Here is an example of how the JSON details are sent to the WebSockets endpoints from JavaScript. var json = JSON.stringify({ "name": "Java 7 JAX-WS Web Services", "author":"Deepak Vohra", "isbn": "123456789" }); function addBook() { websocket.send(json); } The client sends the message using websocket.send() which will cause the onMessage() of the BookCollection.java to be invoked. The BookCollection.java will return the same book to the client. In the process, the decoder will decode the WebSockets message when it is received. To send back the same Book object, first the encoder will encode the Book object to a WebSockets message and send it to the client. The Java WebSocket Client API WebSockets and Server-sent Events , covered the Java WebSockets client API. Any POJO can be transformed into a WebSockets client by annotating it with @ClientEndpoint. Additionally the user can add encoders and decoders attributes to the @ClientEndpoint annotation to encode application objects into WebSockets messages and WebSockets messages into application objects. The following table shows the @ClientEndpoint annotation and its attributes: Annotation Attribute Description @ClientEndpoint   This class-level annotation signifies that the Java class is a WebSockets client that will connect to a WebSockets server endpoint.   value The value is the URI with a leading /.   encoders The encoders contain a list of Java classes that act as encoders for the endpoint. The classes must implement the encoder interface.   decoders The decoders contain a list of Java classes that act as decoders for the endpoint. The classes must implement the decoder interface.   configurator The configurator attribute allows the developer to plug in their implementation of ClientEndpoint.Configurator, which is used when configuring the client endpoint.   subprotocols The sub protocols attribute contains a list of sub protocols that the endpoint can support. Sending different kinds of message data: blob/binary Using JavaScript we can traditionally send JSON or XML as strings. However, HTML5 allows applications to work with binary data to improve performance. WebSockets supports two kinds of binary data Binary Large Objects (blob) arraybuffer A WebSocket can work with only one of the formats at any given time. Using the binaryType property of a WebSocket, you can switch between using blob or arraybuffer: websocket.binaryType = "blob"; // receive some blob data websocket.binaryType = "arraybuffer"; // now receive ArrayBuffer data The following code snippet shows how to display images sent by a server using WebSockets. Here is a code snippet for how to send binary data with WebSockets: websocket.binaryType = 'arraybuffer'; The preceding code snippet sets the binaryType property of websocket to arraybuffer. websocket.onmessage = function(msg) { var arrayBuffer = msg.data; var bytes = new Uint8Array(arrayBuffer); var image = document.getElementById('image'); image.src = 'data:image/png;base64,'+encode(bytes); } When the onmessage is called the arrayBuffer is initialized to the message.data. The Uint8Array type represents an array of 8-bit unsigned integers. The image.src value is in line using the data URI scheme. Security and WebSockets WebSockets are secured using the web container security model. A WebSockets developer can declare whether the access to the WebSocket server endpoint needs to be authenticated, who can access it, or if it needs an encrypted connection. A WebSockets endpoint which is mapped to a ws:// URI is protected under the deployment descriptor with http:// URI with the same hostname,port path since the initial handshake is from the HTTP connection. So, WebSockets developers can assign an authentication scheme, user roles, and a transport guarantee to any WebSockets endpoints. We will take the same sample as we saw in , WebSockets and Server-sent Events , and make it a secure WebSockets application. Here is the web.xml for a secure WebSocket endpoint: <web-app version="3.0" xsi_schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"> <security-constraint> <web-resource-collection> <web-resource-name>BookCollection</web-resource-name> <url-pattern>/index.jsp</url-pattern> <http-method>PUT</http-method> <http-method>POST</http-method> <http-method>DELETE</http-method> <http-method>GET</http-method> </web-resource-collection> <user-data-constraint> <description>SSL</description> <transport-guarantee>CONFIDENTIAL</transport-guarantee> </user-data-constraint> </security-constraint> </web-app> As you can see in the preceding snippet, we used <transport-guarantee>CONFIDENTIAL</transport-guarantee>. The Java EE specification followed by application servers provides different levels of transport guarantee on the communication between clients and application server. The three levels are: Data Confidentiality (CONFIDENTIAL) : We use this level to guarantee that all communication between client and server goes through the SSL layer and connections won't be accepted over a non-secure channel. Data Integrity (INTEGRAL) : We can use this level when a full encryption is not required but we want our data to be transmitted to and from a client in such a way that, if anyone changed the data, we could detect the change. Any type of connection (NONE) : We can use this level to force the container to accept connections on HTTP and HTTPs. The following steps should be followed for setting up SSL and running our sample to show a secure WebSockets application deployed in Glassfish. Generate the server certificate: keytool -genkey -alias server-alias -keyalg RSA -keypass changeit --storepass changeit -keystore keystore.jks Export the generated server certificate in keystore.jks into the file server.cer: keytool -export -alias server-alias -storepass changeit -file server.cer -keystore keystore.jks Create the trust-store file cacerts.jks and add the server certificate to the trust store: keytool -import -v -trustcacerts -alias server-alias -file server.cer -keystore cacerts.jks -keypass changeit -storepass changeit Change the following JVM options so that they point to the location and name of the new keystore. Add this in domain.xml under java-config: <jvm-options>-Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks</jvm-options> <jvm-options>-Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks</jvm-options> Restart GlassFish. If you go to https://localhost:8181/helloworld-ws/, you can see the secure WebSocket application. Here is how the the headers look under Chrome Developer Tools: Open Chrome Browser and click on View and then on Developer Tools . Click on Network . Select book under element name and click on Frames . As you can see in the preceding screenshot, since the application is secured using SSL the WebSockets URI, it also contains wss://, which means WebSockets over SSL. So far we have seen the encoders and decoders for WebSockets messages. We also covered how to send binary data using WebSockets. Additionally we have demonstrated a sample on how to secure WebSockets based application. We shall now cover the best practices for WebSocket based-applications.
Read more
  • 0
  • 0
  • 18374

article-image-linux-desktop-environments
Packt
29 Oct 2013
7 min read
Save for later

Linux Desktop Environments

Packt
29 Oct 2013
7 min read
(For more resources related to this topic, see here.) A computer desktop is normally composed of windows, icons, directories/folders, a toolbar, and some artwork. A window manager handles what the user sees and the tasks that are performed. A desktop is also sometimes referred to as a graphical user interface (GUI). There are many different desktops available for Linux systems. Here is an overview of some of the more common ones. GNOME 2 GNOME 2 is a desktop environment and GUI that is developed mainly by Red Hat, Inc. It provides a very powerful and conventional desktop interface. There is a launcher menu for quicker access to applications, and also taskbars (called panels). Note that in most cases these can be located on the screen where the user desires. The screenshot of GNOME 2 running on Fedora 14 is as follows: This shows the desktop, a command window, and the Computer folder. The top and bottom "rows" are the panels. From the top, starting on the left, are the Applications, Places, and System menus. I then have a screensaver, the Firefox browser, a terminal, Evolution, and a Notepad. In the middle is the lock-screen app, and on the far right is a notification about updates, the volume control, Wi-Fi strength, battery level, the date/time, and the current user. Note that I have customized several of these, for example, the clock. Getting ready If you have a computer running the GNOME 2 desktop, you may follow along in this section. A good way to do this is by running a Live Image, available from many different Linux distributions. The screenshot showing the Add to Panel window is as follows: How to do it... Let's work with this desktop a bit: Bring this dialog up by right-clicking on an empty location on the task bar. Let's add something cool. Scroll down until you see Weather Report, click on it and then click on the Add button at the bottom. On the panel you should now see something like 0 °F. Right-click on it. This will bring up a dialog, select Preferences. You are now on the General tab. Feel free to change anything here you want, then select the Location tab, and put in your information. When done, close the dialog. On my system the correct information was displayed instantly. Now let's add something else that is even more cool. Open the Add to Panel dialog again and this time add Workspace Switcher. The default number of workspaces is two, I would suggest adding two more. When done, close the dialog. You will now see four little boxes on the bottom right of your screen. Clicking on one takes you to that workspace. This is a very handy feature of GNOME 2. There's more... I find GNOME 2 very intuitive and easy to use. It is powerful and can be customized extensively. It does have a few drawbacks, however. It tends to be somewhat "heavy" and may not perform well on less powerful machines. It also does not always report errors properly. For example, using Firefox open a local file that does not exist on your system (that is, file:///tmp/LinuxBook.doc). A File Not Found dialog should appear. Now try opening another local file that does exist, but which you do not have permissions for. It does not report an error, and in fact doesn't seem to do anything. Remember this if it happens to you. KDE desktop The KDE desktop was designed for desktop PCs and powerful laptops. It allows for extensive customization and is available on many different platforms. The following is a description of some of its features. Getting ready If you have a Linux machine running the KDE desktop you can follow along. These screenshots are from KDE running on a Live Media image of Fedora 18. The desktop icon on the far right allows the user to access Tool Box: You can add panels, widgets, activities, shortcuts, lock the screen, and add a lot more using this dialog. The default panel on the bottom begins with a Fedora icon. This icon is called a Kickoff Application Launcher and allows the user to access certain items quickly. These include Favorites, Applications, a Computer folder, a Recently Used folder, and a Leave button. If you click on the next icon it will bring up the Activity Manager. Here you can create the activities and monitor them. The next icon allows you to select which desktop is currently in the foreground, and the next items are the windows that are currently open. Over to the far right is the Clipboard. Here is a screenshot of the clipboard menu: Next is the volume control, device notifier, and networking status. Here is a screenshot of Interfaces and Connections dialog: Lastly, there is a button to show the hidden icons and the time. How to do it... Let's add a few things to this desktop: We should add a console. Right-click on an empty space on the desktop. A dialog will come up with several options; select Konsole. You should now have a terminal. Close that dialog by clicking on some empty space. Now let's add some more desktops. Right-click on the third icon on the bottom left of the screen. A dialog will appear, click on Add Virtual Desktop. I personally like four of these. Now let's add something to the panel. Right-click on some empty space on the panel and hover the mouse over Panel Options; click on AddWidgets. You will be presented with a few widgets. Note that the list can be scrolled to see a whole lot more. For example, scroll over to Web Browser and double-click on it. The web browser icon will appear on the panel near the time. There's more... You can obviously do quite a bit of customization using the KDE desktop. I would suggest trying out all of the various options, to see which ones you like the best. KDE is actually a large community of open source developers, of which KDE Plasma desktop is a part. This desktop is probably the heaviest of the ones reviewed, but also one of the most powerful. I believe this is a good choice for people who need a very elaborate desktop environment. xfce xfce is another desktop environment for Linux and UNIX systems. It tends to run very crisply and not use as many system resources. It is very intuitive and user-friendly. Getting ready The following is a screenshot of xfce running on the Linux machine I am using to write this article: If you have a machine running the xfce desktop, you can perform these actions. I recommend a Live Media image from the Fedora web page. While somewhat similar to GNOME 2, the layout is somewhat different. Starting with the panel on the top (panel 1) is the Applications Menu, following by a LogOut dialog. The currently open windows are next. Clicking on one of these will either bring it up or minimize it depending on its current state. The next item is the Workspaces of which I have four, then the Internet status. To complete the list is the volume and mixer apps and the date and time. The screen contents are mostly self-explanatory; I have three terminal windows open and the File Manager folder. The smaller panel on the bottom of the screen is called panel 2. How to do it... Let's work with the panels a bit: In order to change panel 2 we must unlock it first. Right-click on the top panel, and go to Panel | PanelPreferences. Use the arrows to change to panel 2. See the screenshot below: You can see it is locked. Click on Lock panel to unlock it and then close this dialog. Now go to panel 2 (on the bottom) and right-click on one of the sides. Click on AddNewItems.... Add the applications you desire. There's more... This is by no means an exhaustive list of what xfce can do. The features are modular and can be added as needed. See http://www.xfce.org for more information.
Read more
  • 0
  • 0
  • 6994

article-image-multiserver-installation
Packt
29 Oct 2013
7 min read
Save for later

Multiserver Installation

Packt
29 Oct 2013
7 min read
(For more resources related to this topic, see here.) The prerequisites for Zimbra Let us dive into the prerequisites for Zimbra: Zimbra supports only 64-bit LTS versions of Ubuntu, release 10.04 and above. If you would like to use a 32-bit version, you should use Ubuntu 8.04.x LTS with Zimbra 7.2.3. Having a clean and freshly installed system is preferred for Zimbra; it requires a dedicated system and there is no need to install components such as Apache and MySQL since the Zimbra server contains all the components it needs. Note that installing Zimbra with another service (such as a web server) on the same server can cause operational issues. The dependencies (libperl5.14, libgmp3c2, build-essential, sqlite3, sysstat, and ntp) should be installed beforehand. Configure a fixed IP address on the server. Have a domain name and a well-configured DNS (A and MX entries) that points to the server. The system clocks should be synced on all servers. Configure the file /etc/resolv.conf on all servers to point at the server on which we installed the bind (it can be installed on any Zimbra server or on a separate server). We will explain this point in detail later. Preparing the environment Before starting the Zimbra installation process, we should prepare the environment. In the first part of this section, we will see the different possible configurations and then, in the second part, we will present the needed assumptions to apply the chosen configuration. Multiserver configuration examples One of the greatest advantages of Zimbra is its scalability; we can deploy it for a small business with few mail accounts as well as for a huge organization with thousands of mail accounts. There are many possible configuration options; the following are the most used out of those: Small configuration: All Zimbra components are installed on only one server. Medium configuration: Here, LDAP and message store are installed on one server and Zimbra MTA on a separate server. Note here that we can use more Zimbra MTA servers so we can scale easier for large incoming or outgoing e-mail volume. Large configuration: In this case, LDAP will be installed on a dedicated server and we will have multiple mailbox and MTA servers, so we can scale easier for a large number of users. Very large configuration: The difference between this configuration and large one is the existence of an additional LDAP server, so we will have a Master LDAP and its replica. We choose the medium configuration; so, we will install LDAP and mailbox in one server and MTA on the other server. Install different servers in the following order (for medium configuration, 1 and 2 are combined in only one step): 1. First of all, install and configure the LDAP server. 2. Then, install and configure Zimbra mailbox servers. 3. Finally, install Zimbra MTA servers and finish the whole installation configuration. New installations of Zimbra limit spam/ham training to the first installed MTA. If you uninstall or move this MTA, you should enable spam/ham training on another MTA as one host should have this enabled to run zmtrainsa --cleanup. To do this, execute the following command: zmlocalconfig -e zmtrainsa_cleanup_host=TRUE Assumptions In this article, we will use some specific information as input in the Zimbra installation process, which, in most cases, will be different for each user. Therefore, we will note some of the most redundant ones in this section. Remember that you should specify your own values rather than using the arbitrary values that I have provided. The following is the list of assumptions used : OS version: ubuntu-12.04.2-server-amd64 Zimbra version: zcs-8.0.3_GA_5664.UBUNTU12_64.20130305090204 MTA server name: mta MTA hostname: mta.zimbra-essentials.com Internet domain: zimbra-essentials.com MTA server IP address: 172.16.126.141 MTA server IP subnet mask: 255.255.255.0 MTA server IP gateway: 172.16.126.1 Internal DNS server: 172.16.126.11 External DNS server: 8.8.8.8 MTA admin ID: abdelmonam MTA admin Password: Z!mbra@dm1n Zimbra admin Password: zimbrabook MTA server name: ldap MTA hostname: ldap.zimbra-essentials.com LDAP server IP address: 172.16.126.140 LDAP server IP subnet mask: 255.255.255.0 LDAP server IP gateway: 172.16.126.1 Internal DNS server: 172.16.126.11 External DNS server: 8.8.8.8 LDAP admin ID: abdelmonam LDAP admin password: Z!mbra@dm1n To be able to follow the steps described in the next sections, especially each time we need to perform a configuration, the reader should know how to harness the vi editor. If not, you should develop your skill set for using the vi editor or use another editor instead. You can find good basic training for the vi editor at http://www.cs.colostate.edu/helpdocs/vi.html System requirements For the various system requirements, please refer to the following link: http://www.zimbra.com/docs/os/8.0.0/multi_server_install/wwhelp/wwhimpl/common/html/wwhelp.htm#href=ZCS_Multiserver_Open_8.0.System_Requirements_for_VMware_Zimbra_Collaboration_Server_8.0.html&single=true If you are using another version of Zimbra, please check the correct requirements on the Zimbra website. Ubuntu server installation First of all, choose the appropriate language. Choose Install Ubuntu Server and then press Enter. When the installation prompts you to provide a hostname, configure only a one-word hostname; in the Assumptions section, we've chosen ldap for the LDAP and mailstore server and mta for the MTA server—don't give the fully qualified domain name (for example, mta.zimbra-essentials.com). On the next screen that calls for the domain name, assign it zimbra-essentials.com (without the hostname). The hard disk setup is simple if you are using a single drive; however, in the case of a server, it's not the best way to do things. There are a lot of options for partitioning your drives. In our case, we just make a little partition (2x RAM) for swapping, and what remains will be used for the whole system. Others can recommend separate partitions for mailstore, system, and so on. Feel free to use the recommendation you want depending on your IT architecture; use your own judgment here or ask your IT manager. After finishing the partitioning task, you will be asked to enter the username and password; you can choose what you want except admin and zimbra. When asked if you want to encrypt the home directory, select No and then press Enter. Press Enter to accept an empty entry for the HTTP proxy. Choose Install security updates automatically and then press Enter. On the Software Selection screen, you must select the DNS Server and the OpenSSH Server choices for installation; no other options. This will authorize remote administration (SSH) and mandatorily set up bind9 for a split DNS. For bind9, you can install it on only one server, which is what we've done in this article. Select Yes and then press Enter to install the GRUB boot loader to the master boot record. The installation should have completed successfully. Preparing Ubuntu for Zimbra installation In order to prepare the Ubuntu for the Zimbra installation, the following steps need to be performed: Log in to the newly installed system and update and upgrade Ubuntu using the following commands: sudo apt-get update sudo apt-get upgrade Install the dependencies as follows: sudo apt-get install libperl5.14 libgmp3c2 build-essential sqlite3 sysstat ntp Zimbra recommends (but there's no obligation) to disable and remove Apparmor. sudo /etc/init.d/apparmor stop sudo /etc/init.d/apparmor teardown sudo update-rc.d -f apparmor remove sudo aptitude remove apparmor apparmor-utils Set the static IP for your server as follows: Open the network interfaces file using the following command: sudo vi /etc/network/interfaces Then replace the following line: iface eth0 inet dhcp With: iface eth0 inet static address 172.16.126.14 netmask 255.255.255.0 gateway 172.16.126.1 network 172.16.126.0 broadcast 172.16.126.255 Restart the network process by typing in the following: sudo /etc/init.d/networking restart Sanity test! To verify that your network configuration is configured properly, type in ifconfig and ensure that the settings are correct. Then try to ping any working website (such as google.com) to see if that works. On each server, pay attention when you set the static IP address (172.16.126.140 for the LDAP server and 172.16.126.141 for the MTA server). Summary In this article, we learned the prerequisites for Zimbra multiserver installation and preparing the environment for the installation of the Zimbra server in a multiserver environment. Resources for Article : Further resources on this subject: Routing Rules in AsteriskNOW - The Calling Rules Tables [Article] Users, Profiles, and Connections in Elgg [Article] Integrating Zimbra Collaboration Suite with Microsoft Outlook [Article]
Read more
  • 0
  • 0
  • 4467

article-image-cloudera-hadoop-and-hp-vertica
Packt
29 Oct 2013
9 min read
Save for later

Cloudera Hadoop and HP Vertica

Packt
29 Oct 2013
9 min read
(For more resources related to this topic, see here.) Cloudera Hadoop Hadoop is one of the names we think about when it comes to Big Data. I'm not going into details about it since there is plenty of information out there; moreover, like somebody once said, "If you decided to use Hadoop for your data warehouse, then you probably have a good reason for it". Let's not forget: it is primarily a distributed filesystem, not a relational database. That said, there are many cases when we may need to use this technology for number crunching, for example, together with MicroStrategy for analysis and reporting. There are mainly two ways to leverage Hadoop data from MicroStrategy: the first is Hive and the second is Impala. They both work as SQL bridges to the underlying Hadoop structures, converting standard SELECT statements into jobs. The connection is handled by a proprietary 32-bit ODBC driver available for free from the Cloudera website. In my tests, Impala resulted largely faster than Hive, so I will show you how to use it from our MicroStrategy virtual machine. Please note that I am using Version 9.3.0 for consistency with the rest of the book. If you're serious about Big Data and Hadoop, I strongly recommend upgrading to 9.3.1 for enhanced performance and easier setup. See MicroStrategy knowledge base document TN43588 : Post-Certification of Cloudera Impala 1.0 with MicroStrategy 9.3.1 . The ODBC driver is the same for both Hive and Impala, only the driver settings change. Connecting to a Hadoop database To show how we can connect to a Hadoop database, I will use two virtual machines: one with MicroStrategy Suite and the second with Cloudera Hadoop distribution, specifically, a virtual appliance that is available for download from their website. The configuration of the Hadoop cluster is out of scope; moreover, I am not a Hadoop expert. I'll simply give some hints, feel free to use any other configuration/vendor, the procedure and ODBC parameters should be similar. Getting ready Start by going to http://at5.us/AppAU1 The Cloudera VM download is almost 3 GB (cloudera-quickstart-vm-4.3.0-vmware.tar.gz) and features the CH4 version. After unpacking the archive, you'll find a cloudera-quickstart-vm-4.3.0-vmware.ovf file that can be opened with VMware, see screen capture: Accept the defaults and click on Import to generate the cloudera-quickstart-vm-4.3.0-vmware virtual machine. Before starting the Cloudera appliance, change the network card settings from NAT to Bridged since we need to access the database from another VM: Leave the rest of the parameters, as per the default, and start the machine. After a while, you'll be presented with a graphical interface of Centos Linux. If the network has started correctly, the machine should have received an IP address from your network DHCP. We need a fixed rather than dynamic address in the Hadoop VM, so: Open the System | Preferences | Network Connections menu. Select the name of your card (should be something like Auto eth1 ) and click on Edit… . Move to the IPv4 Settings tab and change the Method from Automatic (DHCP) to Manual . Click on the Add button to create a new address. Ask your network administrator for details here and fill Address , Netmask , and Gateway . Click on Apply… and when prompted type the root password cloudera and click on Authenticate . Then click on Close . Check if the change was successful by opening a Terminal window (Applications | System Tools | Terminal ) and issue the ifconfig command, the answer should include the address that you typed in step 4. From the MicroStrategy Suite virtual machine, test if you can ping the Cloudera VM. When we first start Hadoop, there are no tables in the database, so we create the samples: In the Cloudera virtual machine, from the main page in Firefox open Cloudera Manager , click on I Agree in the Information Assurance Policy dialog. Log in with username admin and password admin. Look for a line with a service named oozie1 , notice that it is stopped. Click on the Actions button and select Start… . Confirm with the Start button in the dialog. A Status window will pop up, wait until the Progress is reported as Finished and close it. Now click on the Hue button in the bookmarks toolbar. Sign up with username admin and password admin, you are now in the Hue home page. Click on the first button in the blue Hue toolbar (tool tip: About Hue ) to go to the quick Start Wizard . Click on the Next button to go to Step 2: Examples tab. Click on Beeswax (Hive UI) and wait until a message over the toolbar says Examples refreshed . Now in the Hue toolbar, click on the seventh button from the left (tool tip: Metastore Manager ), you will see the default database with two tables: sample_07 and sample_08 . Enable the checkbox of sample_08 and click on the Browse Data button. After a while the Results tab shows a grid with data. So far so good. We now go back to the Cloudera Manager to start the Impala service. Click on the Cloudera Manager bookmark button. In the Impala1 row, open the Actions menu and choose Start… , then confirm Start . Wait until the Progress says Finished , then click on Close in the command details window. Go back to Hue and click on the fourth button on the toolbar (tool tip: Cloudera Impala (TM) Query UI ). In the Query Editor text area, type select * from sample_08 and click on Execute to see the table content. Next, we open the MicroStrategy virtual machine and download the 32-bit Cloudera ODBC Driver for Apache Hive, Version 2.0 from http://at5.us/AppAU2. Download the ClouderaHiveODBCSetup_v2_00.exe file and save it in C:install. How to do it... We install the ODBC driver: Run C:installClouderaHiveODBCSetup_v2_00.exe and click on the Next button until you reach Finish at the end of the setup, accepting every default. Go to Start | All Programs | Administrative Tools | Data Sources (ODBC) to open the 32-bit ODBC Data Source Administrator (if you're on 64-bit Windows, it's in the SysWOW64 folder). Click on System DSN and hit the Add… button. Select Cloudera ODBC Driver for Apache Hive and click on Finish . Fill the Hive ODBC DSN Configuration with these case-sensitive parameters (change the Host IP according to the address used in step 4 of the Getting ready section): Data Source Name : Cloudera VM Host : 192.168.1.40 Port : 21050 Database : default Type : HS2NoSasl Click on OK and then on OK again to close the ODBC Data Source Administrator. Now open the MicroStrategy Desktop application and log in with administrator and the corresponding password. Right-click on MicroStrategy Analytics Modules and select Create New Project… . Click on the Create project button and name it HADOOP, uncheck Enable Change Journal for this project and click on OK . When the wizard finishes creating the project click on Select tables from the Warehouse Catalog and hit the button labeled New… . Click on Next and type Cloudera VM in the Name textbox of the Database Instance Definition window. In this same window, open the Database type combobox and scroll down until you find Generic DBMS . Click on Next . In Local system ODBC data sources , pick Cloudera VM and type admin in both Database login and Password textboxes. Click on Next , then on Finish , and then on OK . When a Warehouse Catalog Browser error appears, click on Yes . In the Warehouse Catalog Options window, click on Edit… on the right below Cloudera VM . Select the Advanced tab and enable the radio button labeled Use 2.0 ODBC calls in the ODBC Version group. Click on OK . Now select the category Catalog | Read Settings in the left tree and enable the first radio button labeled Use standard ODBC calls to obtain the database catalog . Click on OK to close this window. When the Warehouse Catalog window appears, click on the lightning button (tool tip: Read the Warehouse Catalog ) to refresh the list of available tables. Pick sample_08 and move it to the right of the shopping cart. Then right-click on it and choose Import Prefix . Click on Save and Close and then on OK twice to close the Project Creation Assistant . You can now open the project and update the schema. From here, the procedure to create objects is the same as in any other project: Go to the Schema Objects | Attributes folder, and create a new Job attribute with these columns: ID : Table: sample_08 Column: code DESC : Table: sample_08 Column: description Go to the Fact folder and create a new Salary fact with salary column. Update the schema. Go to the Public Objects | Metrics folder and create a new Salary metric based on the Salary fact with Sum as aggregation function. Go to My Personal Objects | My Reports and create a new report with the Job attribute and the Salary metric: There you go; you just created your first Hadoop report. How it works... Executing Hadoop reports is no different from running any other standard DBMS reports. The ODBC driver handles the communication with Cloudera machine and Impala manages the creation of jobs to retrieve data. From MicroStrategy perspective, it is just another SELECT query that returns a dataset. There's more... Impala and Hive do not support the whole set of ANSI SQL syntax, so in some cases you may receive an error if a specific feature is not implemented: See the Cloudera documentation for details. HP Vertica Vertica Analytic Database is grid-based, column-oriented, and designed to manage large, fast-growing volumes of data while providing rapid query performance. It features a storage organization that favors SELECT statements over UPDATE and DELETE plus a high compression that stores columns of homogeneous datatype together. The Community (free) Edition allows up to three hosts and 1 TB of data, which is fairly sufficient for small to medium BI projects with MicroStrategy. There are several clients available for different operating systems, including 32-bit and 64-bit ODBC drivers for Windows.
Read more
  • 0
  • 0
  • 2986
article-image-creating-and-using-composer-packages
Packt
29 Oct 2013
7 min read
Save for later

Creating and Using Composer Packages

Packt
29 Oct 2013
7 min read
(For more resources related to this topic, see here.) Using Bundles One of the great features in Laravel is the ease in which we can include the class libraries that others have made using bundles. On the Laravel site, there are already many useful bundles, some of which automate certain tasks while others easily integrate with third-party APIs. A recent addition to the PHP world is Composer, which allows us to use libraries (or packages) that aren't specific to Laravel. In this article, we'll get up-and-running with using bundles, and we'll even create our own bundle that others can download. We'll also see how to incorporate Composer into our Laravel installation to open up a wide range of PHP libraries that we can use in our application. Downloading and installing packages One of the best features of Laravel is how modular it is. Most of the framework is built using libraries, or packages, that are well tested and widely used in other projects. By using Composer for dependency management, we can easily include other packages and seamlessly integrate them into our Laravel app. For this recipe, we'll be installing two popular packages into our app: Jeffrey Way's Laravel 4 Generators and the Imagine image processing packages. Getting ready For this recipe, we need a standard installation of Laravel using Composer. How to do it... For this recipe, we will follow these steps: Go to https://packagist.org/. In the search box, search for way generator as shown in the following screenshot: Click on the link for way/generators : View the details at https://packagist.org/packages/way/generators and take notice of the require line to get the package's version. For our purposes, we'll use "way/generators": "1.0.*" . In our application's root directory, open up the composer.json file and add in the package to the require section so it looks like this: "require": { "laravel/framework": "4.0.*", "way/generators": "1.0.*" }, Go back to http://packagist.org and perform a search for imagine as shown in the following screenshot: Click on the link to imagine/imagine and copy the require code for dev-master : Go back to our composer.json file and update the require section to include the imagine package . It should now look similar to the following code: "require": { "laravel/framework": "4.0.*", "way/generators": "1.0.*", "imagine/imagine": "dev-master" }, Open the command line, and in the root of our application, run the Composer update as follows: php composer.phar update Finally, we'll add the Generator Service Provider, so open the app/config/app.php file and in the providers array, add the following line: 'WayGeneratorsGeneratorsServiceProvider' How it works... To get our package, we first go to packagist.org and search for the package we want. We could also click on the Browse packages link. It will display a list of the most recent packages as well as the most popular. After clicking on the package we want, we'll be taken to the detail page, which lists various links including the package's repository and home page. We could also click on the package's maintainer link to see other packages they have released. Underneath, we'll see the various versions of the package. If we open that version's detail page, we'll find the code we need to use for our composer.json file. We could either choose to use a strict version number, add a wildcard to the version, or use dev-master, which will install whatever is updated on the package's master branch. For the Generators package, we'll only use Version 1.0, but allow any minor fixes to that version. For the imagine package, we'll use dev-master, so whatever is in their repository's master branch will be downloaded, regardless of version number. We then run update on Composer and it will automatically download and install all of the packages we chose. Finally, to use Generators in our app, we need to register the service provider in our app's config file. Using the Generators package to set up an app Generators is a popular Laravel package that automates quite a bit of file creation. In addition to controllers and models, it can also generate views, migrations, seeds, and more, all through a command-line interface. Getting ready For this recipe, we'll be using the Laravel 4 Generators package maintained by Jeffrey Way that was installed in the Downloading and installing packages recipe. We'll also need a properly configured MySQL database. How to do it… Follow these steps for this recipe: Open the command line in the root of our app and, using the generator, create a scaffold for our cities as follows: php artisan generate:scaffold cities --fields="city:string" In the command line, create a scaffold for our superheroes as follows: php artisan generate:scaffold superheroes --fields="name:string, city_id:integer:unsigned" In our project, look in the app/database/seeds directory and find a file named CitiesTableSeeder.php. Open it and add some data to the $cities array as follows: <?php class CitiesTableSeeder extends Seeder { public function run() { DB::table('cities')->delete(); $cities = array( array( 'id' => 1, 'city' => 'New York', 'created_at' => date('Y-m-d g:i:s',time()) ), array( 'id' => 2, 'city' => 'Metropolis', 'created_at' => date('Y-m-d g:i:s',time()) ), array( 'id' => 3, 'city' => 'Gotham', 'created_at' => date('Y-m-d g:i:s',time()) ) ); DB::table('cities')->insert($cities); } } In the app/database/seeds directory, open SuperheroesTableSeeder.php and add some data to it: <?php class SuperheroesTableSeeder extends Seeder { public function run() { DB::table('superheroes')->delete(); $superheroes = array( array( 'name' => 'Spiderman', 'city_id' => 1, 'created_at' => date('Y-m-d g:i:s', time()) ), array( 'name' => 'Superman', 'city_id' => 2, 'created_at' => date('Y-m-d g:i:s', time()) ), array( 'name' => 'Batman', 'city_id' => 3, 'created_at' => date('Y-m-d g:i:s', time()) ), array( 'name' => 'The Thing', 'city_id' => 1, 'created_at' => date('Y-m-d g:i:s', time()) ) ); DB::table('superheroes')->insert($superheroes); } } In the command line, run the migration then seed the database as follows: php artisan migrate php artisan db:seed Open up a web browser and go to http://{your-server}/cities. We will see our data as shown in the following screenshot: Now, navigate to http://{your-server}/superheroes and we will see our data as shown in the following screenshot: How it works... We begin by running the scaffold generator for our cities and superheroes tables. Using the --fields tag, we can determine which columns we want in our table and also set options such as data type. For our cities table, we'll only need the name of the city. For our superheroes table, we'll want the name of the hero as well as the ID of the city where they live. When we run the generator, many files will automatically be created for us. For example, with cities, we'll get City.php in our models, CitiesController.php in controllers, and a cities directory in our views with the index, show, create, and edit views. We then get a migration named Create_cities_table.php, a CitiesTableSeeder.php seed file, and CitiesTest.php in our tests directory. We'll also have our DatabaseSeeder.php file and our routes.php file updated to include everything we need. To add some data to our tables, we opened the CitiesTableSeeder.php file and updated our $cities array with arrays that represent each row we want to add. We did the same thing for our SuperheroesTableSeeder.php file. Finally, we run the migrations and seeder and our database will be created and all the data will be inserted. The Generators package has already created the views and controllers we need to manipulate the data, so we can easily go to our browser and see all of our data. We can also create new rows, update existing rows, and delete rows.
Read more
  • 0
  • 0
  • 7807

article-image-social-networks
Packt
29 Oct 2013
15 min read
Save for later

Social Networks

Packt
29 Oct 2013
15 min read
(For more resources related to this topic, see here.) One window to rule them all Since we want to give our users the ability to post their messages on multiple social networks at once, it makes perfect sense to keep our whole application in a single window. It will be comprised of the following sections: The top section of the window will contain labels and a text area for message input. The text area will be limited to 140 characters in order to comply with Twitter's message limitation. As the user types his or her message, a label showing the number of characters will be updated. The bottom section of the window will use an encapsulating view that will contain multiple image views. Each image view will represent a social network to which the application will post the messages (in our case, Twitter and Facebook). But we can easily add more networks that will have their representation in this section. Each image view acts as a toggle button in order to select if the message will be sent to a particular social network or not. Finally, in the middle of those two sections, we will add a single button that will be used to publish the message from the text area. Code while it is hot! With our design in hand, we can now move on to create our user interface. Our entire application will be contained in a single window, so this is the first thing we will create. We will give it a title as well as a linear background gradient that changes from purple at the top to black at the bottom of the screen. Since we will add other components to it, we will keep its reference in the win variable: var win = Ti.UI.createWindow({ title: 'Unified Status', backgroundGradient: { type: 'linear', startPoint: { x: '0%', y: '0%' }, endPoint: { x: '0%', y: '100%' }, colors: [ { color: '#813eba'}, { color: '#000' } ] } }); The top section We will then add a new white label at the very top of the window. It will span 90% of the window's width, have a bigger font than other labels, and will have its text aligned to the left. Since this label will never be accessed later on, we will invoke our createLabel function right into the add function of the window object: win.add(Ti.UI.createLabel({ text: 'Post a message', color: '#fff', top: 4, width: '90%', textAlign: Ti.UI.TEXT_ALIGNMENT_LEFT, font: { fontSize: '22sp' } })); We will now move on to create our text area where our users will enter their messages. It will be placed right under the label previously created, occupy 90% of the screen's width, have a slightly bigger font, and have a thick, rounded, dark border. It will also be limited to 140 characters. It is also important that we don't forget to add this newly created object to our main window: var txtStatus = Ti.UI.createTextArea({ top: 37, width: '90%', height: 100, color: '#000', maxLength: 140, borderWidth: 3, borderRadius: 4, borderColor: '#401b60', font: { fontSize: '16sp' } }); win.add(txtStatus); The second label that will complement our text area will be used to indicate how many characters are currently present in the user's message. So, we will now create a label just underneath the text area and assign it a default text value. It will span 90% of the screen's width and have its text aligned to the right. Since this label will be updated dynamically when the text area's value changes, we will keep its reference in a variable named lblCount. As we did with our previous UI components, we will add our label to our main window using the following code: var lblCount = Ti.UI.createLabel({ text: '0/140', top: 134, width: '90%', color: '#fff', textAlign: Ti.UI.TEXT_ALIGNMENT_RIGHT }); win.add(lblCount); The last control from the top section will be our Post button. It will be placed right under the text area and centered horizontally using the following code: var btnPost = Ti.UI.createButton({ title: 'Post', top: 140, width: 150 }); win.add(btnPost); By not specifying any left or right property, the component is automatically centered. This is pretty useful to keep in mind while designing our user interfaces, as it frees us from having to do calculations in order to center something on the screen. Staying within the limits Even though the text area's maxLength property will ensure that the message length will not exceed the limitation we have set, we need to give our users feedback as they are typing their message. To achieve this, we will add an event listener on the change event of our text area. Every time; the text area's content changes, we will update the lblCount label with the number of characters our message contains: txtStatus.addEventListener('change', function(e) { lblCount.text = e.value.length + '/140'; We will also add a condition to check if our user's message is close to reaching its limit. If that is the case, we will change the label's color to red, if not, it will return to its original color: if (e.value.length > 120) { lblCount.color = 'red'; } else { lblCount.color = 'white' } Our last conditional check will be enabling the Post button only if there is actually a message to be posted. This will prevent our users from posting empty messages: btnPost.enabled = !(e.value.length === 0); }); Setting up our Post button Probably the most essential component in our application would be the Post button. We won't be able to post anything online (yet) by clicking on it, as there are things we will still need to add. We will add an event listener for the click event on the Post button. If the on-screen keyboard is displayed, we will call the blur function of the text area in order to hide it: btnPost.addEventListener('click', function() { txtStatus.blur(); Also, we will reset the value to the text area and the character count on the label, so that the interface is ready to enter a new message: txtStatus.value = ''; lblCount.text = '0/140'; }); The bottom section With all our message input mechanisms in place, we will now move on to our bottom section. All components from this section will be contained in a view that will be placed at the bottom of the screen. It will span 90% of the screen's width, and its height will adapt to its content. We will store its reference in a variable named bottomView for later use: var bottomView = Ti.UI.createView({ bottom: 4, width: '90%', height: Ti.UI.SIZE }); We will then create our toggle switches for each social network that our application interacts with. Since we want something sexier than regular switch components, we will create our own switch using a regular image view. Our first image view will be used to toggle the use of Facebook. It will have a dark blue background (similar to Facebook's logo) and will have a background image representing Facebook's logo with a gray background, so that it appears disabled. It will be positioned to the left of its parent view, and will have a borderRadius value of 4 in order to give it a rounded aspect. We will keep its reference in the fbView variable and then add this same image view to our bottom view using the following code: var fbView = Ti.UI.createImageView({ backgroundColor: '#3B5998', image: 'images/fb-logo-disabled.png', borderRadius: 4, width: 100, left: 10, height: 100 }); bottomView.add(fbView); We will create a similar image view (in almost every way), but this time for Twitter. So the background color and the image will be different. Also, it will be positioned to the right of our container view. We will store its reference in the twitView variable and then add it to our bottom view using the following code: var twitView = Ti.UI.createImageView({ backgroundColor: '#9AE4E8', image: 'images/twitter-logo-disabled.png', borderRadius: 4, width: 100, right: 10, height: 100 }); bottomView.add(twitView); Last but not the least, it is imperative that we do not forget to add our bottomView container object into our window. Also, we will open the window so that our users can interact with it using the following code: win.add(bottomView); win.open(); What if the user rotates the device? At this stage, if our user were to rotate the device (to landscape), nothing would happen on the screen. The reason behind this is because we have not taken any action to make our application compatible with the landscape mode. In many cases, this would require some changes to how the user interface is created depending on the orientation. But since our application is fairly simple, and most of our layout relies on percentages, we can activate the landscape mode without any modification to our code. To activate the landscape mode, we will update the orientations section from our tiapp.xml configuration file. It is mandatory to have at least one orientation present in this section (it doesn't matter which one it is). We want our users to be able to use the application, no matter how they hold their device: <iphone> <orientations device="iphone"> <orientation>Ti.UI.PORTRAIT</orientation> <orientation>Ti.UI.UPSIDE_PORTRAIT</orientation> <orientation>Ti.UI.LANDSCAPE_LEFT</orientation> <orientation>Ti.UI.LANDSCAPE_RIGHT</orientation> </orientations> </iphone> By default, no changes are required for Android applications since the default behavior supports orientation changes. There are, of course, ways to limit orientation in the manifest section, but this subject falls out of this article's scope. See it in action We have now implemented all the basic user interface. We can now test it and see if it behaves as we anticipated. We will click on the Run button from the App Explorer tab, as we did many times before. We now have our text area at the top with our two big images views at the bottom, each with the social network's logo. We can already test the message entry (with the character counter incrementing) by clicking on the Post button. Now if we rotate the iOS simulator (or the Android emulator), we can see that our layout adapts well to the landscape mode. To rotate the iOS simulator, you need to use the Hardware menu or you can use Cmd-Left and Cmd-Right on the keyboard. If you are using the Android emulator, there is no menu, but you can change the orientation using Ctrl + F12. The reason this all fits so well is because most of our dimensions are done using percentages. That means that our components will adapt and use the available space on the screen. Also, we positioned the bottom view using the bottom property; which meant that it will stick to the bottom of the screen, no matter how tall it is. There is a module for that Since we don't want to interact with the social network through manual asynchronous HTTP requests, we will leverage the Facebook native module provided with Titanium. Since it comes with Titanium SDK, there is no need to download or copy any file. All we need to do is add a reference (one for each target platform), in the modules section of our tiapp.xml file as follows: <modules> <module platform="android">facebook</module> <module platform="iphone">facebook</module> </modules> Also, we need to add the following property at the end of our configuration file, and replace the FACEBOOK_APPID parameter with the application ID that was provided when we created our app online. <property name="ti.facebook.appid">[FACEBOOK_APPID]</property> Why do we have to reference a module even though it comes bundled with the Titanium framework? Mostly, it is to avoid framework bloat; it is safe to assume that most applications developed using Titanium won't require interaction with Facebook. This is the reason for having it in a separate module that can be loaded on demand. Linking our mobile app to our Facebook app With our Facebook module loaded, we will now populate the necessary properties before making any call to the network. We will need to set the appid property in our code; since we have already defined it as a property in our tiapp.xml file, we can access it through the Properties API. This is a neat way to externalize application parameters, thus preventing us to hardcode them in our JavaScript code: fb.appid = Ti.App.Properties.getString('ti.facebook.appid'); We will also set the proper permissions that we will need while interacting with the server. (In this case, we only want to publish messages on the user's wall): fb.permissions = ['publish_actions']; It is important to set the appid and permissions properties before calling the authorize function. This makes sense since we want Facebook to authorize our application with a defined set of permissions from the get-go. Allowing our user to log in and log out at the click of a button We want our users to be able to connect (and disconnect) at their will from one social network, just by pressing the same view on the screen. To achieve this, we will create a function called toggleFacebook that will have a single parameter: function toggleFacebook(isActive) { If we want the function to make the service active, then we will verify if the user is already logged in to Facebook. If not, we will ask Facebook to authorize the application using the function with the same name. If the parameter indicates that we want to make the service inactive, we will log out from Facebook altogether: if (isActive) { if (!fb.loggedin) { fb.authorize(); } } else { fb.logout(); } } Now, all that we need to do is create an event listener on the click event for our Facebook image view and simply toggle between the two states depending whether the user is logged in or not: fbView.addEventListener('click', function() { toggleFacebook(!fb.loggedIn); }); The authorize function prompts the user to log in (if he or she is not already logged in), and authorize his or her application. It kind of makes sense that Facebook requires user validation before delegating the right to post something on his or her behalf. Handling responses from Facebook We have now completely implemented the Facebook login/logout mechanism into our application, but we still need to provide some feedback to our users. The Facebook module provides two event listeners that allow us to track when our user will have logged in or out. In the login event listener, we will check if the user is logged in successfully. If he or she did, we will update the image view's image property with the colored logo. If there was any error during authentication, or if the operation was simply cancelled, we will show an alert, as given in the following code: fb.addEventListener('login', function(e) { if (e.success) { fbView.image = 'images/fb-logo.png'; } else if (e.error) { alert(e.error); } else if (e.cancelled) { alert("Canceled"); } }); In the logout event listener, we will update the image view's image property with the grayed out Facebook logo using the following code: fb.addEventListener('logout', function(e) { fbView.image = 'images/fb-logo-disabled.png'; }); Posting our message on Facebook Since we are now connected to Facebook, and our application is authorized to post, we can now post our messages on this particular social network. To do this, we will create a new function named postFacebookMessage, with a single parameter, and that will be the message string to be posted: function postFacebookMessage(msg) { Inside this function, we will call the requestWithGraphPath function from the Facebook native module. This function can look fairly complex at first glance, but we will go over each parameter in detail. The parameters are as follows: The Graph API path requested (My feed). A dictionary object containing all of the properties required by the call (just the message). The HTTP method used for this call (POST). The callback function invoked when the request completes (this function simply checks the result from the call. In case of error, an alert is displayed). fb.requestWithGraphPath('me/feed', { message: msg }, "POST", function(e) { if (e.success) { Ti.API.info("Success! " + e.result); } else { if (e.error) { alert(e.error); } else { alert("Unknown result"); } } } ); } We will then update the click event handler for the Post button and call the postFacebookMessage function if the user is logged in to Facebook: btnPost.addEventListener('click', function() { if (fb.loggedIn) { postFacebookMessage(txtStatus.value); } ... }); With this, our application can post messages on our user's Facebook wall. Summary In this article, we learned how to create server-side applications on two popular social networking websites. We learned how to interact with those networks in terms of API and authentication. We also learned how to handle device rotation as well as use the native platform setting Windows. Finally, we covered Titanium menus and activities. Resources for Article: Further resources on this subject: Appcelerator Titanium: Creating Animations, Transformations, and Understanding Drag-and-drop [Article] augmentedTi: The application architecture [Article] Basic Security Approaches [Article]
Read more
  • 0
  • 0
  • 5839

Packt
29 Oct 2013
10 min read
Save for later

RESS – The idea and the Controversies

Packt
29 Oct 2013
10 min read
(For more resources related to this topic, see here.) The RWD concept appeared first in 2010 in an article by Ethan Marcotte (available at http://alistapart.com/article/responsive-web-design). He presented an approach that allows us to progressively enhance page design within different viewing contexts with the help of fluid grids, flexible images, and media queries. This approach was opposed to the one that separates websites geared toward specific devices. Instead of two or more websites (desktop and mobile), we could have one that adapts to all devices. The technical foundation of RWD (as proposed in Marcotte's article) consists of three things, fluid grids, flexible images, and media queries. Illustration: Fluid (and responsive) grid adapts to device using both column width and column count Fluid grid is basically nothing more than a concept of dividing the monitor width into modular columns, often accompanied by some kind of a CSS framework (some of the best-known examples were the 960 grid system, blueprint, pure, 1140px grid, and elastic), that is, a base stylesheet that simplifies and standardizes writing website-specific CSS. What makes it fluid is the use of relative measurements like %, em, or rem. With changing the screen (or the window), the number of these columns changes (thanks to CSS statements enclosed in media queries). This allows us to adjust the design layout to device capabilities (screen width and pixel density in particular). Images in such a layout become fluid by using a simple technique of setting width, x% or max-width, 100% in CSS, which causes the image to scale proportionally. With those two methods and a little help from media queries, one can radically change the page layout and handle this enormous, up to 800 percent, difference between the thinnest and the widest screen (WQXGA's 2560px/iPhone's 320px). This is a big step forward and a good base to start creating One Web, that is, to use one URL to deliver content to all the devices. Unfortunately, that is not enough to achieve results that would provide an equally great experience and fast loading websites for everybody. The RESS idea Besides screen width, we may need to take into account other things such as bandwidth and pay-per-bandwidth plans, processor speed, available memory, level of HTML/CSS compatibility, monitoring color depth, and possible navigation methods (touch screen, buttons, and keyboard). On a practical level, it means we may have to optimize images and navigation patterns, and reduce page complexity for some devices. To make this possible, some Server Side solutions need to be engaged. We may use Server Side just for optimizing images. Server Side optimization lets us send pages with just some elements adjusted or a completely changed page; we can rethink the application structure to build a RESTful web interface and turn our Server Side application into a web service. The more we need to place responsibility for device optimization on the Server Side, the closer we get to the old way of disparate desktops and mobile web's separate mobile domains, such as iPhone, Android, or Windows applications. There are many ways to build responsive websites but there is no golden rule to tell you which way is the best. It depends on the target audience, technical contexts, money, and time. Ultimately, the way to be chosen depends on the business decisions of the website owner. When we decide to employ Server Side logic to optimize components of a web page designed in a responsive way, we are going the RESS (Responsive Web Design with Server Side components) way. RESS was proposed by Luke Wroblewski on his blog as a result of his experiences on extending RWD with Server Side components. Essentially, the idea was based on storing IDs of resources (such as images) and serving different versions of the same resource, optimized for some defined classes of devices. Device detection and assigning them to respective classes can be based on libraries such as WURFL or YABFDL. Controversies It is worth noting that both of these approaches raised many controversies. Introducing RWD has broken some long-established rules or habits such as standard screen width (the famous 960px maximum page width limit). It has put in question the long-practiced ways of dealing with mobile web (such as separate desktop and mobile websites). It is no surprise that it raises both delight and rage. One can easily find people calling this fool's gold, useless, too difficult, a fad, amazing, future proof, and so on. Each of those opinions has a reason behind it, for better or worse. A glimpse of the following opinions may help us understand some of the key benefits and issues related to RWD. "Separate mobile websites are a good thing" You may have heard this line in an article by Jason Grigsby, Css media query for mobile is fool's gold , available at http://blog.cloudfour.com/css-media-query-for-mobile-is-fools-gold/. Separate mobile websites allow reduction of bandwidth, prepare pages that are less CPU and memory intensive, and at the same time allow us to use some mobile-only features such as geolocation. Also, not all mobile browsers are wise enough to understand media queries. That is generally true and media queries are not enough in most scenarios, but with some JavaScript (Peter-Paul Koch blog available at, http://www.quirksmode.org/blog/archives/2010/08/combining_media.html#more, describes a method to exclude some page elements or change the page structure via JS paired with media queries), it is possible to overcome many of those problems. At the same time, making a separate mobile website introduces its own problems and requires significant additional investment that can easily get to tens or hundreds of times more than the RWD solution (detecting devices, changing application logic, writing separate templates, integrating, and testing the whole thing). Also, at the end of the day, your visitors may prefer the mobile version, but this doesn't have to be the case. Users are often accessing the same content via various devices and providing consistent experience across all of them becomes more and more important. The preceding controversy is just a part of a wider discussion on channels to provide content on the Internet. RWD and RESS are relatively new kids on the block. For years, technologies to provide content for mobile devices were being built and used, from device-detection libraries to platform-specific applications (such as iStore, Google Play, and MS). When, in 2010, US smartphone users started to spend more time using their mobile apps than browsers (Mobile App Usage Further Dominates Web, Spurred by Facebook, at http://blog.flurry.com/bid/80241/Mobile-App-Usage-Further-Dominates-Web-Spurred-by-Facebook), some hailed it as dangerous for the Web (Apps: The Web Is The Platform, available at http://blog.mozilla.org/webdev/2012/09/14/apps-the-web-is-the-platform/). A closer look at stats reveals though, that most of this time was spent on playing games. No matter how much time kids can spend playing Angry Birds now, after more than two years from then, people still prefer to read the news via a browser rather than via native mobile applications. The Future of Mobile News report from October 2012 reveals that for accessing news, 61 percent mobile users prefer a browser while 28 percent would rather use apps (Future of Mobile News, http://www.journalism.org/analysis_report/future_mobile_news). The British government is not keen on apps either, as they say, "Our position is that native apps are rarely justified" (UK Digital Cabinet Office blog, at http://digital.cabinetoffice.gov.uk/2013/03/12/were-not-appy-not-appy-at-all/). Recently, Tim Berners-Lee, the inventor of the Web, criticized closed world apps such as those released by Apple for threatening openness and universality that the architects of the Internet saw as central to its design. He explains it the following way, "When you make a link, you can link to anything. That means people must be able to put anything on the Web, no matter what computer they have, what software they use, or which human language they speak and regardless of whether they have a wired or a wireless Internet connection." This kind of thinking goes in line with the RWD/RESS philosophy to have one URL for the same content, no matter what way you'd like to access it. Nonetheless, it is just one of the reasons why RWD became so popular during the last year. "RWD is too difficult" CSS coupled with JS can get really complex (some would say messy) and requires a lot of testing on all target browsers/platforms. That is or was true. Building RWD websites requires good CSS knowledge and some battlefield experience in this field. But hey, learning is the most important skill in this industry. It actually gets easier and easier with new tools released nearly every week. "RWD means degrading design" Fluid layouts break the composition of the page; Mobile First and Progressive Enhancement mean, in fact, reducing design to a few simplistic and naive patterns. Actually the Mobile First concept contains two concepts. One is design direction and the second is the structure of CSS stylesheets, in particular the order of media queries. With regard to design direction, the Mobile First concept is meant to describe the sequence of designs. First the design for a mobile should be created and then for a desktop. While there are several good reasons for using this approach, one should never forget the basic truth that at the end of the day only the quality of designs matters, not the order they were created in. With regard to the stylesheet structure, Mobile First means that we first write statements for small screens and then add statements for wider screens, such as @media screen and (min-width: 480px). It is a design principle meant to simplify the whole thing. It is assumed here that CSS for small screens is the simplest version, which will be progressively enhanced for larger screens. The idea is smart and helps to maintain a well-structured CSS but sometimes the opposite, the Desktop First approach, seems natural. Typical examples are tables with many columns. The Mobile First principle is not a religious dogma and should not be treated as such. As a side note, it remains an open question why this is still named Mobile First, while the new iPad-related statements should come here at the end (min-width: 2000px). There are some examples of rather poor designs made by RWD celebrities. But there are also examples of great designs that happened, thanks to the freedom that RWD gave to the web design world. The rapid increase in Internet access via mobile devices during 2012 made RWD one of the hottest topics in web design. The numbers vary across countries and websites but no matter what numbers you look at, one thing is certain, mobile is already big and will soon get even bigger (valuable stats on mobile use are available at http://www.thinkwithgoogle.com/mobileplanet/en/). Statistics are not the only reason why Responsive Web Design became popular. Equally important are the benefits for web designers, users, website owners, and developers. Summary This article, as discussed, covered the RESS idea, as well as the controversies associated with it. Resources for Article: Further resources on this subject: Introduction to RWD frameworks [Article] Getting started with Modernizr using PHP IDE [Article] Understanding Express Routes [Article]
Read more
  • 0
  • 0
  • 11690
article-image-getting-store
Packt
28 Oct 2013
21 min read
Save for later

Getting into the Store

Packt
28 Oct 2013
21 min read
(For more resources related to this topic, see here.) This all starts by visiting https://appdev.microsoft.com/StorePortals, which will get you to the store dashboard that you use to submit and manage your applications. If you already have an account you'll just log in here and proceed. If not, we'll take a look at ways of getting it set up. There are a couple of ways to get a store account, which you will need before you can submit any game or application to the store. There are also two different types of accounts: Individual accounts Company accounts In most cases you will only need the first option. It's cheaper and easier to get, and you won't require the enterprise features provided by the company account for a game. For this reason we'll focus on the individual account. To register you'll need a credit card for verification, even if you gain a free account another way. Just follow the registration instructions, pay the fee, and complete verification, after which you'll be ready to go. Free accounts Students and developers with MSDN subscriptions can access registration codes that waive the fee for a minimum of one year. If you meet either of these requirements you can gain a code using the respective methods, and use that code during the registration process to set the fee to zero. Students can access their free accounts using the DreamSpark service that Microsoft runs. To access this you need to create an account on www.dreamspark.com and create an account. From there follow the steps to verify your student status and visit https://www.dreamspark.com/Student/Windows-Store-Access.aspx to get your registration code. If you have access to an MSDN subscription you can use this to gain a store account for free. Just log in to your account and in your account benefits overview you should be able to generate your registration code. Submitting your game So your game is polished and ready to go. What do you need to do to get it in the store? First log in to the dashboard and select Submit an App from the menu on the left. Here you can see the steps required to submit the app. This may look like a lot to do, but don't worry. Most of these are very simple to resolve and can be done before you even start working on the game. The first step is to choose a name for your game, and this can be done whenever you want. By reserving a name and creating the application entry you have a year to submit your application, giving you plenty of time to complete it. This is why it's a good idea to jump in and register your application once you have a name for it. If you change your mind later and want a different name you can always change it. The next step is to choose how and where you will sell your game. The other thing you need to choose here are the markets you want to sell your game in. This can be an area you need to be careful of, because the markets you choose here define the localization or content you need to watch for in your game. Certain markets are restrictive and including content that isn't appropriate for a market you say you want to sell in can cause you to fail the certification process. Once that is done you need to choose when you want to release your game—you can choose to release as soon as certification finishes or on a specific date, and then you choose the app category, which in this case will be Games. Don't forget to specify the genre of your game as the sub-category so players can find it. The final option on the Selling Details page which applies to us is the Hardware requirements section. Here we define the DirectX feature-level required for the game, and the minimum RAM required to run it. This is important because the store can help ensure that players don't try to play your game on systems that cannot run it. The next section allows you to define the in-app offers that will be made available to players. The Age rating and rating certificates section allows you to define the minimum age required to play the game, as well as submit official ratings certificates from ratings boards so that they may be displayed in the store to meet legal requirements. The latter part is optional in some cases, and may affect where you can submit your game depending on local laws. Aside from official ratings, all applications and games submitted to the store require a voluntary rating, chosen from one of the following age options: 3+ 7+ 12+ 16+ 18+ While all content is checked, the 7+ and 3+ ratings both have extra checks because of the extra requirements for those age ranges. The 3+ rating is especially restrictive as apps submitted with that age limit may not contain features that could connect to online services, collect personal information, or use the webcam or microphone. To play it safe it's recommended the 12+ rating is chosen, and if you're still uncertain, higher is safer. GDF Certificates The other entry required here if you have official ratings certificates is a GDF file. This is a Game Definition File, which defines the different ratings in a single location and provides the necessary information to display the rating and inform any parental settings. To do this you need to use the GDFMAKER.exe utility that ships with the Windows 8 SDK, and generate a GDF file which you can submit to the store. Alongside that you need to create a DLL containing that file (as a resource) without any entry point to include in the application package. For full details on how to create the GDF as well as the DLL, view the following MSDN article: http://msdn.microsoft.com/en-us/library/windows/apps/hh465153.aspx The final section before you need to submit your compiled application package is the cryptography declaration. For most games you should be able to declare that you aren't using any cryptography within the game and quickly move through this step. If you are using cryptography, including encrypting game saves or data files, you will need to declare that here and follow the instructions to either complete the step or provide an Export Control Classification Number (ECCN). Now you need to upload the compiled app package before you can continue, so we'll take a look at what it takes to do that before you continue. App packages To submit your game to the store, you need to package it up in a format that makes it easy to upload, and easy for the store to distribute. This is done by compiling the application as an .appx file. But before that happens we need to ensure we have defined all of the required metadata, and fulfill the certification requirements, otherwise we'll be uploading a package only to fail soon after. Part of this is done through the application manifest editor, which is accessible in Visual Studio by double-clicking on the Package.appxmanifest file in solution explorer. This editor is where you specify the name that will be seen in the start menu, as well as the icons used by the application. To pass certification all icons have to be provided at 100 percent DPI, which is referred to as Scale 100 in the editor. Icon/Image Base resolution Required Standard 150 x 150 px Yes Wide 310 x 150 px If Wide Tile Enabled Small 30 x 30 px Yes Store 50 x 50 px Yes Badge 24 x 24 px If Toasts Enabled Splash 620 x 300 px Yes If you wish to provide a higher quality images for people running on high DPI setups, you can do so with a simple filename change. If you add scale-XXX to your filename, just before the extension, and replace XXX with one of the following values, Windows will automatically make use of it at the appropriate DPI. scale-100 scale-140 scale-180 In the following image you can see the options available for editing the visual assets in the application. These all apply to the start menu and application start-up experience, including the splash screen and toast notifications. Toast Notifications in Windows 8 are pop-up notifications that slide in from the edge of the screen and show the users some information for a short period of time. They can click on the toast to open the application if they want. Alongside Live Tiles, Toast Notifications allow you to give the user information when the application is not running (although they work when the application is running). The previous table shows the different images you required for a Windows 8 application, and if they are mandatory or just required in certain situations. Note that this does not include the imagery required for the store, which includes some screenshots of the application and optional promotional art in case you want your application to be featured. You must replace all of the required icons with your own. Automated checks during certification will detect the use of the default "box" icon shown in the previous screenshot and automatically fail the submission. Capabilities Once you have the visual aspects in place, you need to declare the capabilities that the application will receive. Your game may not need any, however you should still only specify what you need to run, as some of these capabilities come with extra implications and non-obvious requirements. Adding a privacy policy One of those requirements is the privacy policy. Even if you are creating a game, there may be situations where you are collecting private information, which requires you to have a privacy policy. The biggest issue here is connecting to the internet. If your game marks any of the Internet capabilities in the manifest, you automatically trigger a check for a privacy policy as private information—in this case an IP address—is being shared. To avoid failing certification for this, you need to put together a privacy policy if you collect privacy information, or use any of the capabilities that would indicate you collect information. These include the Internet capabilities as well as location, webcam, and microphone capabilities. This privacy policy just needs to describe what you will do with the information, and directly mention your game and publisher name. Once you have the policy written, it needs to be posted in two locations. The first is a publicly accessible website, which you will provide a link to when filling out the description after uploading your game. The second is within the game itself. It is recommended you place this policy in the Windows 8 provided settings menu, which you can build using XAML or your own code. If you're going with a completely native Windows 8 application you may want to display the policy in your own way and link to it from options within your game. Declarations Once you've indicated the capabilities you want, you need to declare any operating system integration you've done. For most games you won't use this, however if you're taking advantage of Windows 8 features such as share targets (the destination for data shared using the Share Charm), or you have a Game Definition File, you will need to declare it here and provide the required information for the operating system. In the case of the GDF, you need to provide the file so that the parental controls system can make use of the ratings to appropriately control access. Certification kit The next step is to make sure you aren't going to fail the automated tests during certification. Microsoft provides the same automated tests used when you submit your app in the Windows Application Certification Kit (WACK). WACK is installed by default with Visual Studio 2012 or higher version. There are two ways to run the test: after you build your application package, or by running the kit directly against an installed app. We'll look at the latter first, as you might want to run the test on your deployed test game well before you build anything for the store. This is also the only way to run the WACK on a WinRT device, if you want to cover all bases. If you haven't already deployed or tested your app, deploy it using the Build menu in Visual Studio and then search for the Windows App Cert Kit using the start menu (just start typing). When you run this you will be given an option to choose which type of application you want to validate. In this case we want to select the Windows Store App option, which will then give you access to the list of apps installed on your machine. From there it's just a matter of selecting the app you want and starting the test. At this point you will want to leave your machine alone until the automated tests are complete. Any interference could lead to an incorrect failure of the certification tests. The results will indicate ways you can fix any issues; however, you should be fine for most of the tests. The biggest issues will arise from third party libraries that haven't been developed or ported to Windows 8. In this case the only option is to fix them yourself (if they're open source) or find an alternative. Once you have the test passing, or you feel confident that it won't be an issue, you need to create app packages that are compatible with the store. At this point your game will be associated with the submission you have created in the Windows Store dashboard so that it is prepared for upload. Creating your app packages To do this, right click on your game project in Visual Studio and click on Create App Packages inside the Store menu. Once you do that, you'll be asked if you want to create a package for the store. The difference between the two options comes down to how the package is signed. If you choose No here, you can create a package with your test certificate, which can be distributed for testing. These packages must be manually installed and cannot be submitted to the store. You can, however, use this type of package on other machines to install your game for testers to try out. Choosing No will give you a folder with a .ps1 file (Powershell), which you can run to execute the install script. Choosing Yes at this option will take you to a login screen where you can enter your Windows Store developer account details. Once you've logged in you will be presented with a list of applications that you have registered with the store. If you haven't yet reserved the name of your application, you can click on the Reserve Name link, which will take you directly to the appropriate page in the store dashboard. Otherwise select the name of the game you're trying to build and click on Next. The next screen will allow you to specify which architectures to build for, and the version number of the built package. As this is a C++ game, we need to provide separate packages for the ARM, x86, and x64 builds, depending on what you want to support. Simply providing an x86 and ARM build will cover the entire market; 64 bit can be nice to have if you need a lot of memory, but ultimately it is optional, and some users may not even be able to run x64 code. When you're ready click on Create and Visual Studio will proceed to build your game and compile the requested packages, placing them in the directory specified. If you've built for the store, you will need the .appxupload files from this directory when you proceed to upload your game. Once the build has completed you will be asked if you want to launch the Windows Application Certification Kit. As mentioned previously this will test your game for certification failures, and if you're submitting to the store it's strongly recommended you run this. Doing so at this screen will automatically deploy the built package and run the test, so ensure you have a little bit of time to let it run. Uploading and submitting Now that you have a built app package you can return to the store dashboard to submit your game. Just edit the submission you made previously and enter the Packages section, which will take you to the page where you can upload the appxupload file. Once you have successfully uploaded your game you will gain access to the next section, the Description. This is where you define the details that will be displayed in the store. This is also where your marketing skills come into play as you prepare the content that will hopefully get players to buy your game. You start with the description of your game, and any big feature bullet points you want to emphasize. This is the best place to mention any reviews or praise, as well as give a quick description that will help the players decide if they want to try your game. You can have a number of app features listed; however, like any "back of the box" bullet points, keep them short and exciting. Along with the description, the store requires at least one screenshot to display to the potential player. These screenshots need to be of the entire screen, and that means they need to be at least 1366x768, which is the minimum resolution of Windows 8. These are also one of the best ways to promote your game, so ensure you take some great screenshots that show off the fun and appeal of your game. There are a few ways to take a screenshot of your game. If you're testing in the simulator you can use the screenshot icon on the right toolbar of the simulator to take the screenshot. If not, you can use Windows Key + Prt Scr SysRq to take a screenshot of your entire screen, and then use that (or edit it if you have multiple monitors). Screenshots taken with either of these tools can be found in the Screenshots folder within your Pictures library. There are two other small pieces of information that are required during this stage: Copyright Info and Support contact info. For the support info, an e-mail address will usually suffice. At this point you can also include your website and, if applicable to your game, a link to the privacy policy included in your game. Note that if you require a privacy policy, it must be included in two places: your game, and the privacy policy field on this form. The last items you may want to add here are promotional images. These images are intended for use in store promotions and allow Microsoft to easily feature your game with larger promotional imagery in prominent locations within the store. If you are serious about maximizing the reach of your game, you will want to include these images. If you don't, the number of places your game can be featured will be reduced. At a minimum the 414x180 px image should be included if you want some form of promotion. Now you're almost done! The next section allows you to leave notes for the testing team. This is where you would leave test account details for any features in your game that require an account so that they can test those features. This is also the location to leave any notes about testing in case there are situations where you can point out any features that might not be obvious. In certain situations you may have an exemption from Microsoft for a certification requirement; this would be where you include that exemption. When every step has been completed and you have tick marks in all of the stages, the Submit for Certification button will unlock, allowing you to complete your submission and send it off for certification. At this stage a number of automated tests will run before human testers will try your game on a variety of devices to ensure it fits the requirements for the store. If all goes well, you will receive an email notifying you of your successful certification and, depending on if you set the release date as ASAP, you will find your game in the store a few hours later (it may take a few hours to appear in the store once you receive an email informing you that your game or app is in the store). Certification tips Your first stop should be the certification requirements page, which lists all of the current requirements your game will be tested for: http://msdn.microsoft.com/en-us/library/windows/apps/hh694083.aspx. There are some requirements that you should take note of, and in this section we'll take a look at ways to help ensure you pass those requirements. Privacy The first of course is the privacy policy. As mentioned before, if your game collects any sort of personal information, you will need that policy in two places: In full text within the game Accessible through an Internet link The default app template generated by Visual Studio automatically enables the Internet capability, and by simply having that enabled you require a privacy policy. If you aren't connecting to the Internet at all in your game, you should always ensure that none of the Internet options are enabled before you package your game. If you share any personal information, then you need to provide players with a method of opting in to the sharing. This could be done by gating the functionality behind a login screen. Note that this functionality can be locked away, and the requirement doesn't demand that you find a way to remain fully functional even if the user opts out. Features One requirement is that your game support both touch input and keyboard/mouse input. You can easily support this by using an input system like the one described in this article; however, by supporting touch input you get mouse input for free and technically fulfill this requirement. It's all about how much effort you want to put into the experience your player will have, and that's why including gamepad input is recommended as some players may want to use a connected Xbox 360 gamepad for their input device in games. Legacy APIs Although your game might run while using legacy APIs, it won't pass certification. This is checked through an automated test that also occurs during the WACK testing process, so you can easily check if you have used any illegal APIs. This often arises in third party libraries which make use of parts of the standard IO library such as the console, or the insecure versions of functions such as strcpy or fopen. Some of these APIs don't exist in WinRT for good reason; the console, for example, just doesn't exist, so calling APIs that work directly with the console makes no sense, and isn't allowed. Debug Another issue that may arise through the use of third-party libraries is that some of them may be compiled in debug mode. This could present issues at runtime for your app, and the packaging system will happily include these when compiling your game, unless it has to compile them itself. This is detected by the WACK and can be resolved by finding a release mode version, or recompiling the library. WACK The final tip is: run WACK. This kit quickly and easily finds most of the issues you may encounter during certification, and you see the issues immediately rather than waiting for it to fail during the certification process. Your final step before submitting to the store should be to run WACK, and even while developing it's a good idea to compile in release mode and run the tests to just make sure nothing is broken. Summary By now you should know how to submit your game to the store, and get through certification with little to no issues. We've looked at what you require for the store including imagery and metadata, as well as how to make use of the Windows Application Certification Kit to find problems early on and fix them up without waiting hours or days for certification to fail your game. One area unique to games that we have covered in this article is game ratings. If you're developing your game for certain markets where ratings are required, or if you are developing children's games, you may need to get a rating certificate, and hopefully you have an idea of where to look to do this. Resources for Article: Further resources on this subject: Introduction to Game Development Using Unity 3D [Article] HTML5 Games Development: Using Local Storage to Store Game Data [Article] Unity Game Development: Interactions (Part 1) [Article]
Read more
  • 0
  • 0
  • 4710

article-image-low-level-index-control
Packt
28 Oct 2013
12 min read
Save for later

Low-Level Index Control

Packt
28 Oct 2013
12 min read
(For more resources related to this topic, see here.) Altering Apache Lucene scoring With the release of Apache Lucene 4.0 in 2012, all the users of this great, full text search library, were given the opportunity to alter the default TF/IDF based algorithm. Lucene API was changed to allow easier modification and extension of the scoring formula. However, that was not the only change that was made to Lucene when it comes to documents score calculation. Lucene 4.0 was shipped with additional similarity models, which basically allows us to use different scoring formula for our documents. In this section we will take a deeper look at what Lucene 4.0 brings and how those features were incorporated into ElasticSearch. Setting per-field similarity Since ElasticSearch 0.90, we are allowed to set a different similarity for each of the fields we have in our mappings. For example, let's assume that we have the following simple mapping that we use, in order to index blog posts (stored in the posts_no_similarity.json file): { "mappings" : { "post" : { "properties" : { "id" : { "type" : "long", "store" : "yes", "precision_step" : "0" }, "name" : { "type" : "string", "store" : "yes", "index" : "analyzed" }, "contents" : { "type" : "string", "store" : "no", "index" : "analyzed" } } } } } What we would like to do is, use the BM25 similarity model for the name field and the contents field. In order to do that, we need to extend our field definitions and add the similarity property with the value of the chosen similarity name. Our changed mappings (stored in the posts_similarity.json file) would appear as shown in the following code: { "mappings" : { "post" : { "properties" : { "id" : { "type" : "long", "store" : "yes", "precision_step" : "0" }, "name" : { "type" : "string", "store" : "yes", "index" : "analyzed", "similarity" : "BM25" }, "contents" : { "type" : "string", "store" : "no", "index" : "analyzed", "similarity" : "BM25" } } } } } And that's all, nothing more is needed. After the preceding change, Apache Lucene will use the BM25 similarity to calculate the score factor for the name and contents fields. In case of the Divergence from randomness and Information based similarity model, we need to configure some additional properties to specify the behavior of those similarities. How to do that is covered in the next part of the current section. Default codec properties When using the default codec we are allowed to configure the following properties: min_block_size: It specifies the minimum block size Lucene term dictionary uses to encode blocks. It defaults to 25. max_block_size: It specifies the maximum block size Lucene term dictionary uses to encode blocks. It defaults to 48. Direct codec properties The direct codec allows us to configure the following properties: min_skip_count: It specifies the minimum number of terms with a shared prefix to allow writing of a skip pointer. It defaults to 8. low_freq_cutoff: The codec will use a single array object to hold postings and positions that have document frequency lower than this value. It defaults to 32. Memory codec properties By using the memory codec we are allowed to alter the following properties: pack_fst: It is a Boolean option that defaults to false and specifies if the memory structure that holds the postings should be packed into the FST. Packing into FST will reduce the memory needed to hold the data. acceptable_overhead_ratio: It is a compression ratio of the internal structure specified as a float value which defaults to 0.2. When using the 0 value, there will be no additional memory overhead but the returned implementation may be slow. When using the 0.5 value, there can be a 50 percent memory overhead, but the implementation will be fast. Values higher than 1 are also possible, but may result in high memory overhead. Pulsing codec properties When using the pulsing codec we are allowed to use the same properties as with the default codec and in addition to them one more property, which is described as follows: freq_cut_off: It defaults to 1. The document frequency at which the postings list will be written into the term dictionary. The documents with the frequency equal to or less than the value of freq_cut_off will be processed. Bloom filter-based codec properties If we want to configure a bloom filter based codec, we can use the bloom_filter type and set the following properties: delegate: It specifies the name of the codec we want to wrap, with the bloom filter. ffp: It is a value between 0 and 1.0 which specifies the desired false positive probability. We are allowed to set multiple probabilities depending on the amount of documents per Lucene segment. For example, the default value of 10k=0.01, 1m=0.03 specifies that the fpp value of 0.01 will be used when the number of documents per segment is larger than 10.000 and the value of 0.03 will be used when the number of documents per segment is larger than one million. For example, we could configure our custom bloom filter based codec to wrap a direct posting format as shown in the following code (stored in posts_bloom_custom.json file): { "settings" : { "index" : { "codec" : { "postings_format" : { "custom_bloom" : { "type" : "bloom_filter", "delegate" : "direct", "ffp" : "10k=0.03, 1m=0.05" } } } } }, "mappings" : { "post" : { "properties" : { "id" : { "type" : "long", "store" : "yes", "precision_step" : "0" }, "name" : { "type" : "string", "store" : "yes", "index" : "analyzed", "postings_format" : "custom_bloom" }, "contents" : { "type" : "string", "store" : "no", "index" : "analyzed" } } } } } NRT, flush, refresh, and transaction log In an ideal search solution, when new data is indexed it is instantly available for searching. At the first glance it is exactly how ElasticSearch works even in multiserver environments. But this is not the truth (or at least not all the truth) and we will show you why it is like this. Let's index an example document to the newly created index by using the following command: curl -XPOST localhost:9200/test/test/1 -d '{ "title": "test" }' Now, we will replace this document and immediately we will try to find it. In order to do this, we'll use the following command chain: curl –XPOST localhost:9200/test/test/1 -d '{ "title": "test2" }' ; curl localhost:9200/test/test/_search?pretty The preceding command will probably result in the response, which is very similar to the following response: {"ok":true,"_index":"test","_type":"test","_id":"1","_version":2}{ "took" : 1, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "test", "_type" : "test", "_id" : "1", "_score" : 1.0, "_source" : { "title": "test" } } ] } } The first line starts with a response to the indexing command—the first command. As you can see everything is correct, so the second, search query should return the document with the title field test2, however, as you can see it returned the first document. What happened? But before we give you the answer to the previous question, we should take a step backward and discuss about how underlying Apache Lucene library makes the newly indexed documents available for searching. Updating index and committing changes The segments are independent indices, which means that queries that are run in parallel to indexing, from time to time should add newly created segments to the set of those segments that are used for searching. Apache Lucene does that by creating subsequent (because of write-once nature of the index) segments_N files, which list segments in the index. This process is called committing. Lucene can do this in a secure way—we are sure that all changes or none of them hits the index. If a failure happens, we can be sure that the index will be in consistent state. Let's return to our example. The first operation adds the document to the index, but doesn't run the commit command to Lucene. This is exactly how it works. However, a commit is not enough for the data to be available for searching. Lucene library use an abstraction class called Searcher to access index. After a commit operation, the Searcher object should be reopened in order to be able to see the newly created segments. This whole process is called refresh. For performance reasons ElasticSearch tries to postpone costly refreshes and by default refresh is not performed after indexing a single document (or a batch of them), but the Searcher is refreshed every second. This happens quite often, but sometimes applications require the refresh operation to be performed more often than once every second. When this happens you can consider using another technology or requirements should be verified. If required, there is possibility to force refresh by using ElasticSearch API. For example, in our example we can add the following command: curl –XGET localhost:9200/test/_refresh If we add the preceding command before the search, ElasticSearch would respond as we had expected. Changing the default refresh time The time between automatic Searcher refresh can be changed by using the index.refresh_interval parameter either in the ElasticSearch configuration file or by using the update settings API. For example: curl -XPUT localhost:9200/test/_settings -d '{ "index" : { "refresh_interval" : "5m" } }' The preceding command will change the automatic refresh to be done every 5 minutes. Please remember that the data that are indexed between refreshes won't be visible by queries. As we said, the refresh operation is costly when it comes to resources. The longer the period of refresh is, the faster your indexing will be. If you are planning for very high indexing procedure when you don't need your data to be visible until the indexing ends, you can consider disabling the refresh operation by setting the index.refresh_interval parameter to -1 and setting it back to its original value after the indexing is done. The transaction log Apache Lucene can guarantee index consistency and all or nothing indexing, which is great. But this fact cannot ensure us that there will be no data loss when failure happens while writing data to the index (for example, when there isn't enough space on the device, the device is faulty or there aren't enough file handlers available to create new index files). Another problem is that frequent commit is costly in terms of performance (as you recall, a single commit will trigger a new segment creation and this can trigger the segments to merge). ElasticSearch solves those issues by implementing transaction log. Transaction log holds all uncommitted transactions and from time to time, ElasticSearch creates a new log for subsequent changes. When something goes wrong, transaction log can be replayed to make sure that none of the changes were lost. All of these tasks are happening automatically, so, the user may not be aware of the fact that commit was triggered at a particular moment. In ElasticSearch, the moment when the information from transaction log is synchronized with the storage (which is Apache Lucene index) and transaction log is cleared is called flushing. Please note the difference between flush and refresh operations. In most of the cases refresh is exactly what you want. It is all about making new data available for searching. From the opposite side, the flush operation is used to make sure that all the data is correctly stored in the index and transaction log can be cleared. In addition to automatic flushing, it can be forced manually using the flush API. For example, we can run a command to flush all the data stored in the transaction log for all indices, by running the following command: curl –XGET localhost:9200/_flush Or we can run the flush command for the particular index, which in our case is the one called library: curl –XGET localhost:9200/library/_flush curl –XGET localhost:9200/library/_refresh In the second example we used it together with the refresh, which after flushing the data opens a new searcher. The transaction log configuration If the default behavior of the transaction log is not enough ElasticSearch allows us to configure its behavior when it comes to the transaction log handling. The following parameters can be set in the elasticsearch.yml file as well as using index settings update API to control transaction log behavior: index.translog.flush_threshold_period: It defaults to 30 minutes (30m). It controls the time, after which flush will be forced automatically even if no new data was being written to it. In some cases this can cause a lot of I/O operation, so sometimes it's better to do flush more often with less data being stored in it. index.translog.flush_threshold_ops: It specifies the maximum number of operations after which the flush operation will be performed. It defaults to 5000. index.translog.flush_threshold_size: It specifies the maximum size of the transaction log. If the size of the transaction log is equal to or greater than the parameter, the flush operation will be performed. It defaults to 200 MB. index.translog.disable_flush: This option disables automatic flush. By default flushing is enabled, but sometimes it is handy to disable it temporarily, for example, during import of large amount of documents. All of the mentioned parameters are specified for an index of our choice, but they are defining the behavior of the transaction log for each of the index shards. Of course, in addition to setting the preceding parameters in the elasticsearch.yml file, they can also be set by using Settings Update API. For example: curl -XPUT localhost:9200/test/_settings -d '{ "index" : { "translog.disable_flush" : true } }' The preceding command was run before the import of a large amount of data, which gave us a performance boost for indexing. However, one should remember to turn on flushing when the import is done. Near Real Time GET Transaction log gives us one more feature for free that is, real-time GET operation, which provides the possibility of returning the previous version of the document including non-committed versions. The real-time GET operation fetches data from the index, but first it checks if a newer version of that document is available in the transaction log. If there is no flushed document, data from the index is ignored and a newer version of the document is returned—the one from the transaction log. In order to see how it works, you can replace the search operation in our example with the following command: curl -XGET localhost:9200/test/test/1?pretty ElasticSearch should return the result similar to the following: { "_index" : "test", "_type" : "test", "_id" : "1", "_version" : 2, "exists" : true, "_source" : { "title": "test2" } } If you look at the result, you would see that again, the result was just as we expected and no trick with refresh was required to obtain the newest version of the document.
Read more
  • 0
  • 0
  • 2106
Modal Close icon
Modal Close icon