Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-vbnet-application-sql-anywhere-10-database-part-1
Packt
24 Oct 2009
4 min read
Save for later

VB.NET Application with SQL Anywhere 10 database: Part 1

Packt
24 Oct 2009
4 min read
SQL Anywhere 10 SQL Anywhere 10 is the latest version of Sybase's feature rich SQL Anywhere database technology. It is highly scalable from the small foot-print UltraLite database all the way to its enterprise server with gigabytes of data. It is a comprehensive database package with built-in support for a wide range of applications, including session based synchronization; data exchange with both relational and non-relational data bases; secure store and forward messaging; messaging with FTP and email; and asynchronous access to mobile web services. You may download an evaluation version of the software and take it for a test drive. Sybase Central is a graphical database management interface to the database and its various supporting applications. The integration features are used in this article to create a Windows application retrieving data from the SQL Anywhere 10’s demonstration database, a database which is a part of the default installation of the developer edition. Overview of SQL Anywhere 10 From Sybase Central you can connect to the demo database quite easily by clicking on the Connections menu item and choosing Connect with SQL Anywhere 10. Figure 1 shows the SQL Anywhere management interface, Sybase Central. Using this interface you may also create an ODBC DSN by following the trail; Tools --> SQL Anywhere 10 --> open ODBC Administrator. Figure 1   It is very easy to connect to the database using the ODBC driver which is provided with the default installation of this product. The Figure 2 shows the User DSN installed with the default installation in the ODBC Data Source Administrator window. Figure 2 The Username is DBA and the Password is sql (case sensitive) for the demo database, demo.db. Please refer to the article, "Migrating from Oracle 10G XE to SQL Anywhere 10" which describes connecting to the demo database in detail. Figure 3 shows the demo database and its objects. Figure 3 VB.NET Windows Application We will create an ASP.NET 2.0 Windows application called SqlAny. We will create forms which display retrieved data from a table on the database as well as from a stored procedure after accepting a parameter passed to the stored procedure interactively. The Figure 4 shows the details of the project in the Solution Explorer as well as the Object Browser. Figure 4 Accessing SQL Anywhere Explorer SQL Anywhere Explorer is a component of SQL Anywhere that lets you connect to SQL Anywhere and UltraLite  databases from Visual Studio .NET. From the View menu of Visual Studio, you can access the SQL Anywhere Explorer as shown in Figure 5 - SQL Anywhere 10 is integrated with Visual Studio (both 1.1 and 2.0 versions). Figure 5   Alternatively, you can access SQL Anywhere Explorer from the Tools menu item as shown in Figure 6. In this case the Sybase Central management interface opens in a separate window. Interactive SQL is another of SQL Anywhere 10's tools for working with SQL queries on this database. Figure 6   When you click on SQL Anywhere Explorer from the View menu, you will be lead to the following window shown in Figure 7 which allows you to establish a data connection. Figure 7 Click on the drop-down, Add Connection, which opens the window shown in Figure 8 where you will be given a choice of two connections that you may connect to, SQL Anywhere or UltraLite. These are both databases. Both can run on mobile devices, but UltraLite has a smaller footprint. Figure 8 By choosing to connect to SQL Anywhere you invoke the authentication window for making the connection, as shown in Figure 9. The Username is DBA and the Password is sql. After entering these values you can get to the ODBC DSN mentioned earlier, from the drop-down. You may also test the connectivity which you see as being a success, for the entered values of Username, Password, and ODBC DSN. Figure 9   Visual Studio makes a data connection as shown in Figure 10. The nodes for Tables, Views, and Procedures are all expanded in this figure showing all the objects that can be accessed on this database. Since we logged in as DBA, all permissions are in place. Figure 10 Before the connection is made, SQL Anywhere starts up as shown in Figure 11. This message console gets minimized and stays up in the system tray of the desktop. This can be restored and closed by activating the icon in the tray.   Figure 11
Read more
  • 0
  • 0
  • 8913

article-image-webcam-and-video-wizardry
Packt
26 Jul 2013
13 min read
Save for later

Webcam and Video Wizardry

Packt
26 Jul 2013
13 min read
(For more resources related to this topic, see here.) Setting up your camera Go ahead, plug in your webcam and boot up the Pi; we'll take a closer look at what makes it tick. If you experimented with the dwc_otg.speed parameter to improve the audio quality during the previous article, you should change it back now by changing its value from 1 to 0, as chances are that your webcam will perform worse or will not perform at all, because of the reduced speed of the USB ports.. Meet the USB Video Class drivers and Video4Linux Just as the Advanced Linux Sound Architecture (ALSA) system provides kernel drivers and a programming framework for your audio gadgets, there are two important components involved in getting your webcam to work under Linux: The Linux USB Video Class (UVC) drivers provide the low-level functions for your webcam, which are in accordance with a specifcation followed by most webcams produced today. Video4Linux (V4L) is a video capture framework used by applications that record video from webcams, TV tuners, and other video-producing devices. There's an updated version of V4L called V4L2, which we'll want to use whenever possible. Let's see what we can find out about the detection of your webcam, using the following command: pi@raspberrypi ~ $ dmesg The dmesg command is used to get a list of all the kernel information messages that of messages, is a notice from uvcvideo. Kernel messages indicating a found webcam In the previous screenshot, a Logitech C110 webcam was detected and registered with the uvcvideo module. Note the cryptic sequence of characters, 046d:0829, next to the model name. This is the device ID of the webcam, and can be a big help if you need to search for information related to your specifc model. Finding out your webcam's capabilities Before we start grabbing videos with our webcam, it's very important that we find out exactly what it is capable of in terms of video formats and resolutions. To help us with this, we'll add the uvcdynctrl utility to our arsenal, using the following command: pi@raspberrypi ~ $ sudo apt-get install uvcdynctrl Let's start with the most important part—the list of supported frame formats. To see this list, type in the following command: pi@raspberrypi ~ $ uvcdynctrl -f List of frame formats supported by our webcam According to the output of this particular webcam, there are two main pixel formats that are supported. The first format, called YUYV or YUV 4:2:2, is a raw, uncompressed video format; while the second format, called MJPG or MJPEG, provides a video stream of compressed JPEG images. Below each pixel format, we find the supported frame sizes and frame rates for each size. The frame size, or image resolution, will determine the amount of detail visible in the video. Three common resolutions for webcams are 320 x 240, 640 x 480 (also called VGA), and 1024 x 768 (also called XGA). The frame rate is measured in Frames Per Second (FPS) and will determine how "fuid" the video will appear. Only two different frame rates, 15 and 30 FPS, are available for each frame size on this particular webcam. Now that you know a bit more about your webcam, if you happen to be the unlucky owner of a camera that doesn't support the MJPEG pixel format, you can still go along, but don't expect more than a slideshow of images of 320 x 240 from your webcam. Video processing is one of the most CPU-intensive activities you can do with the Pi, so you need your webcam to help in this matter by compressing the frames first. Capturing your target on film All right, let's see what your sneaky glass eye can do! We'll be using an excellent piece of software called MJPG-streamer for all our webcam capturing needs. Unfortunately, it's not available as an easy-to-install package for Raspbian, so we will have to download and build this software ourselves. Often when we compile software from source code, the application we're building will want to make use of code libraries and development headers. Our MJPG-streamer application, for example, would like to include functionality for dealing with JPEG images and Video4Linux devices. Install the libraries and headers for JPEG and V4L by typing in the following command: pi@raspberrypi ~ $ sudo apt-get install libjpeg8-dev libv4l-dev Next, we're going to download the MJPG-streamer source code using the following command: pi@raspberrypi ~ $ wget http: // mjpg-streamer.svn.sourceforge.net/viewvc/mjpg-streamer/mjpg-streamer/?view=tar -O mjpg-streamer.tar.gz The wget utility is an extraordinarily handy web download tool with many uses. Here we use it to grab a compressed TAR archive from a source code repository, and we supply the extra -O mjpg-streamer.tar.gz to give the downloaded tarball a proper filename. Now we need to extract our mjpg-streamer.tar.gz article, using the following command: pi@raspberrypi ~ $ tar xvf mjpg-streamer.tar.gz The tar command can both create and extract archives, so we supply three fags here: x for extract, v for verbose (so that we can see where the files are being extracted to), and f to tell tar to use the article we specify as input, instead of reading from the standard input. Once you've extracted it, enter the directory containing the sources: pi@raspberrypi ~ $ cd mjpg-streamer Now type in the following command to build MJPG-streamer with support for V4L2 devices: pi@raspberrypi ~/mjpg-streamer $ make USE_LIBV4L2=true Once the build process has finished, we need to install the resulting binaries and other application data somewhere more permanent, using the following command: pi@raspberrypi ~/mjpg-streamer $ sudo make DESTDIR=/usr install You can now exit the directory containing the sources and delete it, as we won't need it anymore: pi@raspberrypi ~/mjpg-streamer $ cd .. && rm -r mjpg-streamer Let's fre up our newly-built MJPG-streamer! Type in the following command, but adjust the values for resolution and frame rate to a moderate setting that you know (from the previous section) that your webcam will be able to handle: pi@raspberrypi ~ $ mjpg_streamer -i "input_uvc.so -r 640x480 -f 30" -o "output_http.so -w /usr/www" MJPG-streamer starting up You may have received a few error messages saying Inappropriate ioctl for device; these can be safely ignored. Other than that, you might have noticed the LED on your webcam (if it has one) light up as MJPG-streamer is now serving your webcam feed over the HTTP protocol on port 8080. Press Ctrl + C at any time to quit MJPG-streamer. To tune into the feed, open up a web browser (preferably Chrome or Firefox) on a computer connected to the same network as the Pi and enter the following line into the address field of your browser, but change [IP address] to the IP address of your Pi. That is, the address in your browser should look like this: http://[IP address]:8080. You should now be looking at the MJPG-streamer demo pages, containing a snapshot from your webcam. MJPG-streamer demo pages in Chrome The following pages demonstrate the different methods of obtaining image data from your webcam: The Static page shows the simplest way of obtaining a single snapshot frame from your webcam. The examples use the URL http://[IP address]:8080/?action=snapshot to grab a single frame. Just refresh your browser window to obtain a new snapshot. You could easily embed this image into your website or blog by using the <img src = "http://[IP address]:8080/?action=snapshot"/> HTML tag, but you'd have to make the IP address of your Pi reachable on the Internet for anyone outside your local network to see it. The Stream page shows the best way of obtaining a video stream from your webcam. This technique relies on your browser's native support for decoding MJPEG streams and should work fne in most browsers except for Internet Explorer. The direct URL for the stream is http://[IP address]:8080/?action=stream. The Java page tries to load a Java applet called Cambozola, which can be used as a stream viewer. If you haven't got the Java browser plugin already installed, you'll probably want to steer clear of this page. While the Cambozola viewer certainly has some neat features, the security risks associated with the plugin outweigh the benefits of the viewer. The JavaScript page demonstrates an alternative way of displaying a video stream in your browser. This method also works in Internet Explorer. It relies on JavaScript code to continuously fetch new snapshot frames from the webcam, in a loop. Note that this technique puts more strain on your browser than the preferred native stream method. You can study the JavaScript code by viewing the page source of the following page: http://[IP address]:8080/javascript_simple.html The VideoLAN page contains shortcuts and instructions to open up the webcam video stream in the VLC media player. We will get to know VLC quite well during this article; leave it alone for now. The Control page provides a convenient interface for tweaking the picture settings of your webcam. The page should pop up in its own browser window so that you can view the webcam stream live, side-by-side, as you change the controls. Viewing your webcam in VLC media player You might be perfectly content with your current webcam setup and viewing the stream in your browser; for those of you who prefer to watch all videos inside your favorite media player, this section is for you. Also note that we'll be using VLC for other purposes further in this article, so we'll go through the installation here. Viewing in Windows Let's install VLC and open up the webcam stream: Visit http://www.videolan.org/vlc/download-windows.html and download the latest version of the VLC installer package(vlc-2.0.5-win32.exe, at the time of writing). Install VLC media player using the installer. Launch VLC using the shortcut on the desktop or from the Start menu. From the Media drop-down menu, select Open Network Stream…. Enter the direct stream URL we learned from the MJPG-streamer demo pages (http://[IP address]:8080/?action=stream), and click on the Play button. (Optional) You can add live audio monitoring from the webcam by opening up a command prompt window and typing in the following command: "C:Program Files (x86)PuTTYplink" pi@[IP address] -pw [password] sox -t alsa plughw:1 -t sox - | "C:Program Files (x86)sox-14-4-1sox" -q -t sox - -d Viewing in Mac OS X Let's install VLC and open up the webcam stream: Visit http://www.videolan.org/vlc/download-macosx.html and download the latest version of the VLC dmg package for your Mac model. The one at the top, vlc-2.0.5.dmg (at the time of writing), should be fne for most Macs. Double-click on the VLC disk image and drag the VLC icon to the Applications folder. Launch VLC from the Applications folder. From the File drop-down menu, select Open Network…. Enter the direct stream URL we learned from the MJPG-streamer demo pages (http://[IP address]:8080/?action=stream) and click on the Open button. (Optional) You can add live audio monitoring from the webcam by opening up a Terminal window (located in /Applications/Utilities ) and typing in the following command: ssh pi@[IP address] sox -t alsa plughw:1 -t sox - | sox -q -t sox - -d Viewing on Linux Let's install VLC or MPlayer and open up the webcam stream: Use your distribution's package manager to add the vlc or mplayer package. For VLC, either use the GUI to Open a Network Stream or launch it from the command line with vlc http://[IP address]:8080/?action=stream For MPlayer, you need to tag on an MJPG article extension to the stream, using the following command: mplayer "http://[IP address]:8080/?action= stream&stream.mjpg" (Optional) You can add live audio monitoring from the webcam by opening up a Terminal and typing in the following command: ssh pi@[IP address] sox -t alsa plughw:1 -t sox - | sox -q -t sox - -d Recording the video stream The best way to save a video clip from the stream is to record it with VLC, and save it into an AVI article container. With this method, we get to keep the MJPEG compression while retaining the frame rate information. Unfortunately, you won't be able to record the webcam video with sound. There's no way to automatically synchronize audio with the MJPEG stream. The only way to produce a video article with sound would be to grab video and audio streams separately and edit them together manually in a video editing application such as VirtualDub. Recording in Windows We're going to launch VLC from the command line to record our video: Open up a command prompt window from the Start menu by clicking on the shortcut or by typing in cmd in the Run or Search fields. Then type in the following command to start recording the video stream to a article called myvideo.avi, located on the desktop: C:> "C:Program Files (x86)VideoLANVLCvlc.exe" http://[IP address]:8080/?action=stream --sout="#standard{mux=avi,dst=%UserProfile%Desktopmyvideo.avi,access=file}" As we've mentioned before, if your particular Windows version doesn't have a C:Program Files (x86) folder, just erase the (x86) part from the path, on the command line. It may seem like nothing much is happening, but there should now be a growing myvideo.avi recording on your desktop. To confirm that VLC is indeed recording, we can select Media Information from the Tools drop-down menu and then select the Statistics tab. Simply close VLC to stop the recording. Recording in Mac OS X We're going to launch VLC from the command line, to record our video: Open up a Terminal window (located in /Applications/Utilities) and type in the following command to start recording the video stream to a file called myvideo.avi, located on the desktop: $ /Applications/VLC.app/Contents/MacOS/VLC http://[IP address]:8080/?action=stream --sout='#standard{mux=avi,dst=/Users/[username]/Desktop/myvideo.avi,access=file}' Replace [username] with the name of the account you used to log in to your Mac, or remove the directory path to write the video to the current directory. It may seem like nothing much is happening, but there should now be a growing myvideo.avi recording on your desktop. To confirm that VLC is indeed recording, we can select Media Information from the Window drop-down menu and then select the Statistics tab. Simply close VLC to stop the recording. Recording in Linux We're going to launch VLC from the command line to record our video: Open up a Terminal window and type in the following command to start recording the video stream to a file called myvideo.avi, located on the desktop: $ vlc http://[IP address]:8080/?action=stream --sout='#standard{mux=avi,dst=/home/[username]/Desktop/myvideo.avi,access=file}' Replace [username] with your login name, or remove the directory path to write the video to the current directory.
Read more
  • 0
  • 1
  • 8904

article-image-webrtc-sip-and-ims
Packt
29 Oct 2014
28 min read
Save for later

WebRTC with SIP and IMS

Packt
29 Oct 2014
28 min read
In this article by Altanai Bisht, the author of the book, WebRTC Integrator's Guide, has discussed about the interaction of WebRTC client with important IMS nodes and modules. IP Multimedia Subsystem (IMS) is an architectural framework for IP Multimedia communications and IP telephony based on Convergent applications. It specifies three layers in a telecom network: Transport or Access layer: This is the bottom-most segment responsible for interacting with end systems such as phones. IMS layer: This is the middleware responsible for authenticating and routing the traffic and facilitating call control through the Service layer. Service or Application layer: This is the top-most layer where all of the call control applications and Value Added Services (VAS) are hosted. (For more resources related to this topic, see here.) IMS standards are defined by Third Generation Partnership Project (3GPP) which adopt and promote Internet Engineering Task Force (IETF) Request for Comments (RFCs). Refer to http://www.3gpp.org/technologies/keywords-acronyms/109-ims to learn more about 3GPP IMS specification releases. This article will walk us through the interaction of WebRTC client with important IMS nodes and modules. The WebRTC gateway is the first point of contact for the SIP requests from the WebRTC client to enter into the IMS network. The WebRTC gateway converts SIP over WebSocket implementation to legacy/plain SIP, that is, a WebRTC to SIP gateway that connects to the IMS world and is able to communicate with a legacy SIP environment. It also can translate other REST- or JSON-based signaling protocols into SIP. The gateway also handles the media operation that involves DTLS, SRTP, RTP, transcoding, demuxing, and so on. In this article, we will study a case where there exists a simple IMS core environment, and the WebRTC clients are meant to interact after the signals are traversed through core IMS nodes such as Call Session Control Function (CSCF), Home Subscriber Server (HSS), and Telecom Application Server (TAS). The Interaction with core IMS nodes This section describes the sequence of steps that must be followed for the integration of the WebRTC client with IMS. Before you go ahead, set up a Session Border Controller (SBC) / WebRTC gateway / SIP proxy node for the WebRTC client to interact with the IMS control layer. Direct the control towards the CSCF nodes of IMS, namely, Proxy-CSCF, Interrogating-CSCF, and Serving-CSCF. The subscriber details and the location are updated in the HSS. Serving-CSCF (SCSCF) routes the call through the SIP Application Server to invoke any services before the call is processed. The Application Server, which is part of the IMS service layer, is the point of adding logic to call processing in the form of VAS. Additionally, we will uncover the process of integrating media server for an inter-codec conversion between legacy SIP phones and WebRTC clients. The setup will allow us to support all SIP nodes and endpoints as part of the IMS land-scape. The following figure shows the placement of the SIPWS to SIP gateway in the IMS network: The WebRTC client is a web-based dynamic application that is run over a Web Application Server. For simplification, we can club the components of the WebRTC client and the Web Application Server together and address them jointly as the WebRTC client, as shown in the following diagram: There are four major components of the OpenIMS core involved in this setup as described in the following sections. Along with these, two components of the WebRTC infrastructure (the client and the gateway) are also necessary to connect the WebRTC endpoints. Three optional entities are also described as part of this setup. The components of Open IMS are CSCF nodes and HSS. More information on each component is given in the following sections. The Call Session Control Function The three parts of CSCF are described as follows: Proxy-CSCF (P-CSCF) is the first point of contact for a user agent (UA) to which all user equipments (UEs) are attached. It is responsible for routing an incoming SIP request to other IMS nodes, such as registrar and Policy and Charging Rules Function (PCRF), among others. Interrogating-CSCF (I-CSCF) is the inbound SIP proxy server for querying the HSS as to which S-CSCF should be serving the incoming request. Serving-CSCF (S-CFCS) is the heart of the IMS core as it enables centralized IMS service control by defining routing paths that act like the registrar, interact with the Media Server, and much more. Home Subscriber System IMS core Home Subscriber System (HSS) is the database component responsible for maintaining user profiles, subscriptions, and location information. The data is used in functions such as authentication and authorization of users while using IM services. The components of the WebRTC infrastructure primarily comprises of WebRTC Web Application Servers, WebRTC web-based clients, and the SIP gateway. WebRTC Web Application Server and client: The WebRTC client is intrinsically a web application that is composed of user interfaces, data access objects, and controllers to handle HTTP requests. A Web Application Server is where an application is hosted. As WebRTC is a browser-based technique, it is meant to be an HTML-based web application. The call functionalities are rendered through the SIP JavaScript files. The browser's native WebRTC capabilities are utilized to capture and transmit the data. A WebRTC service provider must embed the SIP call functions on a web page that has a call interface. It must provide values for the To and From SIP addresses, div to play audio/video content, and access to users' resources such as camera, mic, and speakers. WebRTC to IMS gateway: This is the point where the conversion of the signal from SIP over WebSockets to legacy/plain SIP takes place. It renders the signaling into a state that the IMS network nodes can understand. For media, it performs the transcoding from WebRTC standard codecs to others. It also performs decryption and demux of audio/video/RTCP/RTP. There are other servers that act as IMS nodes as well, such as the STUN/TURN Server, Media Server, and Application Server. They are described as follows: STUN/TURN Server: These are employed for NAT traversals and overcoming firewall restrictions through ICE candidates. They might not be needed when the WebRTC client is on the Internet and the WebRTC gateway is also listening on a publicly accessible IP. Media Server: Media server plays a role when media relay is required between the UEs instead of a direct peer-to-peer communication. It also comes into picture for services such as voicemail, Interactive Voice Response (IVR), playback, and recording. Application Server (AS): Application Server is the point where developers can make customized logic for call control such as VAS in the form of call redirecting in cases when the receiver is absent and selective call screening. The IP Multimedia Subsystem core IMS is an architecture for real-time multimedia (voice, data, video, and messaging) services using a common IP network. It defines a layered architecture. According to the 3GPP specification, IMS entities are classified into six categories: Session management and route (CSCF, GGSN, and SGSN) Database (HSS and SLF) Interworking elements (BGCF, MGCF, IM-MGW, and SGW) Service (Application Server, MRFC and MRFP) Strategy support entities (PDF) Billing Interoperability with the SIP infrastructure requires a session border controller to decrypt the WebRTC control and media flows. A media node is also set up for transcoding between WebRTC codecs and other legacy phones. When a gateway is involved, the WebRTC voice and video peer connections are between the browser and the border controller. In our case, we have been using Kamailio in this role. Kamailio is an open source SIP server capable of processing both SIP and SIPWS signaling. As WebRTC is made to function over SIP-based signaling, it is applicable to enjoy all of the services and solutions made for the IMS environment. The telecom operators can directly mount the services in the Service layer, and subscribers can avail the services right from their web browsers through the WebRTC client. This adds a new dimension to user accessibility and experience. A WebRTC client's true potential will come into effect only when it is integrated with the IMS framework. We have some readymade, open IMS setups that have been tested for WebRTC-to-IMS integration. The setups are as follows: 3GPP IMS: This is the IMS specification by 3GPP, which is an association of telecommunications group OpenIMS: This is the open source implementation of the IMS CSCFs and a lightweight HSS for the IMS core DubangoIMS: This is the cross-platform and open source 3GPP IMS/LTE framework KamailioIMS: Kamailio Version 4.0 and above incorporates IMS support by means of OpenIMS We can also use any other IMS structure for the integration. In this article, we will demonstrate the use of OpenIMS. For this, it is required that a WebRTC client and a non-WebRTC client must be interoperable by means of signaling and media transcoding. Also, the essential components of IMS world, such as HSS, Media Server, and Application Server, should be integrated with the WebRTC setup. The OpenIMS Core The Open IMS Core is an open source implementation for core elements of the IMS network that includes IMS CSCFs nodes and HSS. The following diagram shows how a connection is made from WebRTC to CSCF: The following are the prerequisites to install the Open IMS core: Make sure that you have the following packages installed on your Linux machine, as their absence can hinder the IMS installation process: Git and Subversion GCC3/4, Make, JDK1.5, Ant MySQL as the database Bison and Flex, the Linux utilities libxml2 (Version 2.6 and above) and libmysql with development versions Install these packages from the Synaptic package manager or using the command prompt. For the LoST interface of E-CSCF, use the following command lines: sudo apt-get install mysql-server libmysqlclient15-dev libxml2libxml2-dev bind9 ant flex bison curl libcurl4-gnutls-dev sudo apt-get install curl libcurl4-gnutls-dev The Domain Name Server (DNS), bind9, should be installed and run. To do this, we can run the following command line: sudo apt-get install bind9 We need a web browser to review the status of the connection on the web console. To download a web browser, go to its download page. For example, Chrome can be downloaded from https://www.google.com/intl/en_in/chrome/browser/. We must verify that the Java version installed is above 1.5 so as to not break the compilation process in between, and set the path of JAVA_HOME as follows: export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre The output of the command line that checks the Java version is as follows: The following are the steps to install OpenIMS. As the source code is preconfigured to work from a standard file path of /opt, we will use the predefined directory for installation. Go to the /opt folder and create a directory to store the OpenIMS core, using the following command lines: mkdir /opt/OpenIMSCorecd /opt/OpenIMSCore Create a directory to store FHOSS, check out the HSS, and compile the source using the following command lines: mkdir FHoSS svn checkout http://svn.berlios.de/svnroot/repos/openimscore/FHoSS/trunk FHoSS cd FHoSS ant compile deploy Note that the code requires Java Version 7 or lower to work. Also, create a directory to store ser_ims, check out the CFCs, and then install ser_ims using the following command lines: mkdir ser_ims svn checkout http://svn.berlios.de/svnroot/repos/openimscore/ser_ims/trunk ser_ims cd ser_ims make install-libs all After downloading and installing the OpenIMS installation directory, its contents are as follows: By default, the nodes are configured to work only on the local loopback, and the default domain configured is open-ims.test. The MySQL access rights are also set only for local access. However, this can be modified using the following steps: Run the following command line: ./opt/ser_ims/cfg/configurator.sh Replace 127.0.0.1 (the default IP for the local host) with the new IP address that is required to configure the IMS Core server. Replace the home domain (open-ims.test) with the required domain name. Change the database passwords. The following figure depicts the domain change process through configurator.sh: To resolve the domain name, we need to add a new IMS domain to bind the configuration directory. Change to the system's bind folder (cd /etc/bind) and copy the open-ims.dnszone file there after replacing the domain name. sudo cp /opt/OpenIMSCore/ser_ims/cfg/open-ims.dnszone /etc/bind/ Open the name.conf file and include open-ims.dnszone in the list that already exists: include "/etc/bind/named.conf.options"; include "/etc/bind/named.conf.local"; include "/etc/bind/named.conf.default-zones"; include "/etc/bind/open-ims.dnszone"; One can also add a reverse zone file, which, contrary to the DNS zone file, converts an address to a name. Restart the naming server using the following command: sudo bind9 restart On occasion of any failure or error note, the system logs/reports can be generated using the following command line: tail -f /var/log/syslog Open the MySQL client (sudo mysql) and add the SQL scripts for the creation of database and tables for HSS operations: mysql -u root -p -h localhost<ser_ims/cfg/icscf.sql mysql -u root -p -h localhost<FHoSS/scripts/hss_db.sql mysql -u root -p -h localhost<FHoSS/scripts/userdata.sql The following screenshot shows the tables for the HSS database: Users should be registered with a domain (that is, one needs to make changes in the userdata.sql file by replacing the default domain name with the required domain name). Note that while it is not mandatory to change the domain, it is a good practice to add a new domain that describes the enterprise or service provider's name. The following screenshot shows user domains changed from the default to the personal domain: Copy the pcscf.cfg, pcscf.sh, icscf.cfg, icscf.xml, icscf.sh, scscf.cfg, scscf.xml, and scscf.sh files to the /opt/OpenIMSCore location. Start the Policy Call Session Control Function (PCSCF) by executing the pcscf.sh script. The default element port assigned for P-CSCF is 4060. A screenshot of the running of PCSCF is as follows: Start the Interrogating Call Session Control Function (I-CSCF) by executing the icscf.sh script. The default element port assigned to I-CSCF is 5060. If the scripts display a warning about connection, it is just because the FHoSS client still needs to be started. A screenshot of the running I-CSCF is as follows: Start SCSCF by executing the scscf.sh script. The default element port assignment for S-CSCF is 6060. A screenshot of the running SCSCF is as follows: Start the FOKUS Home Subscriber Server (FHoSS) by executing FHoss/deploy/startup.sh. The HSS interacts using the diameter protocol. The ports used for this protocol are 3868, 3869, and 3870. A screenshot of the running HSS is shown as follows: Go to http://<yourip>:8080 and log in to the web console with hssAdmin as the username and hss as the password as shown in the following screenshot. To register the WebRTC client with OpenIMS, we must use an IMS gateway that performs the function of converting the SIP over WebSocket format to SIP. In order to achieve this, use the IP port or domain of the PCSCF node while registering the client. The flow will be from the WebRTC client to the IMS gateway to the PCSCF of the IMS Core. The flow can also be from the SIPML5 WebRTC client to the webrtc2sip gateway to the PCSCF of the OpenIMS Core. The subscribers are visible in the IMS subscription section of the portal of OpenIMS. The following screenshot shows the user identities and their statuses on a web-based admin console: As far as other components are concerned, they can be subsequently added to the core network over their respective interfaces. The Telecom server The TAS is where the logic for processing a call resides. It can be used to add applications such as call blocking, call forwarding, and call redirection according to the predefined values. The inputs can be assigned at runtime or stored in a database using a suitable provisioning system. The following diagram shows the connection between WebRTC and the IMS Core Server: For demonstration purposes, we can use an Application Server that can host SIP servlets and integrate them with IMS core. The Mobicents Telecom Application Server Mobicents SIP Servlet and Java APIs for Integrated Networks-Service Logic Execution Environment (JAIN-SLEE) are open platforms to deploy new call controller logic and other converged applications. The steps to install Mobicents TAS are as follows: Download the SIP Application Server logic package from https://code.google.com/p/sipservlets/wiki/Downloads. Unzip the contents. Make sure that the Java environment variables are in place. Start the JBoss container from mobicentsjboss-5.1.0.GAbin In case of MS Windows, click on run.bat, and for Linux, click on run.sh. The following figure displays the traces on the console when the server is started on JBoss: The Mobicents application can also be developed by installing the Tomcat/Mobicents plugin in Eclipse IDE. The server can also be added for Mobicents instance, enabling quick deployment of applications. Open the web console to review the settings. The following screenshot displays the process: In order to deploy Resource Adaptors, enter: ant -f resources/<name of resource adapter>/build.xml deploy To undeploy the resource adapters, execute antundeploy with the name of the resource adapter: ant -f resources/<name of resource adapter>/build.xml undeploy Make sure that you have Apache Ant 1.7. The deployed instances should be visible in a web console as follows: To deploy and run SIP Servlet applications, use the following command line: ant -f examples/<name of application directory>/build.xml deploy-all Configure CSCF to include the Application Server in the path of every incoming SIP request and response. With the introduction of TAS, it is now possible to provide customized call control logic to all subscribers or particular subscribers. The SIP solution and services can range from simple activities, such as call screening and call rerouting, to a complex call-handling application, such as selective call screening based on the user's calendar. Some more examples of SIP applications are given as follows: Speed Dial: This application lets the user make a call using pre-programmed numbers that map to actual SIPURIs of users. Click to Dial: This application makes a call using a web-based GUI. However, it is very different from WebRTC, as it makes/receives the call through an external SIP phone. Find me Follow Me: This application is beneficial if the user is registered on multiple devices simultaneously, for example, SIP phone, X-Lite, and WebRTC. In such a case, when there is an incoming call, each of the user's devices rings for few seconds in order of their recent use so that the user can pick the call from the device that is nearest to him. These services are often referred to as VAS, which can be innovative and can take the user experience to new heights. The Media Server To enable various features such as Interactive Voice Respondent (IVR), record voice mails, and play announcements, the Media Server plays a critical role. The Media Server can be used as a standalone entity in the WebRTC infrastructure or it can be referenced from the SIP server in the IMS environment. The FreeSWITCH Media Server FreeSWITCH has powerful Media Server capabilities, including those for functions such as IVR, conferencing, and voice mails. We will first see how to use FreeSWITCH as a standalone entity that provides SIP and RTP proxy features. Let's try to configure and install a basic setup of FreeSWITCH Media Server using the following steps: Download and store the source code for compilation in the /usr/src folder, and run the following command lines: cd usr/src git clone -b v1.4 https://stash.freeswitch.org/scm/fs/freeswitch.git A directory named freeswitch is made using the following command line and binaries will be stored in this folder. Assign all permissions to it. sudo chown -R <username> /usr/local/freeswitch Replace <username> with the name of the user who has the ownership of the folder. Go to the directory where the source will be stored, that is, the following directory: cd /usr/src/freeswitch Then, run bootstrap using the following command line: ./bootstrap.sh One can add additional modules by editing the configuration file using the vi editor. We can open our file using the following command line: vi modules.conf The names of the module are already listed. Remove the # symbol before the name to include the module at runtime, and add # to skip the module. Then, run the configure command: ./configure --enable-core-pgsql-support Use the make command and install the components: make && make install Go to the Sofia profile and uncomment the parameters defined for WebSocket binding. By doing so, the WebRTC clients can register with FreeSWITCH on port 443. Sofia is an SIP stack used by FreeSWITCH. By default, it supports only pure SIP requests. To get WebRTC clients, register with FreeSWITCH's SIP Server. <!-- uncomment for SIP over WebSocket support --><!-- <param name="ws-binding" value=":443"/> Install the sound files using the following command line: make all cd-sounds-install cd-moh-install Go to the installation directory, and in the vars.xml file under freeswitch/conf/ make sure that the codec preferences are set as follows: <X-PRE-PROCESS cmd="set" data="global_codec_prefs=G722,PCMA,PCMU,GSM"/> <X-PRE-PROCESS cmd="set" data="outbound_codec_prefs=G722,PCMA,PCMU,GSM"/> Make sure that the SIP profile is directly using the codec values as follows: <param name="inbound-codec-prefs" value="$${global_codec_prefs}"/> <param name="outbound-codec-prefs" value="$${global_codec_prefs}"/> We can later add more codecs such as vp8 for video calling/conferencing. To start FreeSWITCH, go to the /freeswitch/bin installation directory and run FreeSWITCH. Run the command-line console that will be used to control and monitor the passing SIP packets by going to the /freeswitch/bin installation directory and executing fs_cli. The following is the screenshot of the FreeSWITCH client console: Go to the /freeswitch/conf/SIP_profile installation-directory and look for the existing configuration files. Load and start the SIP profile using the following command line: sofia profile <name of profile> start load Restart and reload the profile in case of changes using the following command line: sofia profile <name of profile>restart reload Check its working by executing the following command line: Sofia status We can check the status of the individual SIP profile by executing the following command line: sofia status profile <name of profile> reg The preceding figure depicts the status of the users registered with the server at one point of time. Media Services The following steps outline the process of using the FreeSWITCH media services: Register the SIP softphone and WebRTC client using FreeSWITCH. Use sample values between 1000 and 1020 initially. Later, we can configure for more users as specified by the /freeswitch/conf/directory installation directory. The following are the sample values to register Kapanga:      Username: 1002      Display name: any      Domain/ Realm: 14.67.87.45      Outbound proxy: 14.67.87.45:5080      Authorization user: 1002      Password: 1234 The sample value for WebRTC client registration, if, for example, we decide to use the Sipml5webrtc client, for example, will be as follows:      Display name: any      Private identity: 1001      Public identity: SIP:1001@14.67.87.45      Password: 1234      Realm: 14.67.87.45      WebSocket Server URL: ws://14.67.87.45:443 Note that the values used here are arbitrary for the purpose of understanding. IP denotes the public IP of the FreeSWITCH machine and the port is the WebSocket configured port in the Sofia profile. As seen in the following screenshot, it is required that we tick the Enable RTCWeb Breaker option in Expert settings to compensate for the incompatibility between the WebSocket and SIP standards that might arise: Make a call between the SIP softphone and WebRTC client. In this case, the signal and media are passing through FreeSWITCH as proxy. Call from a WebRTC client is depicted in the following screenshot, which consists of SIP messages passing through the FreeSWITCH server and are therefore visible in the FreeSWITCH client console. In this case, the server is operating in the default mode; other modes are bypass and proxy modes. Make a call between two WebRTC clients, where SIP and RTP are passing through FreeSWITCH as proxy. We can use other services of FreeSWITCH as well, such as voicemail, IVR, and conferencing. We can also configure this setup in such a way that media passes through the FreeSWITCH Media Server, and the SIP signaling is via the Telecom Kamailio SIP server. Use the RTP proxy in the SIP proxy server, in our case, Kamailio, to pass the RTP media through the Media Server. The RTP proxy module of Kamailio should be built in a format and configured in the kamailio.cfg file. The RTP proxy forces the RTP to pass through a node as specified in the settings parameters. It makes the communication between SIP user agents behind NAT and will also be used to set up a relaying host for RTP streams. Configure the RTP Engine as the media proxy agent for RTP. It will be used to force the WebRTC media through it and not in the old peer-to-peer fashion in which WebRTC is designed to operate. Perform the following steps to configure the RTP Engine: Go to the Kamailio installation directory and then to the RTPProxy module. Run the make command and install the proxy engine: cd rtpproxy ./configure && make Load the module and parameters in the kamailio.cfg file: listen=udp:<ip>:<port> .. loadmodule "rtpproxy.so" .. modparam("rtpproxy", "rtpproxy_sock",   "unix:/var/run/rtpproxy/rtpproxy.sock") Add rtpproxy_manage() for all of the requests and responses in the kamailio.cfg file. The example of rtpproxy_manage for INVITE is: if (is_method("INVITE")) { ... rtpproxy_manage(); ... }; Get the source code for the RTP Engine using git as follows: https://github.com/sipwise/rtpengine.git Go to the daemon folder in the installation directory and run the make command as follows: sudo make Start rtpengine in the default user space mode on the local machine: sudo ./rtpengine --ip=10.1.5.14 --listen-ng=12334 Check the status of rtpengine, which is running, using the following command: ps -ef|grep rtpengine Note that rtpengine must be installed on the same machine as the Kamailio SIP server. In case of the sipml5 client, after configuring the modules described in the preceding section and before making a call through the Media Server, the flow for the media will become one of the following:      In case of Voicemail/IVR, the flow is as follows:     WebRTC client to RTP proxy node to Media Server      In case of a call through media relay, the flow is as follows:     WebRTC client A to RTP proxy node to Media Server to RTP Proxy to WebRTC client B The following diagram shows the MediaProxy relay between WebRTC clients: The potential of media server lies in its media transcoding of various codecs. Different phones / call clients / softwares that support SIP as the signaling protocol do not necessarily support the same media codecs. In the situation where Media Server is absent and the codecs do not match between a caller and receiver, the attempt to make a call is abruptly terminated when the media exchange needs to take place, that is, after invite, success, response, and acknowledgement are sent. In the following figure, the setup to traverse media through the FreeSWITCH Media Server and signaling through the Kamailio SIP server is depicted: The role of the rtpproxyng engine is to enable media to pass via Media Server; this is shown in the following diagram: WebRTC over firewalls and proxies There are many complicated issues involved with the correct working of WebRTC across domains, NATS, geographies, and so on. It is important for now that the firewall of a system, or any kind of port-blocking policy, should be turned off to be able to make a successful audio-video WebRTC call across any two parties that are not on the same Local Area Network (LAN). For the user to not have to switch the firewall off, we need to configure the Simple Traversal of UDP through NAT (STUN) server or modify the Interactive Connectivity Establishment (ICE) parameter in the SDP exchanged. STUN helps in packet routing of devices behind a NAT firewall. STUN only helps in device discoverability by assigning publicly accessible addresses to devices within a private local network. Traversal Using Relay NAT (TURN) servers also serve to accomplish the task of inter-connecting the endpoints behind NAT. As the name suggests, TURN forces media to be proxied through the server. To learn more about ICE as a NAT-traversal mechanism, refer to the official document named RFC 5245. The ICE features are defined by sipML5 in the sipml.js file. It is added to SIP SDP during the initial phase of setting up the SIP stack. Snippets from the sipml.js file regarding ICE declaration are given as follows: var configuration = { ... websocket_proxy_url: 'ws://192.168.0.10:5060', outbound_proxy_url: 'udp://192.168.0.12:5060', ice_servers: [{ url: 'stun:stun.l.google.com:19302'}, {    url:'turn:user@numb.viagenie.ca', credential:'myPassword'}], ... }; Under the postInit function in the call.htm page add the following function: oConfigCall = { ... events_listener: { events: '*', listener: onSipEventSession },    SIP_caps: [      { name: '+g.oma.SIP-im' },      { name: '+SIP.ice' },      { name: 'language', value: '"en,fr"' }    ] }; Therefore, the WebRTC client is able to reach the client behind the firewall itself; however, the media displays unpredicted behavior. In the need to create our own STUN-TURN server, you can take the help of RFC 5766, or you can refer to open source implementations, such as the project at the following site: https://code.google.com/p/rfc5766-turn-server/ When setting the parameters for WebRTC, we can add our own STUN/TURN server. The following screenshot shows the inputs suitable for ICE Servers if you are using your own TURN/STUN server: If there are no firewall restrictions, for example, if the users are on the same network without any corporate proxies and port blocks, we can omit the ICE by entering empty brackets, [], in the ICE Servers option on the Expert settings page in the WebRTC client. The final architecture for the WebRTC-to-IMS integration At the end of this article, we have arrived at an architecture similar to the following diagram. The diagram depicts a basic WebRTC-to-IMS architecture. The diagram depicts the WebRTC client in the Transport Layer as it is the user end-point. The IMS entities (CSCF and HSS), WebRTC to IMS gateway, and Media Server nodes are placed on the Network Control Layer as they help in signal and media routing. The applications for call control are placed in the top-most Application Layer that processes the call control logic. This architecture serves to provide a basic IMS-based setup for SIP-based WebRTC client interaction. Summary In this article, we saw how to interconnect the WebRTC setup with the IMS infrastructure. It included interaction with CSCF nodes, namely PCSCF, ICSCF, and SCSCF, after building and installing them from their sources. Also, FreeSWITCH Media Server was discussed, and the steps to build and integrate it were practiced. The Application Server to embed call control logic is Kamailio. NAT traversal via STUN / TURN server was also discussed and its importance was highlighted. To deploy the WebRTC solution integrated with the IMS network, we must ensure that all of the required IMS nodes are consulted while making a call, the values are reflected in the HSS data store, and the incoming SIP request and responses are routed via call logic of the Application Server before connecting a call. Resources for Article: Further resources on this subject: Using the WebRTC Data API [Article] Implementing Stacks using JavaScript [Article] Applying WebRTC for Education and E-learning [Article]
Read more
  • 0
  • 0
  • 8899

article-image-configuring-and-deploying-hbase-tutorial
Natasha Mathur
02 Jul 2018
21 min read
Save for later

Configuring and deploying HBase [Tutorial]

Natasha Mathur
02 Jul 2018
21 min read
HBase is inspired by the Google big table architecture, and is fundamentally a non-relational, open source, and column-oriented distributed NoSQL. Written in Java, it is designed and developed by many engineers under the framework of Apache Software Foundation. Architecturally it sits on Apache Hadoop and runs by using Hadoop Distributed File System (HDFS) as its foundation. It is a column-oriented database, empowered by a fault-tolerant distributed file structure known as HDFS. In addition to this, it also provides very advanced features, such as auto sharding, load-balancing, in-memory caching, replication, compression, near real-time lookups, strong consistency (using multi-version). It uses the latest concepts of block cache and bloom filter to provide faster response to online/real-time request. It supports multiple clients running on heterogeneous platforms by providing user-friendly APIs. In this tutorial, we will discuss how to effectively set up mid and large size HBase cluster on top of Hadoop/HDFS framework. We will also help you set up HBase on a fully distributed cluster. For cluster setup, we will consider REH (RedHat Enterprise-6.2 Linux 64 bit); for the setup we will be using six nodes. This article is an excerpt taken from the book ‘HBase High Performance Cookbook’ written by Ruchir Choudhry. This book provides a solid understanding of the HBase basics. Let’s get started! Configuring and deploying Hbase Before we start HBase in fully distributed mode, we will be setting up first Hadoop-2.2.0 in a distributed mode, and then on top of Hadoop cluster we will set up HBase because HBase stores data in HDFS. Getting Ready The first step will be to create a directory at user/u/HBase B and download the tar file from the location given later. The location can be local, mount points or in cloud environments; it can be block storage: wget wget –b http://apache.mirrors.pair.com/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz This –b option will download the tar file as a background process. The output will be piped to wget-log. You can tail this log file using tail -200f wget-log. Untar it using the following commands: tar -xzvf hadoop-2.2.0.tar.gz This is used to untar the file in a folder hadoop-2.2.0 in your current diectory location. Once the untar process is done, for clarity it's recommended use two different folders one for NameNode and other for DataNode. I am assuming app is a user and app is a group on a Linux platform which has access to read/write/execute access to the locations, if not please create a user app and group app if you have sudo su - or root/admin access, in case you don't have please ask your administrator to create this user and group for you in all the nodes and directorates you will be accessing. To keep the NameNodeData and the DataNodeData for clarity let's create two folders by using the following command, inside /u/HBase B: Mkdir NameNodeData DataNodeData NameNodeData will have the data which is used by the name nodes and DataNodeData will have the data which will be used by the data nodes: ls –ltr will show the below results. drwxrwxr-x 2 app app  4096 Jun 19 22:22 NameNodeData drwxrwxr-x 2 app app  4096 Jun 19 22:22 DataNodeData -bash-4.1$ pwd /u/HBase B/hadoop-2.2.0 -bash-4.1$ ls -ltr total 60K drwxr-xr-x 2 app app 4.0K Mar 31 08:49 bin drwxrwxr-x 2 app app 4.0K Jun 19 22:22 DataNodeData drwxr-xr-x 3 app app 4.0K Mar 31 08:49 etc The steps in choosing Hadoop cluster are: Hardware details required for it Software required to do the setup OS required to do the setup Configuration steps HDFS core architecture is based on master/slave, where an HDFS cluster comprises of solo NameNode, which is essentially used as a master node, and owns the accountability for that orchestrating, handling the file system, namespace, and controling access to files by client. It performs this task by storing all the modifications to the underlying file system and propagates these changes as logs, appends to the native file system files, and edits. SecondaryNameNode is designed to merge the fsimage and the edits log files regularly and controls the size of edit logs to an acceptable limit. In a true cluster/distributed environment, it runs on a different machine. It works as a checkpoint in HDFS. We will require the following for the NameNode: Components Details Used for nodes/systems Operating System Redhat-6.2 Linux  x86_64 GNU/Linux, or other standard linux kernel. All the setup for Hadoop/HBase and other components used Hardware /CPUS 16 to 32 CPU cores NameNode/Secondary NameNode 2 quad-hex-/octo-core CPU DataNodes Hardware/RAM 128 to 256 GB, In special caes 128 GB to 512 GB RAM NameNode/Secondary NameNodes 128 GB -512 GB of RAM DataNodes Hardware/storage It's pivotal to have NameNode server on robust and reliable storage platform as it responsible for many key activities like edit-log journaling. As the importance of these machines are very high and the NameNodes plays a central role in orchestrating everything,thus RAID or any robust storage device is acceptable. NameNode/Secondary Namenodes 2 to 4 TB hard disk in a JBOD DataNodes RAID is nothing but a random access inexpensive drive or independent disk. There are many levels of RAID drives, but for master or a NameNode, RAID 1 will be enough. JBOD stands for Just a bunch of Disk. The design is to have multiple hard drives stacked over each other with no redundancy. The calling software needs to take care of the failure and redundancy. In essence, it works as a single logical volume: Before we start for the cluster setup, a quick recap of the Hadoop setup is essential with brief descriptions. How to do it Let's create a directory where you will have all the software components to be downloaded: For the simplicity, let's take it as /u/HBase B. Create different users for different purposes. The format will be as follows user/group, this is essentially required to differentiate different roles for specific purposes: Hdfs/hadoop is for handling Hadoop-related setup Yarn/hadoop is for yarn related setup HBase /hadoop Pig/hadoop Hive/hadoop Zookeeper/hadoop Hcat/hadoop Set up directories for Hadoop cluster. Let's assume /u as a shared mount point. We can create specific directories that will be used for specific purposes. Please make sure that you have adequate privileges on the folder to add, edit, and execute commands. Also, you must set up password less communication between different machines like from name node to the data node and from HBase master to all the region server nodes. Once the earlier-mentioned structure is created; we can download the tar files from the following locations: -bash-4.1$ ls -ltr total 32 drwxr-xr-x  9 app app 4096 hadoop-2.2.0 drwxr-xr-x 10 app app 4096 zookeeper-3.4.6 drwxr-xr-x 15 app app 4096 pig-0.12.1 drwxrwxr-x  7 app app 4096 HBase -0.98.3-hadoop2 drwxrwxr-x  8 app app 4096 apache-hive-0.13.1-bin drwxrwxr-x  7 app app 4096 Jun 30 01:04 mahout-distribution-0.9 You can download these tar files from the following location: wget –o https://archive.apache.org/dist/HBase /HBase -0.98.3/HBase -0.98.3-hadoop1-bin.tar.gz wget -o https://www.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz wget –o https://archive.apache.org/dist/mahout/0.9/mahout-distribution-0.9.tar.gz wget –o https://archive.apache.org/dist/hive/hive-0.13.1/apache-hive-0.13.1-bin.tar.gz wget -o https://archive.apache.org/dist/pig/pig-0.12.1/pig-0.12.1.tar.gz Here, we will list the procedure to achieve the end result of the recipe. This section will follow a numbered bullet form. We do not need to give the reason that we are following a procedure. Numbered single sentences would do fine. Let's assume that there is a /u directory and you have downloaded the entire stack of software from: /u/HBase B/hadoop-2.2.0/etc/hadoop/ and look for the file core-site.xml. Place the following lines in this configuration file: <configuration> <property>    <name>fs.default.name</name>    <value>hdfs://addressofbsdnsofmynamenode-hadoop:9001</value> </property> </configuration> You can specify a port that you want to use, and it should not clash with the ports that are already in use by the system for various purposes. Save the file. This helps us create a master /NameNode. Now, let's move to set up SecondryNodes, let's edit /u/HBase B/hadoop-2.2.0/etc/hadoop/ and look for the file core-site.xml: <property>  <name>fs.defaultFS</name>  <value>hdfs://custome location of your hdfs</value> </property> <configuration> <property>           <name>fs.checkpoint.dir</name>           <value>/u/HBase B/dn001/hadoop/hdf/secdn        /u/HBase B/dn002/hadoop/hdfs/secdn </value>    </property> </configuration> The separation of the directory structure is for the purpose of a clean separation of the HDFS block separation and to keep the configurations as simple as possible. This also allows us to do a proper maintenance. Now, let's move towards changing the setup for hdfs; the file location will be /u/HBase B/hadoop-2.2.0/etc/hadoop/hdfs-site.xml. Add these properties in hdfs-site.xml: For NameNode: <property>          <name>dfs.name.dir</name>          <value> /u/HBase B/nn01/hadoop/hdfs/nn,/u/HBase B/nn02/hadoop/hdfs/nn </value>      </property> For DataNode: <property>          <name>dfs.data.dir</name>          <value> /u/HBase B/dnn01/hadoop/hdfs/dn,/HBase B/u/dnn02/hadoop/hdfs/dn </value> </property> Now, let's go for NameNode for http address or to access using http protocol: <property> <name>dfs.http.address</name> <value>yournamenode.full.hostname:50070</value> </property> <property> <name>dfs.secondary.http.address</name> <value> secondary.yournamenode.full.hostname:50090 </value>      </property> We can go for the https setup for the NameNode too, but let's keep it optional for now: Let's set up the yarn resource manager: Let's look for Yarn setup: /u/HBase B/hadoop-2.2.0/etc/hadoop/ yarn-site.xml For resource tracker a part of yarn resource manager: <property>  <name>yarn.yourresourcemanager.resourcetracker.address</name> <value>youryarnresourcemanager.full.hostname:8025</value> </property> For resource schedule part of yarn resource scheduler: <property> <name>yarn.yourresourcemanager.scheduler.address</name> <value>yourresourcemanager.full.hostname:8030</value> </property> For scheduler address: <property> <name>yarn.yourresourcemanager.address</name> <value>yourresourcemanager.full.hostname:8050</value> </property> For scheduler admin address: <property> <name>yarn.yourresourcemanager.admin.address</name> <value>yourresourcemanager.full.hostname:8041</value> </property> To set up a local dir: <property>         <name>yarn.yournodemanager.local-dirs</name>         <value>/u/HBase /dnn01/hadoop/hdfs /yarn,/u/HBase B/dnn02/hadoop/hdfs/yarn </value>    </property> To set up a log location: <property> <name> yarn.yournodemanager.logdirs </name>          <value>/u/HBase B/var/log/hadoop/yarn</value> </property> This completes the configuration changes required for Yarn. Now, let's make the changes for Map reduce: Let's open the mapred-site.xml: /u/HBase B/hadoop-2.2.0/etc/hadoop/mapred-site.xml Now, let's place this property configuration setup in the mapred-site.xml and place it between the following: <configuration > </configurations > <property><name>mapreduce.yourjobhistory.address</name> <value>yourjobhistoryserver.full.hostname:10020</value> </property> Once we have configured Map reduce job history details, we can move on to configure HBase . Let's go to this path /u/HBase B/HBase -0.98.3-hadoop2/conf and open HBase -site.xml. You will see a template having the following: <configuration > </configurations > We need to add the following lines between the starting and ending tags: <property> <name>HBase .rootdir</name> <value>hdfs://HBase .yournamenode.full.hostname:8020/apps/HBase /data </value> </property> <property> <name>HBase .yourmaster.info.bindAddress</name> <value>$HBase .yourmaster.full.hostname</value> </property> This competes the HBase changes. ZooKeeper: Now, let's focus on the setup of ZooKeeper. In distributed env, let's go to this location and rename the zoo_sample.cfg to zoo.cfg: /u/HBase B/zookeeper-3.4.6/conf Open zoo.cfg by vi zoo.cfg and place the details as follows; this will create two instances of zookeeper on different ports: yourzooKeeperserver.1=zoo1:2888:3888 yourZooKeeperserver.2=zoo2:2888:3888 If you want to test this setup locally, please use different port combinations. In a production-like setup as mentioned earlier, yourzooKeeperserver.1=zoo1:2888:3888 is server.id=host:port:port: yourzooKeeperserver.1= server.id zoo1=host 2888=port 3888=port Atomic broadcasting is an atomic messaging system that keeps all the servers in sync and provides reliable delivery, total order, casual order, and so on. Region servers: Before concluding it, let's go through the region server setup process. Go to this folder /u/HBase B/HBase -0.98.3-hadoop2/conf and edit the regionserver file. Specify the region servers accordingly: RegionServer1 RegionServer2 RegionServer3 RegionServer4 RegionServer1 equal to the IP or fully qualified CNAME of 1 Region server. You can have as many region servers (1. N=4 in our case), but its CNAME and mapping in the region server file need to be different. Copy all the configuration files of HBase and ZooKeeper to the relative host dedicated for HBase and ZooKeeper. As the setup is in a fully distributed cluster mode, we will be using a different host for HBase and its components and a dedicated host for ZooKeeper. Next, we validate the setup we've worked on by adding the following to the bashrc, this will make sure later we are able to configure the NameNode as expected: It preferred to use it in your profile, essentially /etc/profile; this will make sure the shell which is used is only impacted. Now let's format NameNode: Sudo su $HDFS_USER /u/HBase B/hadoop-2.2.0/bin/hadoop namenode -format HDFS is implemented on the existing local file system of your cluster. When you want to start the Hadoop setup first time you need to start with a clean slate and hence any existing data needs to be formatted and erased. Before formatting we need to take care of the following. Check whether there is a Hadoop cluster running and using the same HDFS; if it's done accidentally all the data will be lost. /u/HBase B/hadoop-2.2.0/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start namenode Now let's go to the SecondryNodes: Sudo su $HDFS_USER /u/HBase B/hadoop-2.2.0/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start secondarynamenode Repeating the same procedure in DataNode: Sudo su $HDFS_USER /u/HBase B/hadoop-2.2.0/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start datanode Test 01> See if you can reach from your browser http://namenode.full.hostname:50070: Test 02> sudo su $HDFS_USER touch /tmp/hello.txt Now, hello.txt file will be created in tmp location: /u/HBase B/hadoop-2.2.0/bin/hadoop dfs  -mkdir -p /app /u/HBase B/hadoop-2.2.0/bin/hadoop dfs  -mkdir -p /app/apphduser This will create a specific directory for this application user in the HDFS FileSystem location(/app/apphduser) /u/HBase B/hadoop-2.2.0/bin/hadoop dfs -copyFromLocal /tmp/hello.txt /app/apphduser /u/HBase B/hadoop-2.2.0/bin/hadoop dfs –ls /app/apphduser apphduser is a dirctory which is created in hdfs for a specific user. So that the data is sepreated based on the users, in a true production env many users will be using it. You can also use hdfs dfs –ls / commands if it shows hadoop command as depricated. You must see hello.txt once the command executes: Test 03> Browse http://datanode.full.hostname:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/&nnaddr=$datanode.full.hostname:8020 It is important to change the data host name and other parameters accordingly. You should see the details on the DataNode. Once you hit the preceding URL you will get the following screenshot: On the command line it will be as follows: Validate Yarn/MapReduce setup and execute this command from the resource manager: <login as $YARN_USER> /u/HBase B/hadoop-2.2.0/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager Execute the following command from NodeManager: <login as $YARN_USER > /u/HBase B/hadoop-2.2.0/sbin /yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager Executing the following commands will create the directories in the hdfs and apply the respective access rights: Cd u/HBase B/hadoop-2.2.0/bin hadoop fs -mkdir /app-logs // creates the dir in HDFS hadoop fs -chown $YARN_USER /app-logs //changes the ownership hadoop fs -chmod 1777 /app-logs // explained in the note section Execute MapReduce Start jobhistory servers: <login as $MAPRED_USER> /u/HBase B/hadoop-2.2.0/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR Let's have a few tests to be sure we have configured properly: Test 01: From the browser or from curl use the link to browse: http://yourresourcemanager.full.hostname:8088/. Test 02: Sudo su $HDFS_USER /u/HBase B/hadoop-2.2.0/bin/hadoop jar /u/HBase B/hadoop-2.2.0/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.2.1-alpha.jar teragen 100 /test/10gsort/input /u/HBase B/hadoop-2.2.0/bin/hadoop jar /u/HBase B/hadoop-2.2.0/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.2.1-alpha.jar Validate the HBase setup: Login as $HDFS_USER /u/HBase B/hadoop-2.2.0/bin/hadoop fs –mkdir -p /apps/HBase /u/HBase B/hadoop-2.2.0/bin/hadoop fs –chown app:app –R  /apps/HBase Now login as $HBase _USER: /u/HBase B/HBase -0.98.3-hadoop2/bin/HBase -daemon.sh –-config $HBase _CONF_DIR start master This command will start the master node. Now let's move to HBase Region server nodes: /u/HBase B/HBase -0.98.3-hadoop2/bin/HBase -daemon.sh –-config $HBase _CONF_DIR start regionserver This command will start the regionservers: For a single machine, direct sudo ./HBase master start can also be used. Please check the logs in case of any logs at this location /opt/HBase B/HBase -0.98.5-hadoop2/logs. You can check the log files and check for any errors: Now let's login using: Sudo su- $HBase _USER /u/HBase B/HBase -0.98.3-hadoop2/bin/HBase shell We will connect HBase to the master. Validate the ZooKeeper setup. If you want to use an external zookeeper, make sure there is no internal HBase based zookeeper running while working with the external zookeeper or existing zookeeper and is not managed by HBase : For this you have to edit /opt/HBase B/HBase -0.98.5-hadoop2/conf/ HBase -env.sh. Change the following statement (HBase _MANAGES_ZK=false): # Tell HBase whether it should manage its own instance of Zookeeper or not. export HBase _MANAGES_ZK=true. Once this is done we can add zoo.cfg to HBase 's CLASSPATH. HBase looks into zoo.cfg as a default lookup for configurations dataDir=/opt/HBase B/zookeeper-3.4.6/zooData # this is the place where the zooData will be present server.1=172.28.182.45:2888:3888 # IP and port for server 01 server.2=172.29.75.37:4888:5888 # IP and port for server 02 You can edit the log4j.properties file which is located at /opt/HBase B/zookeeper-3.4.6/conf and point the location where you want to keep the logs. # Define some default values that can be overridden by system properties: zookeeper.root.logger=INFO, CONSOLE zookeeper.console.threshold=INFO zookeeper.log.dir=. zookeeper.log.file=zookeeper.log zookeeper.log.threshold=DEBUG zookeeper.tracelog.dir=. # you can specify the location here zookeeper.tracelog.file=zookeeper_trace.log Once this is done you start zookeeper with the following command: -bash-4.1$ sudo /u/HBase B/zookeeper-3.4.6/bin/zkServer.sh start Starting zookeeper ... STARTED You can also pipe the log to the ZooKeeper logs: /u/logs//u/HBase B/zookeeper-3.4.6/zoo.out 2>&1 2 : refers to the second file descriptor for the process, that is stderr. > : means re-direct &1:  means the target of the rediretion should be the same location as the first file descriptor i.e stdout How it works Sizing of the environment is very critical for the success of any project, and it's a very complex task to optimize it to the needs. We dissect it into two parts, master and slave setup. We can divide it in the following parts: Master-NameNode Master-Secondary NameNode Master-Jobtracker Master-Yarn Resource Manager Master-HBase Master Slave-DataNode Slave-Map Reduce Tasktracker Slave-Yarn Node Manager Slave-HBase Region server NameNode: The architecture of Hadoop provides us a capability to set up a fully fault tolerant/high availability Hadoop/HBase cluster. In doing so, it requires a master and slave setup. In a fully HA setup, nodes are configured in active passive way; one node is always active at any given point of time and the other node remains as passive. Active node is the one interacting with the clients and works as a coordinator to the clients. The other standby node keeps itself synchronized with the active node and to keep the state intact and live, so that in case of failover it is ready to take the load without any downtime. Now we have to make sure that when the passive node comes up in the event of a failure, the passive node is in perfect sync with the active node, which is currently taking the traffic. This is done by Journal Nodes(JNs), these Journal Nodes use daemon threads to keep the primary and sercodry in perfect sync. Journal Node: By design, JournalNodes will only have single NameNode acting as a active/primary to be a writer at a time. In case of failure of the active/primary, the passive NameNode immediately takes the charge and transforms itself as active, this essentially means this newly active node starts writing to Journal Nodes. Thus it totally avoids the other NameNode to stay in active state, this also acknowledges that the newly active node work as a fail over node. JobTracker: This is an integral part of Hadoop EcoSystem. It works as a service which farms MapReduce task to specific nodes in the cluster. ResourceManager (RM): This responsibility is limited to scheduling, that is, only mediating available resources in the system between different needs for the application like registering new nodes, retiring dead nodes, it dose it by constantly monitoring the heartbeats based on the internal configuration. Due to this core design practice of explicit separation of responsibilities and clear orchestrations of modularity and with the inbuilt and robust scheduler API, This allows the resource manager to scale and support different design needs at one end, and on the other, it allows us to cater to different programming models. HBase Master: The Master server is the main orchestrator for all the region servers in the HBase cluster . Usually, it's placed on the ZooKeeper nodes. In a real cluster configuration, you will have 5 to 6 nodes of Zookeeper. DataNode: It's a real workhorse and does most of the heavy lifting; it runs the MapReduce Job and stores the chunks of HDFS data. The core objective of the data node was to be available on the commodity hardware and should be agnostic to the failures. It keeps some data of HDFS, and the multiple copy of the same data is sprinkled around the cluster. This makes the DataNode architecture fully fault tolerant. This is the reason a data node can have JBOD01 rather rely on the expensive RAID02. MapReduce: Jobs are run on these DataNodes in parallel as a subtask. These subtasks provides the consistent data across the cluster and stays consistent. So we learned about the HBase basics and how to configure and set it up. We set up HBase to store data in Hadoop Distributed File System. We also explored the working structure of RAID and JBOD and the differences between both filesystems. If you found this post useful, be sure to check out the book ‘HBase High Perforamnce Cookbook’ to learn more about configuring HBase in terms of administering and managing clusters as well as other concepts in HBase. Understanding the HBase Ecosystem Configuring HBase 5 Mistake Developers make when working with HBase    
Read more
  • 0
  • 0
  • 8894

article-image-learning-how-manage-records-visualforce
Packt
29 Sep 2016
7 min read
Save for later

Learning How to Manage Records in Visualforce

Packt
29 Sep 2016
7 min read
In this article by Keir Bowden, author of the book, Visualforce Development Cookbook - Second Edition we will cover the following styling fields and table columns as per requirement One of the common use cases for Visualforce pages is to simplify, streamline, or enhance the management of sObject records. In this article, we will use Visualforce to carry out some more advanced customization of the user interface—redrawing the form to change available picklist options, or capturing different information based on the user's selections. (For more resources related to this topic, see here.) Styling fields as required Standard Visualforce input components, such as <apex:inputText />, can take an optional required attribute. If set to true, the component will be decorated with a red bar to indicate that it is required, and form submission will fail if a value has not been supplied, as shown in the following screenshot: In the scenario where one or more inputs are required and there are additional validation rules, for example, when one of either the Email or Phone fields is defined for a contact, this can lead to a drip feed of error messages to the user. This is because the inputs make repeated unsuccessful attempts to submit the form, each time getting slightly further in the process. Now, we will create a Visualforce page that allows a user to create a contact record. The Last Name field is captured through a non-required input decorated with a red bar identical to that created for required inputs. When the user submits the form, the controller validates that the Last Name field is populated and that one of the Email or Phone fields is populated. If any of the validations fail, details of all errors are returned to the user. Getting ready This topic makes use of a controller extension so this must be created before the Visualforce page. How to do it… Navigate to the Apex Classes setup page by clicking on Your Name | Setup | Develop | Apex Classes. Click on the New button. Paste the contents of the RequiredStylingExt.cls Apex class from the code downloaded into the Apex Class area. Click on the Save button. Navigate to the Visualforce setup page by clicking on Your Name | Setup | Develop | Visualforce Pages. Click on the New button. Enter RequiredStyling in the Label field. Accept the default RequiredStyling that is automatically generated for the Name field. Paste the contents of the RequiredStyling.page file from the code downloaded into the Visualforce Markup area and click on the Save button. Navigate to the Visualforce setup page by clicking on Your Name | Setup | Develop | Visualforce Pages. Locate the entry for the RequiredStyling page and click on the Security link. On the resulting page, select which profiles should have access and click on the Save button. How it works… Opening the following URL in your browser displays the RequiredStyling page to create a new contact record: https://<instance>/apex/RequiredStyling. Here, <instance> is the Salesforce instance specific to your organization, for example, na6.salesforce.com. Clicking on the Save button without populating any of the fields results in the save failing with a number of errors: The Last Name field is constructed from a label and text input component rather than a standard input field, as an input field would enforce the required nature of the field and stop the submission of the form: <apex:pageBlockSectionItem > <apex:outputLabel value="Last Name"/> <apex:outputPanel id="detailrequiredpanel" layout="block" styleClass="requiredInput"> <apex:outputPanel layout="block" styleClass="requiredBlock" /> <apex:inputText value="{!Contact.LastName}"/> </apex:outputPanel> </apex:pageBlockSectionItem> The required styles are defined in the Visualforce page rather than relying on any existing Salesforce style classes to ensure that if Salesforce changes the names of its style classes, this does not break the page. The controller extension save action method carries out validation of all fields and attaches error messages to the page for all validation failures: if (String.IsBlank(cont.name)) { ApexPages.addMessage(new ApexPages.Message( ApexPages.Severity.ERROR, 'Please enter the contact name')); error=true; } if ( (String.IsBlank(cont.Email)) && (String.IsBlank(cont.Phone)) ) { ApexPages.addMessage(new ApexPages.Message( ApexPages.Severity.ERROR, 'Please supply the email address or phone number')); error=true; } Styling table columns as required When maintaining records that have required fields through a table, using regular input fields can end up with an unsightly collection of red bars striped across the table. Now, we will create a Visualforce page to allow a user to create a number of contact records via a table. The contact Last Name column header will be marked as required, rather than the individual inputs. Getting ready This topic makes use of a custom controller, so this will need to be created before the Visualforce page. How to do it… First, create the custom controller by navigating to the Apex Classes setup page by clicking on Your Name | Setup | Develop | Apex Classes. Click on the New button. Paste the contents of the RequiredColumnController.cls Apex class from the code downloaded into the Apex Class area. Click on the Save button. Next, create a Visualforce page by navigating to the Visualforce setup page by clicking on Your Name | Setup | Develop | Visualforce Pages. Click on the New button. Enter RequiredColumn in the Label field. Accept the default RequiredColumn that is automatically generated for the Name field. Paste the contents of the RequiredColumn.page file from the code downloaded into the Visualforce Markup area and click on the Save button. Navigate to the Visualforce setup page by clicking on Your Name | Setup | Develop | Visualforce Pages. Locate the entry for the RequiredColumn page and click on the Security link. On the resulting page, select which profiles should have access and click on the Save button. How it works… Opening the following URL in your browser displays the RequiredColumn page: https://<instance>/apex/RequiredColumn. Here, <instance> is the Salesforce instance specific to your organization, for example, na6.salesforce.com. The Last Name column header is styled in red, indicating that this is a required field. Attempting to create a record where only First Name is specified results in an error message being displayed against the Last Name input for the particular row: The Visualforce page sets the required attribute on the inputField components in the Last Name column to false, which removes the red bar from the component: <apex:column > <apex:facet name="header"> <apex:outputText styleclass="requiredHeader" value="{!$ObjectType.Contact.fields.LastName.label}" /> </apex:facet> <apex:inputField value="{!contact.LastName}" required="false"/> </apex:column> The Visualforce page custom controller Save method checks if any of the fields in the row are populated, and if this is the case, it checks that the last name is present. If the last name is missing from any record, an error is added. If an error is added to any record, the save does not complete: if ( (!String.IsBlank(cont.FirstName)) || (!String.IsBlank(cont.LastName)) ) { // a field is defined - check for last name if (String.IsBlank(cont.LastName)) { error=true; cont.LastName.addError('Please enter a value'); } String.IsBlank() is used as this carries out three checks at once: to check that the supplied string is not null, it is not empty, and it does not only contain whitespace. Summary Thus in this article we successfully mastered the techniques to style fields and table columns as per the custom needs. Resources for Article: Further resources on this subject: Custom Components in Visualforce [Article] Visualforce Development with Apex [Article] Using Spring JMX within Java Applications [Article]
Read more
  • 0
  • 0
  • 8891

article-image-creating-custom-reports-and-notifications-vsphere
Packt
24 Mar 2015
34 min read
Save for later

Creating Custom Reports and Notifications for vSphere

Packt
24 Mar 2015
34 min read
In this article by Philip Sellers, the author of PowerCLI Cookbook, you will cover the following topics: Getting alerts from a vSphere environment Basics of formatting output from PowerShell objects Sending output to CSV and HTML Reporting VM objects created during a predefined time period from VI Events object Setting custom properties to add useful context to your virtual machines Using PowerShell native capabilities to schedule scripts (For more resources related to this topic, see here.) This article is all about leveraging the information available to you in PowerCLI. As much as any other topic, figuring out how to tap into the data that PowerCLI offers is as important as understanding the cmdlets and syntax of the language. However, once you obtain your data, you will need to alter the formatting and how it's returned to be used. PowerShell, and by extension PowerCLI, offers a big set of ways to control the formatting and the display of information returned by its cmdlets and data objects. You will explore all of these topics with this article. Getting alerts from a vSphere environment Discovering the data available to you is the most difficult thing that you will learn and adopt in PowerCLI after learning the initial cmdlets and syntax. There is a large amount of data available to you through PowerCLI, but there are techniques to extract the data in a way that you can use. The Get-Member cmdlet is a great tool for discovering the properties that you can use. Sometimes, just listing the data returned by a cmdlet is enough; however, when the property contains other objects, Get-Member can provide context to know that the Alarm property is a Managed Object Reference (MoRef) data type. As your returned objects have properties that contain other objects, you can have multiple layers of data available for you to expose using PowerShell dot notation ($variable.property.property). The ExtensionData property found on most objects has a lot of related data and objects to the primary data. Sometimes, the data found in the property is an object identifier that doesn't mean much to an administrator but represents an object in vSphere. In these cases, the Get-View cmdlet can refer to that identifier and return human-readable data. This topic will walk you through the methods of accessing data and converting it to usable, human-readable data wherever needed so that you can leverage it in scripts. To explore these methods, we will take a look at vSphere's built-in alert system. While PowerCLI has native cmdlets to report on the defined alarm states and actions, it doesn't have a native cmdlet to retrieve the triggered alarms on a particular object. To do this, you must get the datacenter, VMhost, VM, and other objects directly and look at data from the ExtensionData property. Getting ready To begin this topic, you will need a PowerCLI window and an active connection to vCenter. You should also check the vSphere Web Client or the vSphere Windows Client to see whether you have any active alarms and to know what to expect. If you do not have any active VM alarms, you can simulate an alarm condition using a utility such as HeavyLoad. For more information on generating an alarm, see the There's more... section of this topic. How to do it… In order to access data and convert it to usable, human-readable data, perform the following steps: The first step is to retrieve all of the VMs on the system. A simple Get-VM cmdlet will return all VMs on the vCenter you're connected to. Within the VM object returned by Get-VM, one of the properties is ExtensionData. This property is an object that contains many additional properties and objects. One of the properties is TriggeredAlarmState: Get-VM | Where {$_.ExtensionData.TriggeredAlarmState -ne $null} To dig into TriggeredAlarmState more, take the output of the previous cmdlet and store it into a variable. This will allow you to enumerate the properties without having to wait for the Get-VM cmdlet to run. Add a Select -First 1 cmdlet to the command string so that only a single object is returned. This will help you look inside without having to deal with multiple VMs in the variable: $alarms = Get-VM | Where {$_.ExtensionData.TriggeredAlarmState -ne $null} | Select -First 1 Now that you have extracted an alarm, how do you get useful data about what type of alarm it is and which vSphere object has a problem? In this case, you have VM objects since you used Get-VM to find the alarms. To see what is in the TriggeredAlarmState property, output the contents of TriggeredAlarmState and pipe it to Get-Member or its shortcut GM: $alarms.ExtensionData.TriggeredAlarmState | GM The following screenshot shows the output of the preceding command line: List the data in the $alarms variable without the Get-Member cmdlet appended and view the data in a real alarm. The data returned does tell you the time when the alarm was triggered, the OverallStatus property or severity of the alarm, and whether the alarm has been acknowledged by an administrator, who acknowledged it and at what time. You will see that the Entity property contains a reference to a virtual machine. You can use the Get-View cmdlet on a reference to a VM, in this case, the Entity property, and return the virtual machine name and other properties. You will also see that Alarm is referred to in a similar way and we can extract usable information using Get-View also: Get-View $alarms.ExtensionData.TriggeredAlarmState.Entity Get-View $alarms.ExtensionData.TriggeredAlarmState.Alarm You can see how the output from these two views differs. The Entity view provides the name of the VM. You don't really need this data since the top-level object contains the VM name, but it's good to understand how to use Get-View with an entity. On the other hand, the data returned by the Alarm view does not show the name or type of the alarm, but it does include an Info property. Since this is the most likely property with additional information, you should list its contents. To do so, enclose the Get-View cmdlet in parenthesis and then use dot notation to access the Info variable: (Get-View $alarms.ExtensionData.TriggeredAlarmState.Alarm).Info In the output from the Info property, you can see that the example alarm in the screenshot is a Virtual Machine CPU usage alarm. Your alarm can be different, but it should appear similar to this. After retrieving PowerShell objects that contain the data that you need, the easiest way to return the data is to use calculated expressions. Since the Get-VM cmdlet was the source for all lookup data, you will need to use this object with the calculated expressions to display the data. To do this, you will append a Select statement after the Get-VM and Where statement. Notice that you use the same Get-View statement, except that you change your variable to $_, which is the current object being passed into Select: Get-VM | Where {$_.ExtensionData.TriggeredAlarmState -ne $null} | Select Name, @{N="AlarmName";E={(Get-View $_.ExtensionData. TriggeredAlarmState.Alarm).Info.Name}}, @{N="AlarmDescription";E={(Get-View $_.ExtensionData. TriggeredAlarmState.Alarm).Info.Description}}, @ {N="TimeTriggered"; E={$_.ExtensionData.TriggeredAlarmState. Time}}, @{N="AlarmOverallStatus"; E={$_.ExtensionData. TriggeredAlarmState. OverallStatus}} How it works When the data you really need is several levels below the top-level properties of a data object, you need to use calculated expressions to return these at the top level. There are other techniques where you can build your own object with only the data you want returned, but in a large environment with thousands of objects in vSphere, the method in this topic will execute faster than looping through many objects to build a custom object. Calculated expressions are extremely powerful since nearly anything can be done with expressions. More than that, you explored techniques to discover the data you want. Data exploration can provide you with incredible new capabilities. The point is you need to know where the data is and how to pull that data back to the top level. There's more… It is likely that your test environment has no alarms and in this case, it might be up to you to create an alarm situation. One of the easiest to control and create is heavy CPU load with a CPU load-testing tool. JAM Software created software named HeavyLoad that is a stress-testing tool. This utility can be loaded into any Windows VM on your test systems and can consume all of the available CPU that the VM is configured with. To be safe, configure the VM with a single vCPU and the utility will consume all of the available CPU. Once you install the utility, go to the Test Options menu and you can uncheck the Stress GPU option, ensure that Stress CPU and Allocate Memory are checked. The utility also has shortcut buttons on the Menu bar to allow you to set these options. Click on the Start button (which looks like a Play button) and the utility begins to stress the VM. For users who wish to do the same test, but utilize Linux, StressLinux is a great option. StressLinux is a minimal distribution designed to create high load on an operating system. See also You can read more about the HeavyLoad Utility available under the JAM Software page at http://www.jam-software.com/heavyload/ You can read more about StressLinux at http://www.stresslinux.org/sl/ Basics of formatting output from PowerShell objects Anything that exists in a PowerShell object can be output as a report, e-mail, or editable file. Formatting the output is a simple task in PowerShell. Sometimes, the information you receive in the object is in a long decimal number format, but to make it more readable, you want to truncate the output to just a couple decimal places. In this section, you will take a look at the Format-Table, Format-Wide, and Format-List cmdlets. You will dig into the Format-Custom cmdlet and also take a look at the -f format operator. The truth is that native cmdlets do a great job returning data using default formatting. When we start changing and adding our own data to the list of properties returned, the formatting can become unoptimized. Even in the returned values of a native cmdlet, some columns might be too narrow to display all of the information. Getting ready To begin this topic, you will need the PowerShell ISE. How to do it… In order to format the output from PowerShell objects, perform the following steps: Run Add-PSSnapIn VMware.VimAutomation.Core in the PowerShell ISE to initialize a PowerCLI session and bring in the VMware cmdlet. Connect to your vCenter server. Start with a simple object from a Get-VM cmdlet. The default output is in a table format. If you pipe the object to Format-Wide, it will change the default output into a multicolumn with a single property, just like running a dir /w command at the Windows Command Prompt. You can also use FW, an alias for Format-Wide: Get-VM | Format-Wide Get-VM | FW If you take the same object and pipe it to Format-Table or its alias FT, you will receive the same output if you use the default output for Get-VM: Get-VM Get-VM | Format-Table However, as soon as you begin to select a different order of properties, the default formatting disappears. Select the same four properties and watch the formatting change. The default formatting disappears. Get-VM | Select Name, PowerState, NumCPU, MemoryGB | FT To restore formatting to table output, you have a few choices. You can change the formatting on the data in the object using the Select statement and calculated expressions. You can also pass formatting through the Format-Table cmdlet. While setting formatting in the Select statement changes the underlying data, using Format-Table doesn't change the data, but only its display. The formatting looks essentially like a calculated expression in a Select statement. You provide Label, Expression, and formatting commands: Get-VM | Select * | FT Name, PowerState, NumCPU, @{Label="MemoryGB"; Expression={$_.MemoryGB}; FormatString="N2"; Alignment="left"} If you have data in a number data type, you can convert it into a string using the ToString() method on the object. You can try this method on NumCPU: Get-VM | Select * | FT Name, PowerState, @{Label="Num CPUs"; Expression={($_.NumCpu).ToString()}; Alignment="left"}, @{Label="MemoryGB"; Expression={$_.MemoryGB}; FormatString="N2"; Alignment="left"} The other method is to format with the -f operator, which is basically a .NET derivative. To better understand the formatting and string, the structure is {<index>[,<alignment>][:<formatString>]}. Index sets that are a part of the data being passed, will be transformed. The alignment is a numeric value. A positive number will right-align those number of characters. A negative number will left-align those number of characters. The formatString parameter is the part that defines the format to apply. In this example, let's take a datastore and compute the percentage of free disk space. The format for percent is p: Get-Datastore | Select Name, @{N="FreePercent";E={"{0:p} -f ($_.FreeSpaceGB / $_.CapacityGB)}} To make the FreePercent column 15 characters wide, you add 0,15:p to the format string: Get-Datastore | Select Name, @{N="FreePercent";E={"{0,15:p} -f ($_.FreeSpaceGB / $_.CapacityGB)}} How it works… With the Format-Table, Format-List, and Format-Wide cmdlets, you can change the display of data coming from a PowerCLI object. These cmdlets all apply basic transformations without changing the data in the object. This is important to note because once the data is changed, it can prevent you from making changes. For instance, if you take the percentage example, after transforming the FreePercent property, it is stored as a string and no longer as a number, which means that you can't reformat it again. Applying a similar transformation from the Format-Table cmdlet would not alter your data. This doesn't matter when you're performing a one-liner, but in a more complex script or in a routine, where you might need to not only output the data but also reuse it, changing the data in the object is a big deal. There's more… This topic only begins to tap the full potential of PowerShell's native -f format operator. There are hundreds of blog posts about this topic, and there are use cases and examples of how to produce the formatting that you are looking for. The following link also gives you more details about the operator and formatting strings that you can use in your own code. See also For more information, refer to the PowerShell -f Format operator page available at http://ss64.com/ps/syntax-f-operator.html Sending output to CSV and HTML On the screen the output is great, but there are many times when you need to share your results with other people. When looking at sharing information, you want to choose a format that is easy to view and interpret. You might also want a format that is easy to manipulate and change. Comma Separated Values (CSV) files allow the user to take the output you generate and use it easily within a spreadsheet software. This allows you the ability to compare the results from vSphere versus internal tracking databases or other systems easily to find differences. It can also be useful to compare against service contracts for physical hosts as examples. HTML is a great choice for displaying information for reading, but not manipulation. Since e-mails can be in an HTML format, converting the output from PowerCLI (or PowerShell) into an e-mail is an easy way to assemble an e-mail to other areas of the business. What's even better about these cmdlets is the ease of use. If you have a data object in PowerCLI, all that you need to do is pipe that data object into the ConvertTo-CSV or ConvertTo-HTML cmdlets and you instantly get the formatted data. You might not be satisfied with the HTML-generated version alone, but like any other HTML, you can transform the look and formatting of the HTML using CSS by adding a header. In this topic, you will examine the conversion cmdlets with a simple set of Get- cmdlets. You will also take a look at trimming results using the Select statements and formatting HTML results with CSS. This topic will pull a list of virtual machines and their basic properties to send to a manager who can reconcile it against internal records or system monitoring. It will export to a CSV file that will be attached to the e-mail and you will use the HTML to format a list in an e-mail to send to the manager. Getting ready To begin this topic, you will need to use the PowerShell ISE. How to do it… In order to examine the conversion cmdlets using Get- cmdlets, trim results using the Select statements, and format HTML results with CSS, perform the following steps: Open the PowerShell ISE and run Add-PSSnapIn VMware.VimAutomation.Core to initialize a PowerCLI session within the ISE. Again, you will use the Get-VM cmdlet as the base for this topic. The fields that we care about are the name of the VM, the number of CPUs, the amount of memory, and the description: $VMs = Get-VM | Select Name, NumCPU, MemoryGB, Description In addition to the top-level data, you also want to provide the IP address, hostname, and the operating system. These are all available from the ExtensionData.Guest property: $VMs = Get-VM | Select Name, NumCPU, MemoryGB, Description, @{N="Hostname";E={$_.ExtensionData.Guest.HostName}}, @{N="IP";E={$_.ExtensionData.Guest.IPAddress}}, @{N="OS";E={$_.ExtensionData.Guest.GuestFullName}} The next step is to take this data and format it to be sent as an HTML e-mail. Converting the information to HTML is actually easy. Pipe the variable you created with the data into ConvertTo-HTML and store in a new variable. You will need to reuse the data to convert it to a CSV file to attach: $HTMLBody = $VMs | ConvertTo-HTML If you were to output the contents of $HTMLBody, you will see that it is very plain, inheriting the defaults of the browser or e-mail program used to display it. To dress this up, you need to define some basic CSS to add some style for the <body>, <table>, <tr>, <td>, and <th> tags. You can add this by running the ConvertTo-HTML cmdlet again with the -PreContent parameter: $css = "<style> body { font-family: Verdana, sans-serif; fontsize: 14px; color: #666; background: #FFF; } table{ width:100%; border-collapse:collapse; } table td, table th { border:1px solid #333; padding: 4px; } table th { text-align:left; padding: 4px; background-color:#BBB; color:#FFF;} </style>" $HTMLBody = $VMs | ConvertTo-HTML -PreContent $css It might also be nice to add the date and time generated to the end of the file. You can use the -PostContent parameter to add this: $HTMLBody = $VMs | ConvertTo-HTML -PreContent $css -PostContent "<div><strong>Generated:</strong> $(Get-Date)</div>" Now, you have the HTML body of your message. To take the same data from $VMs and save it to a CSV file that you can use, you will need a writable directory, and a good choice is to use your My Documents folder. You can obtain this using an environment variable: $tempdir = [environment]::getfolderpath("mydocuments") Now that you have a temp directory, you can perform your export. Pipe $VMs to Export-CSV and specify the path and filename: $VMs | Export-CSV $tempdirVM_Inventory.csv At this point, you are ready to assemble an e-mail and send it along with your attachment. Most of the cmdlets are straightforward. You set up a $msg variable that is a MailMessage object. You create an Attachment object and populate it with your temporary filename and then create an SMTP server with the server name: $msg = New-Object Net.Mail.MailMessage $attachment = new-object Net.Mail.Attachment("$tempdirVM_Inventory.csv") $smtpServer = New-Object Net.Mail.SmtpClient("hostname") You set the From, To, and Subject parameters of the message variable. All of these are set with dot notation on the $msg variable: $msg.From = "fromaddress@yourcompany.com" $msg.To.Add("admin@yourcompany.com") $msg.Subject = "Weekly VM Report" You set the body you created earlier, as $HTMLBody, but you need to run it through Out-String to convert any other data types to a pure string for e-mailing. This prevents an error where System.String[] appears instead of your content in part of the output: $msg.Body = $HTMLBody | Out-String You need to take the attachment and add it to the message: $msg.Attachments.Add($attachment) You need to set the message to an HTML format; otherwise, the HTML will be sent as plain text and not displayed as an HTML message: $msg.IsBodyHtml = $true Finally, you are ready to send the message using the $smtpServer variable that contains the mail server object. Pass in the $msg variable to the server object using the Send method and it transmits the message via SMTP to the mail server: $smtpServer.Send($msg) Don't forget to clean up the temporary CSV file you generated. To do this, use the PowerShell Remove-Item cmdlet that will remove the file from the filesystem. Add a -Confirm parameter to suppress any prompts: Remove-Item $tempdirVM_Inventory.csv -Confirm:$false How it works… Most of this topic relies on native PowerShell and less on the PowerCLI portions of the language. This is the beauty of PowerCLI. Since it is based on PowerShell and only an extension, you lose none of the functions of PowerShell, a very powerful set of commands in its own right. The ConvertTo-HTML cmdlet is very easy to use. It requires no parameters to produce HTML, but the HTML it produces isn't the most legible if you display it. However, a bit of CSS goes a long way to improve the look of the output. Add some colors and style to the table and it becomes a really easy and quick way to format a mail message of data to be sent to a manager on a weekly basis. The Export-CSV cmdlet lets you take the data returned by a cmdlet and convert that into an editable format for use. You can place this onto a file share for use or you can e-mail it along, as you did in this topic. This topic takes you step by step through the process of creating a mail message, formatting it in HTML, and making sure that it's relayed as an HTML message. You also looked at how to attach a file. To send a mail, you define a mail server as an object and store it in a variable for reuse. You create a message object and store it in a variable and then set all of the appropriate configuration on the message. For an attachment, you create a third object and define a file to be attached. That is set as a property on the message object and then finally, the message object is sent using the server object. There's more… ConvertTo-HTML is just one of four conversion cmdlets in PowerShell. In addition to ConvertTo-HTML, you can convert data objects into XML. ConvertTo-JSON allows you to convert a data object into an XML format specific for web applications. ConvertTo-CSV is identical to Export-CSV except that it doesn't save the content immediately to a defined file. If you had a use case to manipulate the CSV before saving it, such as stripping the double quotes or making other alternations to the contents, you can use ConvertTo-CSV and then save it to a file at a later point in your script. Reporting VM objects created during a predefined time period from VI Events object An important auditing tool in your environment can be a report of when virtual machines were created, cloned, or deleted. Unlike snapshots, that store a created date on the snapshot, virtual machines don't have this property associated with them. Instead, you have to rely on the events log in vSphere to let you know when virtual machines were created. PowerCLI has the Get-VIEvents cmdlet that allows you to retrieve the last 1,000 events on the vCenter, by default. The cmdlet can accept a parameter to include more than the last 1,000 events. The cmdlet also allows you to specify a start date, and this can allow you to search for things within the past week or the past month. At a high level, this topic works the same in both PowerCLI and the vSphere SDK for Perl (VIPerl). They both rely on getting the vSphere events and selecting the specific events that match your criteria. Even though you are looking for VM creation events in this topic, you will see that the code can be easily adapted to look for many other types of events. Getting ready To begin this topic, you will need a PowerCLI window and an active connection to a vCenter server. How to do it… In order to report VM objects created during a predefined time period from VI Events object, perform the following steps: You will use the Get-VIEvent cmdlet to retrieve the VM creation events for this topic. To begin, get a list of the last 50 events from the vCenter host using the -MaxSamples parameter: Get-VIEvent -MaxSamples 50 If you pipe the output from the preceding cmdlet to Get-Member, you will see that this cmdlet can return a lot of different objects. However, the type of object isn't really what you need to find the VM's created events. Looking through the objects, they all include a GetType() method that returns the type of event. Inside the type, there is a name parameter. Create a calculated expression using GetType() and then group it based on this expression, you will get a usable list of events you can search for. This list is also good for tracking the number of events your systems have encountered or generated: Get-VIEvent -MaxSamples 2000 | Select @{N="Type";E={$_.GetType().Name}} | Group Type In the preceding screenshot, you will see that there are VMClonedEvent, VmRemovedEvent, and VmCreatedEvent listed. All of these have to do with creating or removing virtual machines in vSphere. Since you are looking for created events, VMClonedEvent and VmCreatedEvent are the two needed for this script. Write a Where statement to return only these events. To do this, we can use a regular expression with both the event names and the -match PowerShell comparison parameter: Get-VIEvent -MaxSamples 2000 | Where {$_.GetType().Name -match "(VmCreatedEvent|VmClonedEvent)"} Next, you want to select just the properties that you want in your output. To do this, add a Select statement and you will reuse the calculated expression from Step 3. If you want to return the VM name, which is in a Vm property with the type of VMware.Vim.VVmeventArgument, you can create a calculated expression to return the VM name. To round out the output, you can include the FullFormattedMessage, CreatedTime, and UserName properties: Get-VIEvent -MaxSamples 2000 | Where {$_.GetType().Name -match "(VmCreatedEvent|VmClonedEvent)"} | Select @{N="Type",E={$_.GetType().Name}}, @{N="VMName",E={$_.Vm.Name}},FullFormattedMessage, CreatedTime, UserName The last thing you will want to do is go back and add a time frame to the Get-VIEvent cmdlet. You can do this by specifying the -Start parameter along with (Get-Date).AddMonths(-1) to return the last month's events: Get-VIEvent -MaxSamples 2000 | Where {$_.GetType().Name -match "(VmCreatedEvent|VmClonedEvent)"} | Select @{N="Type",E={$_.GetType().Name}}, @{N="VMName",E={$_.Vm.Name}},FullFormattedMessage, CreatedTime, UserName How it works… The Get-VIEvent cmdlet drives a majority of this topic, but in this topic you only scratched the surface of the information you can unearth with Get-VIEvent. As you saw in the screenshot, there are so many different types of events that can be reported, queried, and acted upon from the vCenter server. Once you discover and know which events you are looking for specifically, then it's a matter of scoping down the results with a Where statement. Last, you use calculated expressions to pull data that is several levels deep in the returned data object. One of the primary things employed here is a regular expression used to search for the types of events you were interested in: VmCreatedEvent and VmClonedEvent. By combining a regular expression with the -match operator, you were able to use a quick and very understandable bit of code to find more than one type of object you needed to return. There's more… Regular Expressions (RegEx) are big topics on their own. These types of searches can match any type of pattern that you can establish or in the case of this topic, a number of defined values that you are searching for. RegEx are beyond the scope of this article, but they can be a big help anytime you have a pattern you need to search for and match, or perhaps more importantly, replace. You can use the -replace operator instead of –match to not only to find things that match your pattern, but also change them. See also For more information on Regular Expressions refer to http://ss64.com/ps/syntax-regex.html The PowerShell.com page: Text and Regular Expressions is available at http://powershell.com/cs/blogs/ebookv2/archive/2012/03/20/chapter-13-text-and-regular-expressions.aspx Setting custom properties to add useful context to your virtual machines Building on the use case for the Get-VIEvent cmdlet, Alan Renouf of VMware's PowerCLI team has a useful script posted on his personal blog (refer to the See also section) that helps you pull the created date and the user who created a virtual machine and populate this into a custom attribute. This is a great use for a custom attribute on virtual machines and makes some useful information available that is not normally visible. This is a process that needs to be run often to pick up details for virtual machines that have been created. Rather than looking specifically at a VM and trying to go back and find its creation date as Alan's script does, in this article, you will take a different approach building on the previous article and populate the information from the found creation events. Maintenance in this form would be easier by finding creation events for the last week, running the script weekly, and updating the VMs with the data in the object rather than looking for VMs with missing data and searching through all of the events. This article assumes that you are using a Windows system that is joined to AD on the same domain as your vCenter. It also assumes that you have loaded the Remote Server Administration Tools for Windows so that the Active Directory PowerShell modules are available. This is a separate download for Windows 7. The Active Directory Module for PowerShell can be enabled on Windows 7, Windows 8, Windows Server 2008, and Windows Server 2012 in the Programs and Features control panel under Turn Windows features on or off. Getting ready To begin this script, you will need the PowerShell ISE. How to do it… I order to set custom properties to add useful context to your virtual machines, perform the following steps: Open the PowerShell ISE and run Add-PSSnapIn VMware.VimAutomation.Core to initialize a PowerCLI session within the ISE. The first step is to create a custom attribute in vCenter for the CreatedBy and CreateDate attributes: New-CustomAttribute -TargetType VirtualMachine -Name CreatedBy New-CustomAttribute -TargetType VirtualMachine -Name CreateDate Before you begin the scripting, you will need to run ImportSystemModules to bring in the Active Directory cmdlets that you will use later to lookup the username and reference it back to a display name: ImportSystemModules Next, you need to locate and pull out all of the creation events with the same code as the Reporting VM objects created during a predefined time period from VI Events object topic. You will assign the events to a variable for processing in a loop in this case; however, you will also want to change the period to 1 week (7 days) instead of 1 month: $Events = Get-VIEvent -Start (Get-Date).AddDays(-7) -MaxSamples25000 | Where {$_.GetType().Name -match "(VmCreatedEvent|VmClonedEvent)"} The next step is to begin a ForEach loop to pull the data and populate it into a custom attribute: ForEach ($Event in $Events) { The first thing to do in the loop is to look up the VM referenced in the Event's Vm parameter by name using Get-VM: $VM = Get-VM -Name $Event.Vm.Name Next, you can use the CreatedTime parameter on the event and set this as a custom attribute on the VM using the Set-Annotation cmdlet: $VM | Set-Annotation -CustomAttribute "CreateDate" -Value $Event.CreatedTime Next, you can use the Username parameter to lookup the display name of the user account who created the VM using Active Directory cmdlets. For the Active Directory cmdlets to be available, your client system or server needs to have the Microsoft Remote Server Administration Tools (RSAT) installed to make the Active Directory cmdlets available. The data coming from $Event.Username is in DOMAINusername format. You need just the username to perform a lookup with Get-AdUser, so that you can split on the backslash and return only the second item in the array resulting from the split command. After the lookup, the display name that you will want to use is in the Name property. You can retrieve it with dot notation: $User = (($Event.UserName.split(""))[1]) $DisplayName = (Get-AdUser $User).Name To do this, you need to use a built-in on the event and set this as a custom attribute on the VM using the Set-Annotation cmdlet: $VM | Set-Annotation -CustomAttribute "CreatedBy" -Value $DisplayName Finally, close the ForEach loop. } <# End ForEach #> How it works… This topic works by leveraging the Get-VIEvent cmdlet to search for events in the log from the last number of days. In larger environments, you might need to expand the -MaxSamples cmdlet well beyond the number in this example. There might be thousands of events per day in larger environments. The topic looks through the log and the Where statement returns only the creation events. Once you have the object with all of the creation events, you can loop through this and pull out the username of the person who created each virtual machine and the time they were created. Then, you just need to populate the data into the custom attributes created. There's more… Combine this script with the next topic and you have a great solution for scheduling this routine to run on a daily basis. Running it daily would certainly cut down on the number of events you need to process through to find and update the virtual machines that have been created with the information. You should absolutely go and read Alan Renouf's blog post on which this topic is based. This primary difference between this topic and the one Alan presents is the use of native Windows Active Directory PowerShell lookups in this topic instead of the Quest Active Directory PowerShell cmdlets. See also Virtu-Al.net: Who created that VM? is available at http://www.virtu-al.net/2010/02/23/who-created-that-vm/ Using PowerShell native capabilities to schedule scripts There is potentially a better and easier way to schedule your processes to run from PowerShell and PowerCLI and those are known as Scheduled Jobs. Scheduled Jobs were introduced in PowerShell 3.0 and distributed as part of the Windows Management Framework 3.0 and higher. While Scheduled Tasks can execute any Windows batch file or executable, Scheduled Jobs are specific to PowerShell and are used to generate and create background jobs that run once or on a specified schedule. Scheduled Jobs appear in the Windows Task Scheduler and can be managed with the scheduled task cmdlets of PowerShell. The only difference is that the scheduled jobs cmdlets cannot manage scheduled tasks. These jobs are stored in the MicrosoftWindowsPowerShellScheduledJobs path of the Windows Task Scheduler. You can see and edit them through the management console in Windows after creation. What's even greater about Scheduled Jobs in PowerShell is that you are not forced into creating a .ps1 file for every new job you need to run. If you have a PowerCLI one-liner that provides all of the functionality you need, you can simply include it in a job creation cmdlet without ever needing to save it anywhere. Getting ready To being this topic, you will need a PowerCLI window with an active connection to a vCenter server. How to do it… In order to schedule scripts using the native capabilities of PowerShell, perform the following steps: If you are running PowerCLI on systems lower than Windows 8 or Windows Server 2012, there's a chance that you are running PowerShell 2.0 and you will need to upgrade in order to use this. To check, run Get-PSVersion to see which version is installed on your system. If less than version 3.0, upgrade before continuing this topic. Throw back a script you have already written, the script to find and remove snapshots over 30 days old: Get-Snapshot -VM * | Where {$_.Created -LT (Get-Date).AddDays(-30)} | Remove-Snapshot -Confirm:$false To schedule a new job, the first thing you need to think about is what triggers your job to run. To define a new trigger, you use the New-JobTrigger cmdlet: $WeeklySundayAt6AM = New-JobTrigger -Weekly -At "6:00 AM" -DaysOfWeek Sunday –WeeksInterval 1 Like scheduled tasks, there are some options that can be set for a scheduled job. These include whether to wake the system to run: $Options = New-ScheduledJobOption –WakeToRun –StartIfIdle –MultipleInstancePolicy Queue Next, you will use the Register-ScheduledJob cmdlet. This cmdlet accepts a parameter named ScriptBlock and this is where you will specify the script that you have written. This method works best with one-liners, or scripts that execute in a single line of piped cmdlets. Since this is PowerCLI and not just PowerShell, you will need to add the VMware cmdlets and connect to vCenter at the beginning of the script block. You also need to specify the -Trigger and -ScheduledJobOption parameters that are defined in the previous two steps: Register-ScheduledJob -Name "Cleanup 30 Day Snapshots"-ScriptBlock { Add-PSSnapIn VMware.VimAutomation.Core; Connect-VIServer servers; Get-Snapshot -VM * | Where {$_.Created -LT (Get-Date).AddDays(-30)} | Remove-Snapshot -Confirm:$false} -Trigger$WeeklySundayAt6AM-ScheduledJobOption $Options You are not restricted to only running a script block. If you have a routine in a .ps1 file, you can easily run it from ScheduledJob also. For illustration, if you have a .ps1 file stored in c:Scripts named 30DaySnaps.ps1, you can use the following cmdlet to register a job: Register-ScheduledJob -Name "Cleanup 30 Day Snapshots"–FilePath c:Scripts30DaySnaps.ps1 -Trigger $WeeklySundayAt6AM-ScheduledJobOption $Options Rather than scheduling the scheduled job and defining the PowerShell in the job, a more maintainable method can be to write the module and then call the function from the scheduled job. One other requirement is that Single Sign-On should be configured so that the Connect-VIServer works correctly in the script: Register-ScheduledJob -Name "Cleanup 30 Day Snapshots"-ScriptBlock {Add-PSSnapIn VMware.VimAutomation.Core; Connect-ViServer server; Import-Module 30DaySnaps; Remove-30DaySnaps -VM*} - How it works… This topic leverages the scheduled jobs framework developed specifically for running PowerShell as scheduled tasks. It doesn't require you to configure all of the extra settings as you have seen in previous examples of scheduled tasks. These are PowerShell native cmdlets that know how to implement PowerShell on a schedule. One thing to keep in mind is that these jobs will begin with a normal PowerShell session—one that knows nothing about PowerCLI, by default. You will need to include Add-PSSnapIn VMware.VimAutomation.Core in each script block or the .ps1 file that you use with a scheduled job. There's more… There is a full library of cmdlets to implement and maintain scheduled jobs. You have Set-ScheduleJob that allows you to change the settings of a registered scheduled job on a Windows system. You can disable and enable scheduled jobs using the Disable-ScheduledJob and Enable-Scheduled job cmdlets. This allows you to pause the execution of a job during maintenance, or for other reasons, without needing to remove and resetup the job. This is especially helpful since the script blocks are inside the job and not saved in a separate .ps1 file. You can also configure remote scheduled jobs on other systems using the Invoke-Command PowerShell cmdlet. This concept is shown in examples on Microsoft TechNet in the documentation for the Register-ScheduledJob cmdlet. In addition to scheduling new jobs, you can remove jobs using the Unregister-ScheduledJob cmdlet. This cmdlet requires one of three identifying properties to unschedule a job. You can pass -Name with a string, -ID with the number identifying the job, or an object reference to the scheduled job with -InputObject. You can combine the Get-ScheduledJob cmdlet to find and pass the object by pipeline. See also To read more about Microsoft TechNet: PSScheduledJob Cmdlets, refer to http://technet.microsoft.com/en-us/library/hh849778.aspx Summary This article was all about leveraging the information and data from PowerCLI as well as how we can format and display this information. Resources for Article: Further resources on this subject: Introduction to vSphere Distributed switches [article] Creating an Image Profile by cloning an existing one [article] VMware View 5 Desktop Virtualization [article]
Read more
  • 0
  • 0
  • 8874
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at ₹800/month. Cancel anytime
article-image-posting-your-wordpress-blog
Packt
16 Oct 2009
12 min read
Save for later

Posting on Your WordPress Blog

Packt
16 Oct 2009
12 min read
The central activity you'll be doing with your blog is adding posts. A post is like an article in a magazine; it's got a title, content, and an author (you). If a blog is like an online diary, then every post is an entry in that diary. A blog post also has a lot of other information attached to it, such as a date and categories. In this article, you will learn how to create a new post and what kind of information you can attach to it. Adding a simple post Let's review the process of adding a simple post to your blog. Whenever you want to do maintenance on your WordPress website, you have to start by logging in to the WP Admin (WordPress Administration panel) for your site. To get to the admin panel, just point your web browser to http://yoursite.com/wp-admin. Remember that if you have installed WordPress in a subfolder (for example, blog), then your URL has to include the subfolder (that is, http://yoursite.com/blog/wp-admin). When you first log into the WP Admin, you'll be at the Dashboard. The Dashboard has a lot of information on it. The very top bar, which I'll refer to as the top menu, is mostly dark grey and on the left, of course, is the main menu. The top menu and the main menu exist on every page within the WP Admin. The main section on the right contains information for the current page you're on. In this case, we're on the Dashboard. It contains boxes that have a variety of information about your blog, and about WordPress in general. The quickest way to get to the Add New Post page at any time is to click on the New Post link at the top of the page in the top bar (top menu). This is the Add New Post page: To quickly add a new post to your site, all you have to do is: Type in a title into the text field under Add New Post (for example, Making Lasagne). Type the text of your post in the content box. Note that the default view is Visual, but you actually have a choice of the HTML view as well. Click on the Publish button, which is at the far right. Note that you can choose to save a draft or view a preview of your post. In the following image, the title field, the content box, and the Publish button of the Add New Post page are highlighted: Once you click on the Publish button, you have to wait while WordPress performs its magic. You'll see yourself still on the Edit Post page, but now the following message has appeared telling you that your post was published and giving you a link to View post: If you go to the front page of your site, you'll see that your new post has been added at the top (newest posts are always at the top): Common post options Now that we've reviewed the basics of adding a post, let's investigate some of the other options on the Add New Post page. In this section we'll look at the most commonly used options, and in the next section we'll look at the more advanced options. Categories and tags Categories and tags are two similar types of information that you can add to a blog post. We use them to organize the information in your blog by topic and content (rather than just by, say, date), and to help visitors find what they are looking for on your blog. Categories are primarily used for structural organizing. They can be hierarchical. A relatively busy blog will probably have at least 10 categories, but probably not more than 15 or 20. Each post in this blog will likely have one to four categories assigned to it. For example, a blog about food might have these categories: Cooking Adventures, In The Media, Ingredients, Opinion, Recipes Found, Recipes Invented, and Restaurants. Tags are primarily used as shorthand for describing the topics covered in a particular blog post. A relatively busy blog will have anywhere from 15 to 30 tags in use. Each post in this blog will likely have three to ten tags assigned to it. For example, a post on the food blog about a recipe for butternut squash soup may have these tags: soup, vegetarian, autumn, hot, easy. Let's add a new post to the blog. This time, we'll give it not only a title and content, but also tags and categories. When adding tags, just type your list of tags into the Tags box on the right, separated by commas: Then click on the Add button. The tags you just typed in will appear below the text field with little xs next to them. You can click on an x to delete a tag. Once you've used some tags in your blog, you'll be able to click on the Choose from the most popular tags link in this box so that you can easily re-use tags. Categories work a bit differently than tags. Once you get your blog going, you'll usually just check the boxes next to existing categories in the Categories box. In this case, as we don't have any existing categories, we'll have to add one or two. In the Categories box on the right, click on the + Add New Category link. Type your category into the text field and click on the Add button. Your new category will show up in the list, already checked. Look at the following screenshot: If in the future you want to add a category that needs a parent category, select Parent category from the pull-down menu before clicking on the Add button. If you want to manage more details about your categories, move them around, rename them, assign parent categories, and assign descriptive text. You can do this on the Categories page, which we'll see in detail later in this article. Now fill in your title and content here: Click on the Publish button and you're done. When you look at the front page of your site, you'll see your new post on the top, your new category in the sidebar, and the tags and category (that you chose for your post) listed under the post itself: Adding an image to a post You may often want to have an image show up in your post. WordPress makes this very easy. Let's add an image to the post we just created. You can click on Edit underneath your post on the front page of your site to get there quickly. Alternatively, go back to the WP Admin, open Posts in the main menu, and then click on Edit underneath your new post. To add an image to a post, first you'll need to have that image on your computer. Before you get ready to upload an image, make sure that your image is optimized for the Web. Huge files will be uploaded slowly and slow down the process of viewing your site. You can re-size and optimize images using software such as GIMP or Photoshop. For the example in this article, I have used a photo of butternut squash soup that I have taken from the website where I got the recipe, and I know it's on the desktop of my computer. Once you have a picture on your computer and know where it is, follow these steps to add the photo to your blog post: Click on the little photo icon, which is next to the word Upload/Insert and below the box for the title: In the box that appears, click on the Select Files button and browse to your image. Then click on Open and watch the uploader bar. When it's done, you'll have a number of fields you can fill in: The only fields that are important right now are Title, Alignment, and Size. Title is a description for the image, Alignment will tell the image whether to have text wrap around it, and Size is the size of the image. As you can see, I've chosen the Right alignment and the Thumbnail size. Now click on Insert into Post. This box will disappear, and your image will show up in the post on the edit page itself: Now click on the Update Post button and go look at the front page of your site again. There's your image! You may be wondering about those image sizes. What if you want bigger or smaller thumbnails? You can set the pixel dimensions of your uploaded images and other preferences by opening Settings in the main menu and then clicking on Media. This takes you to the Media Settings page: Here you can specify the size of the uploaded images for: Thumbnail Medium Large If you change the dimensions on this page and click on the Save Changes button, only images you upload in the future will be affected. Images you've already uploaded to the site will have had their thumbnail, medium, and large versions created already using the old dimensions. Using the Visual editor versus the HTML editor WordPress comes with a Visual editor, otherwise known as a WYSIWYG editor (pronounced wissy-wig, which stands for What You See Is What You Get). This is the default editor for typing and editing your posts. If you're comfortable with HTML, you may prefer to write and edit your posts using the HTML editor—particularly useful if you want to add special content or styling. To switch from the rich text editor to the HTML editor, click on the HTML tab next to the Visual tab at the top of the content box: You'll see your post in all its raw HTML glory and you'll get a new set of buttons that lets you quickly bold and italicize text as well as add link code, image code, and so on. You can make changes and swap back and forth between the tabs to see the result. If you want the HTML tab to be your default editor, you can change this on your Profile page. Navigate to Users | Your Profile, and select the Disable the visual editor when writing checkbox. Drafts, timestamps, and managing posts There are three additional, simple but common, items I'd like to cover in this section: drafts, timestamps, and managing posts. Drafts WordPress gives you the option to save a draft of your post so that you don't have to publish it right away but can still save your work. If you've started writing a post and want to save a draft, just click on the Save Draft button at the right (in the Publish box), instead of the Publish button. Even if you don't click on the Save Draft button, WordPress will attempt to save a draft of your post for you about once a minute. You'll see this in the area just below the content box. The text will say Saving Draft... and then the time of the last draft saved: At this point, after a manual save or an auto-save, you can leave the Edit Post page and do other things. You'll be able to access all of your draft posts from the Dashboard or from the Edit Posts page. Timestamps WordPress will also let you alter the timestamp of your post. This is useful if you are writing a post today that you wish you'd published yesterday, or if you're writing a post in advance and don't want it to show up until the right day. The default timestamp will always be set to the moment you publish your post. To change it, just find the Publish box and click on the Edit link (next to the calendar icon and Publish immediately), and fields will show up with the current date and time for you to change: Change the details, click on the OK button, and then Publish your post (or save a draft). Managing posts If you want to see a list of your posts so that you can easily skim and manage them, you just need to go to the Edit Posts page in the WP Admin by navigating to Posts in the main menu. You'll see a detailed list of your posts, as seen in the next screenshot: There are so many things you can do on this page! You can: Choose a post to edit—click on a post title and you'll go back to the main Edit Post page Quick-edit a post—click on the Quick Edit link for any post and new options will appear right in the list, which will let you edit the title, timestamp, categories, tags, and more Delete one or more posts—click on the checkboxes next to the posts you want to delete, choose Delete from the Bulk Actions drop-down menu at the bottom, and click on the Apply button Bulk edit posts—choose Edit from the Bulk Actions menu at the bottom, click on the Apply button, and you'll be able to assign categories and tags to multiple posts, as well as edit other information about them You can experiment with the other links and options on this page. Just click on the pull-down menus and links, and see what happens.
Read more
  • 0
  • 0
  • 8869

article-image-overview-chips
Packt
16 Dec 2014
7 min read
Save for later

Overview of Chips

Packt
16 Dec 2014
7 min read
In this article by Olliver M. Schinagl, author of Getting Started with Cubieboard, we will overview various development boards and compare a few popular ones to help you choose a board tailored to your requirements. In the last few years, ARM-based Systems on Chips (SoCs) have become immensely popular. Compared to the regular x86 Intel-based or AMD-based CPUs, they are much more energy efficient and still performs adequately. They also incorporate a lot of peripherals, such as a Graphics Processor Unit (GPU), a Video Accelerator (VPU), an audio controller, various storage controllers, and various buses (I2C and SPI), to name a few. This immensely reduces the required components on a board. With the reduction in the required components, there are a few obvious advantages, such as reduction in the cost and, consequentially, a much easier design of boards. Thus, many companies with electronic engineers are able to design and manufacture these boards cheaply. (For more resources related to this topic, see here.) So, there are many boards; does that mean there are also many SoCs? Quite a few actually, but to keep the following list short, only the most popular ones are listed: Allwinner's A-series Broadcom's BCM-series Freescale's i.MX-series MediaTek's MT-series Rockchip's RK-series Samsung's Exynos-series NVIDIA's Tegra-series Texas Instruments' AM-series and OMAP-series Qualcomm's APQ-series and MSM-series While many of the potential chips are interesting, Allwinner's A-series of SoCs will be the focus of this book. Due to their low price and decent availability, quite a few companies design development boards around these chips and sell them at a low cost. Additionally, the A-series is, presently, the most open source friendly series of chips available. There is a fully open source bootloader, and nearly all the hardware is supported by open source drivers. Among the A-series of chips, there are a few choices. The following is the list of the most common and most interesting devices: A10: This is the first chip of the A-series and the best supported one, as it has been around for long. It is able to communicate with the outside world over I2C, SPI, MMC, NAND, digital and analog video out, analog audio out, SPDIF, I2S, Ethernet MAC, USB, SATA, and HDMI. This chip initially targeted everything, such as phones, tablets, set-top boxes, and mini PC sticks. For its GPU, it features the MALI-400. A10S: This chip followed the A10; it focused mainly on the PC stick market and left out several parts, such as SATA and analog video in/out, and it has no LCD interface. These parts were left out to reduce the cost of the chip, making it interesting for cheap TV sticks. A13: This chip was introduced more or less simultaneously with the A10S for primary use in tablets. It lacked SATA, Ethernet MAC, and also HDMI, which reduced the chip's cost even more. A20: This chip was introduced way after the others and hence it was pin-compatible to the A10 intended to replace it. As the name hints, the A20 is a dual-core variant of the A10. The ARM cores are slightly different; Cortex-A7 has been used in the A10 instead of Cortex-A8. A23: This chip was introduced after the A31 and A31S and is reasonably similar to the A31 in its design. It features a dual-core Cortex-A7 design and is intended to replace the A13. It is mainly intended to be used in tablets. A31: This chip features four Cortex-A7 cores and generally has all the connections that the A10 has. It is, however, not popular within the community because it features a PowerVR GPU that, until now, has seen no community support at all. Additionally, there are no development boards commonly available for this chip. A31S: This chip was released slightly after the A31 to solve some issues with the A31. There are no common development boards available. Choosing the right development board Allwinner's A-series of SoCs was produced and sold so cheaply that many companies used these chips in their products, such as tablets, set-top boxes, and eventually, development boards. Before the availability of development boards, people worked on and with tablets and set-top boxes. The most common and popular boards are from Cubietech and Olimex, in part because both companies handed out development boards to community developers for free. Olimex Olimex has released a fair amount of different development boards and peripherals. A lot of its boards are open source hardware with schematics and layout files available, and Olimex is also very open source friendly. You can see the Olimex board in the following image: Olimex offers the A10-OLinuXino-LIME, an A10-based micro board that is marketed to compete with the famous Raspberry Pi price-wise. Due to its small size, it uses less standard, 1.27 mm pitch headers for the pins, but it has nearly all of these pins exposed for use. You can see the A10-OLinuXino-LIME board in the following image: The Olimex OLinuXino series of boards is available in the A10, A13, and A20 flavors and has more standard, 2.54 mm pitch headers that are compatible with the old IDE and serial connectors. Olimex has various sensors, displays, and other peripherals that are also compatible with these headers. Cubietech Cubietech was formed by previous Allwinner employees and was one of the first development boards available using the Allwinner SoC. While it is not open source hardware, it does offer the schematics for download. Cubietech released three boards: the Cubieboard1, the Cubieboard2, and the Cubieboard3—also known as the Cubietruck. Interfacing with these boards can be quite tricky, as they use 2 mm pitch headers that might be hard to find in Europe or America. You can see the Cubietech board in the following image: Cubieboard1 and Cubieboard2 use identical boards; the only difference is that A20 is used instead of A10 in Cubieboard2. These boards only have a subset of the pins exposed. You can see the Cubietruck board in the following image: Cubietruck is quite different but a well-designed A20 board. It features everything that the previous boards offer, along with Gigabit Ethernet, VGA, Bluetooth, Wi-Fi, and an optical audio out. This does come at the cost that there are fewer pins to keep the size reasonably small. Compared to Raspberry Pi or LIME, it is almost double the size. Lemaker Lemaker made a smart design choice when releasing its Banana Pi board. It is an Allwinner A20-based board but uses the same board size and connector placement as Raspberry Pi and hence the name Banana Pi. Because of this, many of those Raspberry Pi cases could fit the Banana Pi and even shields will fit. Software-wise, it is quite different and does not work when using Raspberry Pi image files. Nevertheless, it features composite video out, stereo audio out, HDMI out Gigabit Ethernet, two USB ports, one USB OtG port, CSI out and LVDS out, and a handful of pins. Also available are a LiPo battery connector and a SATA connector and two buttons, but those might not be accessible on a lot of standard cases. See the following image for the topside of the Banana Pi: Itead and Olimex Itead and Olimex both offer an interesting board, which is worth mentioning separately. The Iteaduino Plus and the Olimex A20-SoM are quite interesting concepts; the computing module, which is a board with the SoC, memory, and flash, which are plugin modules, and a separated baseboard. Both of them sell a very complete baseboard as open source hardware, but anybody can design their own baseboard and buy the computing module. You can see the following board by Itead: Refer to the following board by Olimex: Additional hardware While a development board is a key ingredient, there are several other items that are also required. A power supply, for example, is not always supplied and does have some considerations. Also, additional hardware is required for the initial communication and to debug. Summary In this article, you looked at the additional hardware and a few extra peripherals that will help you understand the stuff you require for your projects. Resources for Article: Further resources on this subject: Home Security by BeagleBone [article] Mobile Devices [article] Making the Unit Very Mobile – Controlling the Movement of a Robot with Legs [article]
Read more
  • 0
  • 0
  • 8868

article-image-scaling-friendly-font-rendering-distance-fields
Packt
28 Oct 2014
8 min read
Save for later

Scaling friendly font rendering with distance fields

Packt
28 Oct 2014
8 min read
This article by David Saltares Márquez and Alberto Cejas Sánchez, the authors of Libgdx Cross-platform Game Development Cookbook, describes how we can generate a distance field font and render it in Libgdx. As a bitmap font is scaled up, it becomes blurry due to linear interpolation. It is possible to tell the underlying texture to use the nearest filter, but the result will be pixelated. Additionally, until now, if you wanted big and small pieces of text using the same font, you would have had to export it twice at different sizes. The output texture gets bigger rather quickly, and this is a memory problem. (For more resources related to this topic, see here.) Distance field fonts is a technique that enables us to scale monochromatic textures without losing out on quality, which is pretty amazing. It was first published by Valve (Half Life, Team Fortress…) in 2007. It involves an offline preprocessing step and a very simple fragment shader when rendering, but the results are great and there is very little performance penalty. You also get to use smaller textures! In this article, we will cover the entire process of how to generate a distance field font and how to render it in Libgdx. Getting ready For this, we will load the data/fonts/oswald-distance.fnt and data/fonts/oswald.fnt files. To generate the fonts, Hiero is needed, so download the latest Libgdx package from http://libgdx.badlogicgames.com/releases and unzip it. Make sure the samples projects are in your workspace. Please visit the link https://github.com/siondream/libgdx-cookbook to download the sample projects which you will need. How to do it… First, we need to generate a distance field font with Hiero. Then, a special fragment shader is required to finally render scaling-friendly text in Libgdx. Generating distance field fonts with Hiero Open up Hiero from the command line. Linux and Mac users only need to replace semicolons with colons and back slashes with forward slashes: java -cp gdx.jar;gdx-natives.jar;gdx-backend-lwjgl.jar;gdx-backend-lwjgl-natives.jar;extensions gdx-toolsgdx-tools.jar com.badlogic.gdx.tools.hiero.Hiero Select the font using either the System or File options. This time, you don't need a really big size; the point is to generate a small texture and still be able to render text at high resolutions, maintaining quality. We have chosen 32 this time. Remove the Color effect, and add a white Distance field effect. Set the Spread effect; the thicker the font, the bigger should be this value. For Oswald, 4.0 seems to be a sweet spot. To cater to the spread, you need to set a matching padding. Since this will make the characters render further apart, you need to counterbalance this by the setting the X and Y values to twice the negative padding. Finally, set the Scale to be the same as the font size. Hiero will struggle to render the charset, which is why we wait until the end to set this property. Generate the font by going to File | Save BMFont files (text).... The following is the Hiero UI showing a font texture with a Distance field effect applied to it: Distance field fonts shader We cannot use the distance field texture to render text for obvious reasons—it is blurry! A special shader is needed to get the information from the distance field and transform it into the final, smoothed result. The vertex shader found in data/fonts/font.vert is simple. The magic takes place in the fragment shader, found in data/fonts/font.frag and explained later. First, we sample the alpha value from the texture for the current fragment and call it distance. Then, we use the smoothstep() function to obtain the actual fragment alpha. If distance is between 0.5-smoothing and 0.5+smoothing, Hermite interpolation will be used. If the distance is greater than 0.5+smoothing, the function returns 1.0, and if the distance is smaller than 0.5-smoothing, it will return 0.0. The code is as follows: #ifdef GL_ES precision mediump float; precision mediump int; #endif   uniform sampler2D u_texture;   varying vec4 v_color; varying vec2 v_texCoord;   const float smoothing = 1.0/128.0;   void main() {    float distance = texture2D(u_texture, v_texCoord).a;    float alpha = smoothstep(0.5 - smoothing, 0.5 + smoothing, distance);    gl_FragColor = vec4(v_color.rgb, alpha * v_color.a); } The smoothing constant determines how hard or soft the edges of the font will be. Feel free to play around with the value and render fonts at different sizes to see the results. You could also make it uniform and configure it from the code. Rendering distance field fonts in Libgdx Let's move on to DistanceFieldFontSample.java, where we have two BitmapFont instances: normalFont (pointing to data/fonts/oswald.fnt) and distanceShader (pointing to data/fonts/oswald-distance.fnt). This will help us illustrate the difference between the two approaches. Additionally, we have a ShaderProgram instance for our previously defined shader. In the create() method, we instantiate both the fonts and shader normally: normalFont = new BitmapFont(Gdx.files.internal("data/fonts/oswald.fnt")); normalFont.setColor(0.0f, 0.56f, 1.0f, 1.0f); normalFont.setScale(4.5f);   distanceFont = new BitmapFont(Gdx.files.internal("data/fonts/oswald-distance.fnt")); distanceFont.setColor(0.0f, 0.56f, 1.0f, 1.0f); distanceFont.setScale(4.5f);   fontShader = new ShaderProgram(Gdx.files.internal("data/fonts/font.vert"), Gdx.files.internal("data/fonts/font.frag"));   if (!fontShader.isCompiled()) {    Gdx.app.error(DistanceFieldFontSample.class.getSimpleName(), "Shader compilation failed:n" + fontShader.getLog()); } We need to make sure that the texture our distanceFont just loaded is using linear filtering: distanceFont.getRegion().getTexture().setFilter(TextureFilter.Linear, TextureFilter.Linear); Remember to free up resources in the dispose() method, and let's get on with render(). First, we render some text with the regular font using the default shader, and right after this, we do the same with the distance field font using our awesome shader: batch.begin(); batch.setShader(null); normalFont.draw(batch, "Distance field fonts!", 20.0f, VIRTUAL_HEIGHT - 50.0f);   batch.setShader(fontShader); distanceFont.draw(batch, "Distance field fonts!", 20.0f, VIRTUAL_HEIGHT - 250.0f); batch.end(); The results are pretty obvious; it is a huge win of memory and quality over a very small price of GPU time. Try increasing the font size even more and be amazed at the results! You might have to slightly tweak the smoothing constant in the shader code though: How it works… Let's explain the fundamentals behind this technique. However, for a thorough explanation, we recommend that you read the original paper by Chris Green from Valve (http://www.valvesoftware.com/publications/2007/SIGGRAPH2007_AlphaTestedMagnification.pdf). A distance field is a derived representation of a monochromatic texture. For each pixel in the output, the generator determines whether the corresponding one in the original is colored or not. Then, it examines its neighborhood to determine the 2D distance in pixels, to a pixel with the opposite state. Once the distance is calculated, it is mapped to a [0, 1] range, with 0 being the maximum negative distance and 1 being the maximum positive distance. A value of 0.5 indicates the exact edge of the shape. The following figure illustrates this process: Within Libgdx, the BitmapFont class uses SpriteBatch to render text normally, only this time, it is using a texture with a Distance field effect applied to it. The fragment shader is responsible for performing a smoothing pass. If the alpha value for this fragment is higher than 0.5, it can be considered as in; it will be out in any other case: This produces a clean result. There's more… We have applied distance fields to text, but we have also mentioned that it can work with monochromatic images. It is simple; you need to generate a low resolution distance field transform. Luckily enough, Libgdx comes with a tool that does just this. Open a command-line window, access your Libgdx package folder and enter the following command: java -cp gdx.jar;gdx-natives.jar;gdx-backend-lwjgl.jar;gdx-backend-lwjgl-natives.jar;extensionsgdx-tools gdx-tools.jar com.badlogic.gdx.tools.distancefield.DistanceFieldGenerator The distance field font generator takes the following parameters: --color: This parameter is in hexadecimal RGB format; the default is ffffff --downscale: This is the factor by which the original texture will be downscaled --spread: This is the edge scan distance, expressed in terms of the input Take a look at this example: java […] DistanceFieldGenerator --color ff0000 --downscale 32 --spread 128 texture.png texture-distance.png Alternatively, you can use the gdx-smart-font library to handle scaling. It is a simpler but a bit more limited solution (https://github.com/jrenner/gdx-smart-font). Summary In this article, we have covered the entire process of how to generate a distance field font and how to render it in Libgdx. Further resources on this subject: Cross-platform Development - Build Once, Deploy Anywhere [Article] Getting into the Store [Article] Adding Animations [Article]
Read more
  • 0
  • 0
  • 8868

article-image-optimizing-magento-performance-using-hhvm
Packt
16 May 2014
5 min read
Save for later

Optimizing Magento Performance — Using HHVM

Packt
16 May 2014
5 min read
(For more resources related to this topic, see here.) HipHop Virtual Machine As we can write a whole book (or two) about HHVM, we will just give the key ideas here. HHVM is a virtual machine that will translate any called PHP file into a HHVM byte code in the same spirit as the Java or .NET virtual machine. HHVM transforms your PHP code into a lower level language that is much faster to execute. Of course, the transformation time (compiling) does cost a lot of resources, therefore, HHVM is shipped with a cache mechanism similar to APC. This way, the compiled PHP files are stored and reused when the original file is requested. With HHVM, you keep the PHP flexibility and ease in writing, but you now have a performance like that of C++. Hear the words of the HHVM team at Facebook: "HHVM (aka the HipHop Virtual Machine) is a new open-source virtual machine designed for executing programs written in PHP. HHVM uses a just-in-time compilation approach to achieve superior performance while maintaining the flexibility that PHP developers are accustomed to. To date, HHVM (and its predecessor HPHPc) has realized over a 9x increase in web request throughput and over a 5x reduction in memory consumption for Facebook compared with the Zend PHP 5.2 engine + APC. HHVM can be run as a standalone webserver (in other words, without the Apache webserver and the "modphp" extension). HHVM can also be used together with a FastCGI-based webserver, and work is in progress to make HHVM work smoothly with Apache." If you think this is too good to be true, you're right! Indeed, HHVM have a major drawback. HHVM was and still is focused on the needs of Facebook. Therefore, you might have a bad time trying to use your custom made PHP applications inside it. Nevertheless, this opportunity to speed up large PHP applications has been seen by talented developers who improve it, day after day, in order to support more and more framework. As our interest is in Magento, I will introduce you Daniel Sloof who is a developer from Netherland. More interestingly, Daniel has done (and still does) an amazing work at adapting HHVM for Magento. Here are the commands to install Daniel Sloof's version of HHVM for Magento: $ sudo apt-get install git $ git clone https://github.com/danslo/hhvm.git $ sudo chmod +x configure_ubuntu_12.04.sh $ sudo ./configure_ubuntu_12.04.sh $ sudo CMAKE_PREFIX_PATH=`pwd`/.. make If you thought that the first step was long, you will be astonished by the time required to actually build HHVM. Nevertheless, the wait is definitely worth it. The following screenshot shows how your terminal will look for the next hour or so: Create a file named hhvm.hdf under /etc/hhvm and write the following code inside: Server { Port = 80 SourceRoot = /var/www/_MAGENTO_HOME_ } Eval { Jit = true } Log { Level = Error UseLogFile = true File = /var/log/hhvm/error.log Access { * { File = /var/log/hhvm/access.log Format = %h %l %u %t \"%r\" %>s %b } } } VirtualHost { * { Pattern = .* RewriteRules { dirindex { pattern = ^/(.*)/$ to = $1/index.php qsa = true } } } } StaticFile { FilesMatch { * { pattern = .*\.(dll|exe) headers { * = Content-Disposition: attachment } } } Extensions { css = text/css gif = image/gif html = text/html jpe = image/jpeg jpeg = image/jpeg jpg = image/jpeg png = image/png tif = image/tiff tiff = image/tiff txt = text/plain } } Now, run the following command: $ sudo ./hhvm –mode daemon –config /etc/hhvm.hdf The hhvm executable is under hhvm/hphp/hhvm. Is all of this worth it? Here's the response: ab -n 100 -c 5 http://192.168.0.105192.168.0.105/index.php/furniture/livingroom.html Server Software: Server Hostname: 192.168.0.105192.168.0.105 Server Port: 80 Document Path: /index.php/furniture/living-room.html Document Length: 35552 bytes Concurrency Level: 5 Time taken for tests: 4.970 seconds Requests per second: 20.12 [#/sec] (mean) Time per request: 248.498 [ms] (mean) Time per request: 49.700 [ms] (mean, across all concurrent requests) Transfer rate: 707.26 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 2 12.1 0 89 Processing: 107 243 55.9 243 428 Waiting: 107 242 55.9 242 427 Total: 110 245 56.7 243 428 We literally reach a whole new world here. Indeed, our Magento instance is six times faster than after all our previous optimizations and about 20 times faster than the default Magento served by Apache. The following graph shows the performances: Our Magento instance is now flying at lightening speed, but what are the drawbacks? Is it still as stable as before? All the optimization we did so far, are they still effective? Can we go even further? In what follows, we present a non-exhaustive list of answers: Fancy extensions and modules may (and will) trigger HHVM incompatibilities. Magento is a relatively old piece of software and combining it with a cutting edge technology such as HHVM can have some unpredictable (and undesirable) effects. HHVM is so complex that fixing a Magento-related bug requires a lot of skill and dedication. HHVM takes care of PHP, not of cache mechanisms or the accelerator we installed before. Therefore, APC, memcached, and Varnish are still running and helping to improve our performances. If you become addicted to performances, HHVM is now supporting Fast-CGI through Nginx and Apache. You can find out more about that at: http://www.hhvm.com/blog/1817/fastercgi-with-hhvm. Summary In this article, we successfully used the HipHop Virtual Machine (HHVM) from Facebook to serve Magento. This improvement optimizes our Magento performance incredibly (20 times faster), that is, the time required initially was 110 seconds while now it is less then 5 seconds. Resources for Article: Further resources on this subject: Magento: Exploring Themes [article] Getting Started with Magento Development [article] Enabling your new theme in Magento [article] Call Send SMS Add to Skype You'll need Skype CreditFree via Skype
Read more
  • 0
  • 0
  • 8858
article-image-running-your-applications-aws-part-2
Cheryl Adams
19 Aug 2016
6 min read
Save for later

Running Your Applications with AWS - Part 2

Cheryl Adams
19 Aug 2016
6 min read
An active account with AWS means you are on your way with building in the cloud.  Before you start building, you need to tackle the Billing and Cost Management, under Account. It is likely that you are starting with a Free-Tier, so it is important to know that you still have the option of paying for additional services. Also, if you decide to continue with AWS,you should get familiar with this page.  This is not your average bill or invoice page—it is much more than that. The Billing & Cost Management Dashboard is a bird’s-eye view of all of your account activity. Once you start accumulating pay-as-you-go services, this page will give you a quick review of your monthly spending based on services. Part of managing your cloud services includes billing, so it is a good idea to become familiar with this from the start. Amazon also gives you the option of setting up cost-based alerts for your system, which is essential if youwant to be alerted by any excessive cost related to your cloud services. Budgets allow you to receive e-mailed notifications or alerts if spending exceeds the budget that you have created.    If you want to dig in even deeper, try turning on the Cost Explorer for an analysis of your spending. The Billing and Cost Management section of your account is much more than just invoices. It is the AWS complete cost management system for your cloud. Being familiar with all aspects of the cost management system will help you to monitor your cloud services, and hopefully avoid any expenses that may exceed your budget. In our previous discussion, we considered all AWSservices.  Let’s take another look at the details of the services. Amazon Web Services Based on this illustration, you can see that the build options are grouped by words such asCompute, Storage & Content Delivery and  Databases.  Each of these objects or services lists a step-by-step routine that is easy to follow. Within the AWS site, there are numerous tutorials with detailed build instructions. If you are still exploring in the free-tier, AWS also has an active online community of users whotry to answer most questions. Let’s look at the build process for Amazon’s EC2 Virtual Server. The first thing that you will notice is that Amazon provides 22 different Amazon Machine Images (AMIs) to choose from (at the time this post was written).At the top of the screen is a Step process that will guide you through the build. It should be noted that some of the images available are not defined as a part of the free-tier plan. The remaining images that do fit into the plan should fit almost any project need. For this walkthrough, let’s select SUSE Linux (free eligible). It is important to note that just because the image itself is free, that does not mean all the options available within that image are free. Notice on this screen that Amazon has pre-selected the only free-tier option available for this image. From this screen you are given two options: (Review and Launch) or (Next Configure Instance Details).  Let’s try Review and Launch to see what occurs. Notice that our Step process advanced to Step 7. Amazon gives you a soft warning regarding the state of the build and potential risk. If you are okay with these risks, you can proceed and launch your server. It is important to note that the Amazon build process is user driven. It will allow you to build a server with these potential risks in your cloud. It is recommended that you carefully consider each screen before proceeding. In this instance,select Previous and not Cancel to return to Step 3. Selecting Cancelwill stop the build process and return you to the AWS main services page. Until you actually launch your server, nothing is built or saved. There are information bubbles for each line in Step 3: Configure Instance Details. Review the content of each bubble, make any changes if needed, and then proceed to the next step. Select the storage size; then select Next Tag Instance. Enter Values and Continue or Learn More for further information. Select the Next: Configure Security Group button. Security is an extremely important part of setting up your virtual server. It is recommended that you speak to your security administrator to determine the best option. For source, it is recommended that you avoid using the Anywhereoption. This selection will put your build at risk. Select my IP or custom IP as shown. If you are involved in a self-study plan, you can select the Learn More link to determine the best option. Next: Review and Launch The full details of this screen be expanded, reviewed or edited. If everything appears to be okay,proceed to Launch. One additional screen will appear for adding Private and/or Public Keys to access your new server. Make the appropriate selection and proceed to the Launch Instances. One more screen will appear for adding Private and/or Public Keys to access your new server. Make the appropriate selection and proceed to Launch Instances to see the build process. You can access your new server from the EC2 Dashboard. This example of a build process gives you a window into how the  AWS build process works. The other objects and services have a similar step-through process. Once you have launched your server, you should be able to access it and proceed with your development. Additional details for development are also available through the site. Amazon’s Web Services Platform is an all-in-one solution for your graduation to the cloud. Not only can you manage your technical environment, but also it has features that allow you to manage your budget. By setting up your virtual applicances and servers appropriately, you can maximize the value of the first  12 months of your free-tier. Carefully monitoring activities through alerts and notification will help you to avoid having any billing surprises. Going through the tutorials and visting the online community will only aid to increase your knowledge base of AWS. AWS is inviting everyone to test their services on this exciting platform, so I would definitely recommend taking advantage of it. Have fun! About the author Cheryl Adams is a senior cloud data andinfrastructure architect in the healthcare data realm. She is also the co-author of Professional Hadoop by Wrox.
Read more
  • 0
  • 0
  • 8857

article-image-calling-your-fellow-agents
Packt
04 Feb 2015
12 min read
Save for later

Calling your fellow agents

Packt
04 Feb 2015
12 min read
In this article by Stefan Sjogelid, author of book Raspberry Pi for Secret Agents Second Edition. We will be setting up SIP Witch by adding softphones, connect them together, and then we will run the softphone on the Pi. When you're out in the field and need to call in a favor from a fellow agent or report back to HQ, you don't want to depend on the public phone network if you can avoid it. Landlines and cell phones alike can be tapped by all sorts of shady characters and to add insult to injury, you have to pay good money for this service. We can do better. Welcome to the wonderful world of Voice over IP (VoIP). VoIP is a blanket term for any technology capable of delivering speech between two end users over IP networks. There are plenty of services and protocols out there that try to meet this demand, most of which force you to connect through a central server that you don't own or control. We're going to turn the Pi into the central server of our very own phone network. To aid us with this task, we'll deploy GNU SIP Witch—a peer-to-peer VoIP server that uses Session Initiation Protocol (SIP) to route calls between phones. While there are many excellent VoIP servers available (Asterisk, FreeSwitch, and Yate and so on) SIP Witch has the advantage of being very lightweight on the Pi because its only concern is connecting phones and not much else. (For more resources related to this topic, see here.) Setting up SIP Witch Once we have the SIP server up and running we'll be adding one or more software phones or softphones. It's assumed that server and phones will all be on the same network. Let's get started! Install SIP Witch using the following command: pi@raspberrypi ~ $ sudo apt-get install sipwitch Just as the output of the previous command says, we have to define PLUGINS in /etc/default/sipwitch before running SIP Witch. Let's open it up for editing: pi@raspberrypi ~ $ sudo nano /etc/default/sipwitch Find the line that reads #PLUGINS="zeroconf scripting subscriber forward" and remove the # character to uncomment the line. This directive tells SIP Witch that we want the standard plugins to be loaded. Next we'll have a look at the main SIP Witch configuration file: pi@raspberrypi ~ $ sudo nano /etc/sipwitch.conf Note how some blocks of text are between <!-- and --> tags. These are comments in XML documents and are ignored by SIP Witch. Whatever changes you want to make, ensure they go outside of those tags. Now we're going to add a few softphone user accounts. It's up to you how many phones you'd like on your system, but each account needs a username, an extension (short phone number) and a password. Find the <provision> tag, make a new line and add your users: <user id="phone1"> <extension>201</extension> <secret>SecretSauce201</secret> <display>Agent 201</display> </user> <user id="phone2"> <extension>202</extension> <secret>SecretSauce202</secret> <display>Agent 202</display> </user> The user ID will be used as a user/login name later from the softphones. In this default configuration, the extensions can be any number between 201 and 299. The secret is the password that will go together with the username on the softphones. We will look into a better way of storing passwords later in this chapter. Finally, the display string defines an identity to present to other phones when calling. One more thing that we need to configure is how SIP Witch should treat local names. This makes it possible to call a phone by user ID in addition to the extension. Find the <stack> tag, make a new line and add the following directive, but replace [IP address] with the IP address of your Pi: <localnames>[IP address]</localnames> Those are all the changes we need to make to the configuration at the moment. Basic SIP Witch configuration for two phones With our configuration in place, let's start up the SIP Witch service: pi@raspberrypi ~ $ sudo service sipwitch start The SIP Witch server runs in the background and only outputs to a log file viewable with this command: pi@raspberrypi ~ $ sudo cat /var/log/sipwitch.log Now we can use the sipwitch command to interact with the running service. Type sipwitch for a list of all possible commands. Here's a short list of particularly handy ones: Command Description sudo sipwitch dump Shows how the SIP Witch server is currently configured. sudo sipwitch registry Lists all currently registered softphones. sudo sipwitch calls Lists active calls. sudo sipwitch message [extension] "[text]" Sends a text message from the server to an extension. Perfect for sending status updates from the Pi through scripting. Connecting the softphones Running your own telecommunications service is kind of boring without actual phones to make use of it. Fortunately, there are softphone applications available for most common electronic devices out there. The configuration of these phones will be pretty much identical no matter which platform they're running on. This is the basic information that will always need to be specified when configuring your softphone application: User / Login name: phone1 or phone2 in our example configuration Password / Authentication: The user's secret in our configuration Server / Host name / Domain: The IP address of your Pi Once a softphone is successfully registered with the SIP Witch server, you should be able to see that phone listed using the sudo sipwitch registry command. What follows is a list of verified decent softphones that will get the job done. Windows (MicroSIP) MicroSIP is an open source softphone that also supports video calls. Visit http://www.microsip.org/downloads to obtain and install the latest version (MicroSIP-3.8.1.exe at the time of writing).   Configuring the MicroSIP softphone for Windows Right-click on either the status bar in the main application window or the system tray icon to bring up the menu that lets you access the Account settings. Mac OS X (Telephone) Telephone is a basic open source softphone that is easily installed through the Mac App store. Configuring the Telephone softphone for Mac OS X Linux (SFLphone) SFLphone is an open source softphone with packages available for all major distributions and client interfaces for both GNOME and KDE. Use your distribution's package manager to find and install the application. Configuring SFLphone GNOME client in Ubuntu Android (CSipSimple) CSipSimple is an excellent open source softphone available from the Google Play store. When adding your account, use the basic generic wizard. Configuring the CSipSimple softphone on Android iPhone/iPad (Linphone) Linphone is an open source softphone that is easily installed through the iPhone App store. Select I have already a SIP-account to go to the setup assistant. Configuring Linphone on the iPhone Running a softphone on the Pi It's always good to be able to reach your agents directly from HQ, that is, the Pi itself. Proving once again that anything can be done from the command line, we're going to install a softphone called Linphone that will make good use of your USB microphone. This new softphone obviously needs a user ID and password just like the others. We will take this opportunity to look at a better way of storing passwords in SIP Witch. Encrypting SIP Witch passwords Type sudo sipwitch dump to see how SIP Witch is currently configured. Find the accounts: section and note how there's already a user ID named pi with extension 200. This is the result of a SIP Witch feature that automatically assigns an extension number to certain Raspbian user accounts. You may also have noticed that the display string for the pi user looks empty. We can easily fix that by filling in the full name field for the Raspbian pi user account with the following command: pi@raspberrypi ~ $ sudo chfn -f "Agent HQ" pi Now restart the SIP Witch server with sudo service sipwitch restart and verify with sudo sipwitch dump that the display string has changed. So how do we set the password for this automatically added pi user? For the other accounts, we specified the password in clear text inside <secret> tags in /etc/sipwitch.conf. This is not the best solution from a security perspective if your Pi would happen to fall into the wrong hands. Therefore, SIP Witch supports specifying passwords in encrypted digest form. Use the following command to create an encrypted password for the pi user: pi@raspberrypi ~ $ sudo sippasswd pi We can then view the database of SIP passwords that SIP Witch knows about: pi@raspberrypi ~ $ sudo cat /var/lib/sipwitch/digests.db Now you can add digest passwords for your other SIP users as well and then delete all <secret> lines from /etc/sipwitch.conf to be completely free of clear text. Setting up Linphone With our pi user account up and ready to go, let's proceed to set up Linphone: Linphone does actually have a graphical user interface, but we'll specify that we want the command-line only client: pi@raspberrypi ~ $ sudo apt-get install linphone-nogtk Now we fire up the Linphone command-line client: pi@raspberrypi ~ $ linphonec You will immediately receive a warning that reads: Warning: Could not start udp transport on port 5060, maybe this port is already used. That is, in fact, exactly what is happening. The standard communication channel for the SIP protocol is UDP port 5060, and it's already in use by our SIP Witch server. Let's tell Linphone to use port 5062 with this command: linphonec> ports sip 5062 Next we'll want to set up our microphone. Use these three commands to list, show, and select what audio device to use for phone calls: linphonec> soundcard list linphonec> soundcard show linphonec> soundcard use [number] For the softphone to perform reasonably well on the Pi, we'll want to make adjustments to the list of codecs that Linphone will try to use. The job of a codec is to compress audio as much as possible while retaining high quality. This is a very CPU-intensive process, which is why we want to use the codec with the least amount of CPU load on the Pi, namely, PCMU or PCMA. Use the following command to list all currently supported codecs: linphonec> codec list Now use this command to disable all codecs that are not PCMU or PCMA: linphonec> codec disable [number] It's time to register our softphone to the SIP Witch server. Use the following command but replace [IP address] with the IP address of your Pi and [password] with the SIP password you set earlier for the pi user: linphonec> register sip:pi@[IP address] sip:[IP address] [password] That's all you need to start calling your fellow agents from the Pi itself. Type help to get a list of all commands that Linphone accepts. The basic commands are call [user id] to call someone, answer to pick up incoming calls and quit to exit Linphone. All the settings that you've made will be saved to ~/.linphonerc and loaded the next time you start linphonec. Playing files with Linphone Now that you know the Linphone basics, let's explore some interesting features not offered by most other softphones. At any time (except during a call), you can switch Linphone into file mode, which lets us experiment with alternative audio sources. Use this command to enable file mode: linphonec> soundcard use files Do you remember eSpeak from earlier in this chapter? While you rest your throat, eSpeak can provide its soothing voice to carry out entire conversations with your agents. If you haven't already got it, install eSpeak first: pi@raspberrypi ~ $ sudo apt-get install espeak Now we tell Linphone what to say next: linphonec> speak english Greetings! I'm a Linphone, obviously. This sentence will be spoken as soon as there's an established call. So you can either make an outgoing call or answer an incoming call to start the conversation, after which you're free to continue the conversation in Italian: linphonec> speak italian Buongiorno! Mi chiamo Enzo Gorlami. Should you want a message to play automatically when someone calls, just toggle auto answer: linphonec> autoanswer enable How about playing a pre-recorded message or some nice grooves? If you have a WAV or MP3 file that you'd like to play over the phone, it has to be converted to a suitable format first. A simple SoX command will do the trick: pi@raspberrypi ~ $ sox "original file.mp3" -c 1 -r 48000 playme.wav Now we can tell Linphone to play the file: linphonec> play playme.wav Finally, you can also record a call to file. Note that only the remote part of the conversation can be recorded, which makes this feature more suitable for leaving messages and such. Use the following command to record: linphonec> record message.wav Summary In this article, we set up our very own phone network using SIP Witch and connected softphones running on a wide variety of platforms including the Pi itself. Resources for Article: Further resources on this subject: Our First Project – A Basic Thermometer [article] Testing Your Speed [article] Creating a 3D world to roam in [article]
Read more
  • 0
  • 0
  • 8854

article-image-showing-cached-content-first-then-networks
Packt
03 Aug 2016
9 min read
Save for later

Showing cached content first then networks

Packt
03 Aug 2016
9 min read
In this article by Sean Amarasinghe, the author of the book, Service Worker Development Cookbook, we are going to look at the methods that enable us to control cached content by creating a performance art event viewer web app. If you are a regular visitor to a certain website, chances are that you may be loading most of the resources, like CSS and JavaScript files, from your cache, rather than from the server itself. This saves us necessary bandwidth for the server, as well as requests over the network. Having the control over which content we deliver from the cache and server is a great advantage. Server workers provide us with this powerful feature by having programmatic control over the content. (For more resources related to this topic, see here.) Getting ready To get started with service workers, you will need to have the service worker experiment feature turned on in your browser settings. Service workers only run across HTTPS. How to do it... Follow these instructions to set up your file structure. Alternatively, you can download the files from the following location: https://github.com/szaranger/szaranger.github.io/tree/master/service-workers/03/02/ First, we must create an index.html file as follows: <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Cache First, then Network</title> <link rel="stylesheet" href="style.css"> </head> <body> <section id="events"> <h1><span class="nyc">NYC</span> Events TONIGHT</h1> <aside> <img src="hypecal.png" /> <h2>Source</h2> <section> <h3>Network</h3> <input type="checkbox" name="network" id="network- disabled-checkbox"> <label for="network">Disabled</label><br /> <h3>Cache</h3> <input type="checkbox" name="cache" id="cache- disabled-checkbox"> <label for="cache">Disabled</label><br /> </section> <h2>Delay</h2> <section> <h3>Network</h3> <input type="text" name="network-delay" id="network-delay" value="400" /> ms <h3>Cache</h3> <input type="text" name="cache-delay" id="cache- delay" value="1000" /> ms </section> <input type="button" id="fetch-btn" value="FETCH" /> </aside> <section class="data connection"> <table> <tr> <td><strong>Network</strong></td> <td><output id='network-status'></output></td> </tr> <tr> <td><strong>Cache</strong></td> <td><output id='cache-status'></output><td> </tr> </table> </section> <section class="data detail"> <output id="data"></output> </section> <script src="index.js"></script> </body> </html> Create a CSS file called style.css in the same folder as the index.html file. You can find the source code in the following location on GitHub: https://github.com/szaranger/szaranger.github.io/blob/master/service-workers/03/02/style.css Create a JavaScript file called index.js in the same folder as the index.html file. You can find the source code in the following location on GitHub: https://github.com/szaranger/szaranger.github.io/blob/master/service-workers/03/02/index.js Open up a browser and go to index.html. First we are requesting data from the network with the cache enabled. Click on the Fetch button. If you click fetch again, the data has been retrieved first from cache, and then from the network, so you see duplicate data. (See the last line is same as the first.) Now we are going to select the Disabled checkbox under the Network label, and click the Fetch button again, in order to fetch data only from the cache. Select the Disabled checkbox under the Network label, as well as the Cache label, and click the Fetch button again. How it works... In the index.js file, we are setting a page specific name for the cache, as the caches are per origin based, and no other page should use the same cache name: var CACHE_NAME = 'cache-and-then-network'; If you inspect the Resources tab of the development tools, you will find the cache inside the Cache Storage tab. If we have already fetched network data, we don't want the cache fetch to complete and overwrite the data that we just got from the network. We use the networkDataReceived flag to let the cache fetch callbacks to know if a network fetch has already completed: var networkDataReceived = false; We are storing elapsed time for both network and cache in two variables: var networkFetchStartTime; var cacheFetchStartTime; The source URL for example is pointing to a file location in GitHub via RawGit: var SOURCE_URL = 'https://cdn.rawgit.com/szaranger/ szaranger.github.io/master/service-workers/03/02/events'; If you want to set up your own source URL, you can easily do so by creating a gist, or a repository, in GitHub, and creating a file with your data in JSON format (you don't need the .json extension). Once you've done that, copy the URL of the file, head over to https://rawgit.com, and paste the link there to obtain another link with content type header as shown in the following screenshot: Between the time we press the Fetch button, and the completion of receiving data, we have to make sure the user doesn't change the criteria for search, or press the Fetch button again. To handle this situation, we disable the controls: function clear() { outlet.textContent = ''; cacheStatus.textContent = ''; networkStatus.textContent = ''; networkDataReceived = false; } function disableEdit(enable) { fetchButton.disabled = enable; cacheDelayText.disabled = enable; cacheDisabledCheckbox.disabled = enable; networkDelayText.disabled = enable; networkDisabledCheckbox.disabled = enable; if(!enable) { clear(); } } The returned data will be rendered to the screen in rows: function displayEvents(events) { events.forEach(function(event) { var tickets = event.ticket ? '<a href="' + event.ticket + '" class="tickets">Tickets</a>' : ''; outlet.innerHTML = outlet.innerHTML + '<article>' + '<span class="date">' + formatDate(event.date) + '</span>' + ' <span class="title">' + event.title + '</span>' + ' <span class="venue"> - ' + event.venue + '</span> ' + tickets + '</article>'; }); } Each item of the events array will be printed to the screen as rows. The function handleFetchComplete is the callback for both the cache and the network. If the disabled checkbox is checked, we are simulating a network error by throwing an error: var shouldNetworkError = networkDisabledCheckbox.checked, cloned; if (shouldNetworkError) { throw new Error('Network error'); } Because of the reason that request bodies can only be read once, we have to clone the response: cloned = response.clone(); We place the cloned response in the cache using cache.put as a key value pair. This helps subsequent cache fetches to find this update data: caches.open(CACHE_NAME).then(function(cache) { cache.put(SOURCE_URL, cloned); // cache.put(URL, response) }); Now we read the response in JSON format. Also, we make sure that any in-flight cache requests will not be overwritten by the data we have just received, using the networkDataReceived flag: response.json().then(function(data) { displayEvents(data); networkDataReceived = true; }); To prevent overwriting the data we received from the network, we make sure only to update the page in case the network request has not yet returned: result.json().then(function(data) { if (!networkDataReceived) { displayEvents(data); } }); When the user presses the fetch button, they make nearly simultaneous requests of the network and the cache for data. This happens on a page load in a real world application, instead of being the result of a user action: fetchButton.addEventListener('click', function handleClick() { ... } We start by disabling any user input while the network fetch requests are initiated: disableEdit(true); networkStatus.textContent = 'Fetching events...'; networkFetchStartTime = Date.now(); We request data with the fetch API, with a cache busting URL, as well as a no-cache option in order to support Firefox, which hasn't implemented the caching options yet: networkFetch = fetch(SOURCE_URL + '?cacheBuster=' + now, { mode: 'cors', cache: 'no-cache', headers: headers }) In order to simulate network delays, we wait before calling the network fetch callback. In situations where the callback errors out, we have to make sure that we reject the promise we received from the original fetch: return new Promise(function(resolve, reject) { setTimeout(function() { try { handleFetchComplete(response); resolve(); } catch (err) { reject(err); } }, networkDelay); }); To simulate cache delays, we wait before calling the cache fetch callback. If the callback errors out, we make sure that we reject the promise we got from the original call to match: return new Promise(function(resolve, reject) { setTimeout(function() { try { handleCacheFetchComplete(response); resolve(); } catch (err) { reject(err); } }, cacheDelay); }); The formatDate function is a helper function for us to convert the date format we receive in the response into a much more readable format on the screen: function formatDate(date) { var d = new Date(date), month = (d.getMonth() + 1).toString(), day = d.getDate().toString(), year = d.getFullYear(); if (month.length < 2) month = '0' + month; if (day.length < 2) day = '0' + day; return [month, day, year].join('-'); } If you consider a different date format, you can shuffle the position of the array in the return statement to your preferred format. Summary In this article, we have learned how to control cached content by creating a performance art event viewer web app. Resources for Article: Further resources on this subject: AngularJS Web Application Development Cookbook [Article] Being Offline [Article] Consuming Web Services using Microsoft Dynamics AX [Article]
Read more
  • 0
  • 0
  • 8853
article-image-profiling-app
Packt
24 Apr 2015
9 min read
Save for later

Profiling an app

Packt
24 Apr 2015
9 min read
This article is written by Cecil Costa, the author of the book, Swift Cookbook. We'll delve into what profiling is and how we can profile an app by following some simple steps. It's very common to hear about issues, but if an app doesn't have any important issue, it doesn't mean that it is working fine. Imagine that you have a program that has a memory leak, presumably you won't find any problem using it for 10 minutes. However, a user may find it after using it for a few days. Don't think that this sort of thing is impossible; remember that iOS apps don't terminate, so if you do have memory leaks, it will be kept until your app blows up. Performance is another important, common topic. What if your app looks okay, but it gets slower with the passing of time? We, therefore, have to be aware of this problem. This kind of test is called profiling and Xcode comes with a very good tool for realizing this operation, which is called Instruments. In this instance, we will profile our app to visualize the amount of energy wasted by our app and, of course, let's try to reduce it. (For more resources related to this topic, see here.) Getting ready For this recipe you will need a physical device, and to install your app into the device you will need to be enrolled on the Apple Developer Program. If you have both the requirements, the next thing you have to do is create a new project called Chapter 7 Energy. How to do it... To profile an app, follow these steps: Before we start coding, we will need to add a framework to the project. Click on the Build Phases tab of your project, go to the Link Binaries with Libraries section, and press the plus sign. Once Xcode opens a dialog window asking for the framework to add, choose CoreLocation and MapKit. Now, go to the storyboard and place a label and a MapKit view. You might have a layout similar to this one: Link the MapKit view and call it just map and the UILabel class and call it just label:    @IBOutlet var label: UILabel!    @IBOutlet var map: MKMapView! Continue with the view controller; let's click at the beginning of the file to add the core location and MapKit imports: import CoreLocation import MapKit After this, you have to initialize the location manager object on the viewDidLoad method:    override func viewDidLoad() {        super.viewDidLoad()        locationManager.delegate = self        locationManager.desiredAccuracy =          kCLLocationAccuracyBest        locationManager.requestWhenInUseAuthorization()        locationManager.startUpdatingLocation()    } At the moment, you may get an error because your view controller doesn't conform with CLLocationManagerDelegate, so let's go to the header of the view controller class and specify that it implements this protocol. Another error we have to deal with is the locationManager variable, because it is not declared. Therefore, we have to create it as an attribute. And as we are declaring attributes, we will add the geocoder, which will be used later: class ViewController: UIViewController, CLLocationManagerDelegate {    var locationManager = CLLocationManager()    var geocoder = CLGeocoder() Before we implement this method that receives the positioning, let's create another method to detect whether there was any authorization error:    func locationManager(manager: CLLocationManager!,       didChangeAuthorizationStatus status:          CLAuthorizationStatus) {            var locationStatus:String            switch status {            case CLAuthorizationStatus.Restricted:                locationStatus = "Access: Restricted"               break            case CLAuthorizationStatus.Denied:                locationStatus = "Access: Denied"                break            case CLAuthorizationStatus.NotDetermined:                locationStatus = "Access: NotDetermined"               break            default:                locationStatus = "Access: Allowed"            }            NSLog(locationStatus)    } And then, we can implement the method that will update our location:    func locationManager(manager:CLLocationManager,      didUpdateLocations locations:[AnyObject]) {        if locations[0] is CLLocation {            let location:CLLocation = locations[0] as              CLLocation            self.map.setRegion(              MKCoordinateRegionMakeWithDistance(            location.coordinate, 800,800),              animated: true)                       geocoder.reverseGeocodeLocation(location,              completionHandler: { (addresses,              error) -> Void in                    let placeMarket:CLPlacemark =                      addresses[0] as CLPlacemark                let curraddress:String = (placeMarket.                  addressDictionary["FormattedAddressLines"                  ] as [String]) [0] as String                    self.label.text = "You are at                      (curraddress)"            })        }    } Before you test the app, there is still another step to follow. In your project navigator, click to expand the supporting files, and then click on info.plist. Add a row by right-clicking on the list and selecting add row. On this new row, type NSLocationWhenInUseUsageDescription as a key and on value Permission required, like the one shown here: Now, select a device and install this app onto it, and test the application walking around your street (or walking around the planet earth if you want) and you will see that the label will change, and also the map will display your current position. Now, go back to your computer and plug the device in again. Instead of clicking on play, you have to hold the play button until you see more options and then you have to click on the Profile option. The next thing that will happen is that instruments will be opened; probably, a dialog will pop up asking for an administrator account. This is due to the fact that instruments need to use some special permission to access some low-level information. On the next dialog, you will see different kinds of instruments, some of them are OS X specific, some are iOS specific, and others are for both. If you choose the wrong platform instrument, the record button will be disabled. For this recipe, click on Energy Diagnostics. Once the Energy Diagnostics window is open, you can click on the record button, which is on the upper-left corner and try to move around—yes, you need to keep the device connected to your computer, so you have to move around with both elements together—and do some actions with your device, such as pressing the home button and turning off the screen. Now, you may have a screen that displays an output similar to this one: Now, you can analyze who is spending more energy on you app. To get a better idea of this, go to your code and replace the constant kCLLocationAccuracyBest with kCLLocationAccuracyThreeKilometers and check whether you have saved some energy. How it works... Instruments are a tool used for profiling your application. They give you information about your app which can't be retrieved by code, or at least can't be retrieved easily. You can check whether your app has memory leaks, whether it is loosing performance, and as you can see, whether it is wasting lots of energy or not. In this recipe we used the GPS because it is a sensor that requires some energy. Also, you can check on the table at the bottom of your instrument to see that Internet requests were completed, which is something that if you do very frequently will also empty your battery fast. Something you might be asking is: why did we have to change info.plist? Since iOS 8, some sensors require user permission; the GPS is one of them, so you need to report what is the message that will be shown to the user. There's more... I recommend you to read the way instruments work, mainly those that you will use. Check the Apple documentation about instruments to get more details about this (https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/Introduction/Introduction.html). Summary In this article, we looked at all the hows and whats of profiling an app. We specifically looked at profiling our app to visualize the amount of energy wasted by our app. So, go ahead to try doing it. Resources for Article: Further resources on this subject: Playing with Swift [article] Using OpenStack Swift [article] Android Virtual Device Manager [article]
Read more
  • 0
  • 0
  • 8841

article-image-transactions-for-async-programming-in-javaee
Aaron Lazar
31 Jul 2018
5 min read
Save for later

Using Transactions with Asynchronous Tasks in JavaEE [Tutorial]

Aaron Lazar
31 Jul 2018
5 min read
Threading is a common issue in most software projects, no matter which language or other technology is involved. When talking about enterprise applications, things become even more important and sometimes harder. Using asynchronous tasks could be a challenge: what if you need to add some spice and add a transaction to it? Thankfully, the Java EE environment has some great features for dealing with this challenge, and this article will show you how. This article is an extract from the book Java EE 8 Cookbook, authored by Elder Moraes. Usually, a transaction means something like code blocking. Isn't it awkward to combine two opposing concepts? Well, it's not! They can work together nicely, as shown here. Adding Java EE 8 dependency Let's first add our Java EE 8 dependency: <dependency> <groupId>javax</groupId> <artifactId>javaee-api</artifactId> <version>8.0</version> <scope>provided</scope> </dependency> Let's first create a User POJO: public class User { private Long id; private String name; public Long getId() { return id; } public void setId(Long id) { this.id = id; } public String getName() { return name; } public void setName(String name) { this.name = name; } public User(Long id, String name) { this.id = id; this.name = name; } @Override public String toString() { return "User{" + "id=" + id + ", name=" + name + '}'; } } And here is a slow bean that will return User: @Stateless public class UserBean { public User getUser(){ try { TimeUnit.SECONDS.sleep(5); long id = new Date().getTime(); return new User(id, "User " + id); } catch (InterruptedException ex) { System.err.println(ex.getMessage()); long id = new Date().getTime(); return new User(id, "Error " + id); } } } Now we create a task to be executed that will return User using some transaction stuff: public class AsyncTask implements Callable<User> { private UserTransaction userTransaction; private UserBean userBean; @Override public User call() throws Exception { performLookups(); try { userTransaction.begin(); User user = userBean.getUser(); userTransaction.commit(); return user; } catch (IllegalStateException | SecurityException | HeuristicMixedException | HeuristicRollbackException | NotSupportedException | RollbackException | SystemException e) { userTransaction.rollback(); return null; } } private void performLookups() throws NamingException{ userBean = CDI.current().select(UserBean.class).get(); userTransaction = CDI.current() .select(UserTransaction.class).get(); } } And finally, here is the service endpoint that will use the task to write the result to a response: @Path("asyncService") @RequestScoped public class AsyncService { private AsyncTask asyncTask; @Resource(name = "LocalManagedExecutorService") private ManagedExecutorService executor; @PostConstruct public void init(){ asyncTask = new AsyncTask(); } @GET public void asyncService(@Suspended AsyncResponse response){ Future<User> result = executor.submit(asyncTask); while(!result.isDone()){ try { TimeUnit.SECONDS.sleep(1); } catch (InterruptedException ex) { System.err.println(ex.getMessage()); } } try { response.resume(Response.ok(result.get()).build()); } catch (InterruptedException | ExecutionException ex) { System.err.println(ex.getMessage()); response.resume(Response.status(Response .Status.INTERNAL_SERVER_ERROR) .entity(ex.getMessage()).build()); } } } To try this code, just deploy it to GlassFish 5 and open this URL: http://localhost:8080/ch09-async-transaction/asyncService How the Asynchronous execution works The magic happens in the AsyncTask class, where we will first take a look at the performLookups method: private void performLookups() throws NamingException{ Context ctx = new InitialContext(); userTransaction = (UserTransaction) ctx.lookup("java:comp/UserTransaction"); userBean = (UserBean) ctx.lookup("java:global/ ch09-async-transaction/UserBean"); } It will give you the instances of both UserTransaction and UserBean from the application server. Then you can relax and rely on the things already instantiated for you. As our task implements a Callabe<V> object that it needs to implement the call() method: @Override public User call() throws Exception { performLookups(); try { userTransaction.begin(); User user = userBean.getUser(); userTransaction.commit(); return user; } catch (IllegalStateException | SecurityException | HeuristicMixedException | HeuristicRollbackException | NotSupportedException | RollbackException | SystemException e) { userTransaction.rollback(); return null; } } You can see Callable as a Runnable interface that returns a result. Our transaction code lives here: userTransaction.begin(); User user = userBean.getUser(); userTransaction.commit(); And if anything goes wrong, we have the following: } catch (IllegalStateException | SecurityException | HeuristicMixedException | HeuristicRollbackException | NotSupportedException | RollbackException | SystemException e) { userTransaction.rollback(); return null; } Now we will look at AsyncService. First, we have some declarations: private AsyncTask asyncTask; @Resource(name = "LocalManagedExecutorService") private ManagedExecutorService executor; @PostConstruct public void init(){ asyncTask = new AsyncTask(); } We are asking the container to give us an instance from ManagedExecutorService, which It is responsible for executing the task in the enterprise context. Then we call an init() method, and the bean is constructed (@PostConstruct). This instantiates the task. Now we have our task execution: @GET public void asyncService(@Suspended AsyncResponse response){ Future<User> result = executor.submit(asyncTask); while(!result.isDone()){ try { TimeUnit.SECONDS.sleep(1); } catch (InterruptedException ex) { System.err.println(ex.getMessage()); } } try { response.resume(Response.ok(result.get()).build()); } catch (InterruptedException | ExecutionException ex) { System.err.println(ex.getMessage()); response.resume(Response.status(Response. Status.INTERNAL_SERVER_ERROR) .entity(ex.getMessage()).build()); } } Note that the executor returns Future<User>: Future<User> result = executor.submit(asyncTask); This means this task will be executed asynchronously. Then we check its execution status until it's done: while(!result.isDone()){ try { TimeUnit.SECONDS.sleep(1); } catch (InterruptedException ex) { System.err.println(ex.getMessage()); } } And once it's done, we write it down to the asynchronous response: response.resume(Response.ok(result.get()).build()); The full source code of this recipe is at Github. So now, using Transactions with Asynchronous Tasks in JavaEE isn't such a daunting task, is it? If you found this tutorial helpful and would like to learn more, head on to this book Java EE 8 Cookbook. Oracle announces a new pricing structure for Java Design a RESTful web API with Java [Tutorial] How to convert Java code into Kotlin
Read more
  • 0
  • 0
  • 8836
Modal Close icon
Modal Close icon