Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-rounding
Packt
25 Nov 2013
3 min read
Save for later

Rounding up...

Packt
25 Nov 2013
3 min read
(For more resources related to this topic, see here.) We have now successfully learned how to secure our users' passwords using hashes; however, we should take a look at the big picture, just in case. The following figure shows what a very basic web application looks like: Note the https transmission tag: HTTPS is a secure transfer protocol, which allows us to transport information in a secure way. When we transport sensitive data such as passwords in a Web Application, anyone who intercepts the connection can easily get the password in plain text, and our users' data would be compromised. In order to avoid this, we should always use HTTPS when there's sensitive data involved. HTTPS is fairly easy to setup, you just need to buy an SSL certificate and configure it with your hosting provider. Configuration varies depending on the provider, but usually they provide an easy way to do it. It is strongly suggested to use HTTPS for authentication, sign up, sign in, and other sensitive data processes. As a general rule, most (if not all) of the data exchange that requires the user to be logged in should be protected. Keep in mind that HTTPS comes at a cost, so try to avoid using HTTPS on static pages that have public information. Always keep in mind that to protect the password, we need ensure secure transport (with HTTPS) and secure storage (with strong hashes) as well. Both are critical phases and we need to be very careful with them. Now that our passwords and other sensitive data are being transferred in a secure way, we can get into the application workflow. Consider the following steps for an authentication process: The application receives an Authentication Request. The Web Layer takes care of it as it gets the parameters (username and password), and passes them to the Authentication Service. The Authentication Service calls the Database Access Layer to retrieve the user from the database. The Database Access Layer queries the database, gets the user, and returns it to the Authentication Service. The Authentication Service gets the stored hash from the users' data retrieved from the database, extracts the salt and the amount of iterations, and calls the Hashing Utility passing the password from the authentication request, the salt, and the iterations. The Hashing Utility generates the hash and returns it to the Authentication Service. The Authentication Service performs a constant-time comparison between the stored hash and the generated hash, and we inform the Web Layer if the user is authenticated or not. The Web Layer returns the corresponding view to the user depending on whether they are authenticated or not. The following figure can help us understand how this works, please consider that flows 1, 2, 3, and 4 are bidirectional: The Authentication Service and the Hashing Utility components are the ones we have been working with so far. We already know how to create hashes, this workflow is an example to understand when we should it. Summary In this article we learned how to create hashes and have now successfully learned how to secure our users' passwords using hashes. We have also learned that we need to ensure secure transport (with HTTPS) and secure storage (with strong hashes) as well. Resources for Article: Further resources on this subject: FreeRADIUS Authentication: Storing Passwords [Article] EJB 3.1: Controlling Security Programmatically Using JAAS [Article] So, what is Spring for Android? [Article]
Read more
  • 0
  • 0
  • 13507

article-image-working-java-se-projects-should-know
Packt
25 Nov 2013
5 min read
Save for later

Working with Java SE projects (Should know)

Packt
25 Nov 2013
5 min read
(For more resources related to this topic, see here.) How to do it... The following steps will show you how to experiment with JRebel on a Java SE application: Create a simple Swing application and test it. Enable JRebel on your project. Try live code injection with JRebel. How it works... We will create a simple Swing application, a frame that contains a label and a button. The action associated with the button will update the label. We will use JRebel to change the button's action without recompiling or restarting the application. Start NetBeans and create a new Java application project. Create a package, delete the default main class, and use the NetBeans assistant to create a new JFrame object by navigating to File | New File | Swing GUI Forms | JFrame Form, and then choose a name and validate. The Projects view Use the Palette tab on your right to drag-and-drop a JLabel and a JButton to your JForm (these two components are located in the Swing Controls section, under the Label and Button names). The Palette tab Now, double-click on the button you have generated. You will be redirected to the jButton1ActionPerformed method of your JFrame object. private void jButton1ActionPerformed (java.awt.event.ActionEvent evt) { // TODO add your handling code here: } Insert a code to update JLabel of your form, as shown in the following code: private void jButton1ActionPerformed (java.awt.event.ActionEvent evt) { // TODO add your handling code here: jLabel1.setText("Hello World"); } The application is now ready for testing. You can test it by pressing F6 key. NetBeans will ask you for the name of the main class. Select the JFrame form and validate. Use the jButton1 button to update the label. The first Hello World window Do not close the application immediately, and return to the code editor to update the jButton1 button's action in order to display a new message and save the new code. private void jButton1ActionPerformed (java.awt.event.ActionEvent evt) { // TODO add your handling code here: // jLabel1.setText("Hello World"); // previous message jLabel1.setText("Hello Planet"); // new message } Hit the jButton1 button in your application again; the change is not effective. Actually, your application hasn't been updated and it can't reflect code changes immediately. You will have to restart your application to see the new behavior, and this is normal. Now, let's see how JRebel will accelerate the development. You may have noticed the presence of a JRebel node in the Projects view. Right-click on it and choose Generate rebel.xml. The JRebel XML The rebel.xml file will contain the path of the compiled classes. JRebel will use these compiled classes to update your running application. Also, that means we have to ensure the Compile on Save feature is turned on for your project. Every time you apply changes to a Java class, NetBeans will recompile it and JRebel will update your running application with it. To proceed, go to the Projects Properties panel and enable the Compile on Save feature. Enabling the Compile on Save feature To finish, you have to activate JRebel (locate the JRebel button on the NetBeans toolbar). Enabling the JRebel button Now, restart your application and hit the jButton1 button a first time. Return to the code editor, modify the button's action to display a new message, save, and carefully observe the NetBeans console; you will see two interesting messages: Firstly, JRebel indicates if your license is active Secondly, you can see a message that indicates a class which has been reloaded, as follows: 2013-07-28 15:08:03 JRebel: Reloading class 'demo.ant.OurJFrame'. 2013-07-28 15:08:03 JRebel: Reloading class 'demo.ant. OurJFrame$2'. 2013-07-28 15:08:03 JRebel: Reloading class 'demo.ant. OurJFrame$1'. Hit the jButton1 button again; the message changes without restarting the application. It works! You can now continue to update and save code in order to see the changes. Try to update the displayed message again, it works again; you don't need to restart your application again (except for some changes that are not supported by JRebel). There's more... Ant is not the only build system, so you may want to use JRebel with Maven-based projects. Maven- and Ant-based projects are handled the same way: generate a rebel.xml file, activate the Compile on Save feature, enable JRebel, and simply update your code without restarting your application. You don't have to deal with the Maven pom.xml file. Also, you will still see the two same JRebel messages in the NetBeans console. They are useful to check the JRebel license and class reloading. Last but not least, to activate the Compile On Save feature, the Compile On Save menu should be set to For application execution only (or better). Maven's Compile On Save feature Summary In this article, we learnt how to work with JRebel on the Ant Java SE projects. We also learnt the about Maven Java SE projects support. Resources for Article: Further resources on this subject: The Business Layer (Java EE 7 First Look) [Article] Developing Secure Java EE Applications in GlassFish [Article] Service Oriented Java Business Integration - What's & Why's [Article]
Read more
  • 0
  • 0
  • 3662

article-image-organizing-virtual-filesystem
Packt
23 Nov 2013
13 min read
Save for later

Organizing a Virtual Filesystem

Packt
23 Nov 2013
13 min read
(For more resources related to this topic, see here.) Files are the building blocks of any computer system. This article deals with portable handling of read-only application resources, and provides recipes to store the application data. Let us briefly consider the problems covered in this article. The first one is the access to application data files. Often, application data for desktop operating systems resides in the same folder as the executable file. With Android, things get a little more complicated. The application files are packaged in the .apk file, and we simply cannot use the standard fopen()-like functions, or the std::ifstream and std::ofstream classes. The second problem results from the different rules for the filenames and paths. Windows and Linux-based systems use different path separator characters, and provide different low-level file access APIs. The third problem comes from the fact that file I/O operations can easily become the slowest part in the whole application. User experience can become problematic if interaction lags are involved. To avoid delays, we should perform the I/O on a separate thread and handle the results of the Read() operation on yet another thread. To implement this, we have all the tools required. We start with abstract I/O interfaces, implement a portable .zip archives handling approach, and proceed to asynchronous resources loading. Abstracting file streams File I/O APIs differ slightly between Windows and Android (POSIX) operating systems, and we have to hide these differences behind a consistent set of C++ interfaces. Getting ready Please make sure you are familiar with the UNIX concept of the file and memory mapping. Wikipedia may be a good start (http://en.wikipedia.org/wiki/Memory-mapped_file). How to do it... From now on, our programs will read input data using the following simple interface. The base class iObject is used to add an intrusive reference counter to instances of this class: class iIStream: public iObject { public: virtual void Seek( const uint64 Position ) = 0; virtual uint64 Read( void* Buf, const uint64 Size ) = 0; virtual bool Eof() const = 0; virtual uint64 GetSize() const = 0; virtual uint64 GetPos() const = 0; The following are a few methods that take advantage of memory-mapped files: virtual const ubyte* MapStream() const = 0; virtual const ubyte* MapStreamFromCurrentPos() const = 0; }; This interface supports both memory-mapped access using the MapStream() and MapStreamFromCurrentPos() member functions, and sequential access with the BlockRead() and Seek() methods. To write some data to the storage, we use an output stream interface, as follows (again, the base class iObject is used to add a reference counter): class iOStream: public iObject { public: virtual void Seek( const uint64 Position ) = 0; virtual uint64 GetFilePos() const = 0; virtual uint64 Write( const void* B, const uint64 Size ) = 0; }; The Seek(), GetFileSize(), GetFilePos(), and filename-related methods of the iIStream interface can be implemented in a single class called FileMapper: class FileMapper: public iIStream { public: explicit FileMapper( clPtr<iRawFile> File ); virtual ~FileMapper(); virtual std::string GetVirtualFileName() const{return FFile->GetVirtualFileName(); } virtual std::string GetFileName() const{ return FFile->GetFileName(); } Read a continuous block of data from this stream and return the number of bytes actually read: virtual uint64 BlockRead( void* Buf, const uint64 Size ) { uint64 RealSize =( Size > GetBytesLeft() ) ? GetBytesLeft() : Size; Return zero if we have already read everything: if ( RealSize < 0 ) { return 0; } memcpy( Buf, ( FFile->GetFileData() + FPosition ),static_cast<size_t>( RealSize ) ); Advance the current position and return the number of copied bytes: FPosition += RealSize; return RealSize; } virtual void Seek( const uint64 Position ) { FPosition = Position; } virtual uint64 GetFileSize() const { return FFile->GetFileSize(); } virtual uint64 GetFilePos() const { return FPosition; } virtual bool Eof() const { return ( FPosition >= GetFileSize() ); } virtual const ubyte* MapStream() const { return FFile->GetFileData(); } virtual const ubyte* MapStreamFromCurrentPos() const { return ( FFile->GetFileData() + FPosition ); } private: clPtr<iRawFile> FFile; uint64 FPosition; }; The FileMapper uses the following iRawFile interface to abstract the data access: class iRawFile: public iObject { public: iRawFile() {}; virtual ~iRawFile() {}; void SetVirtualFileName( const std::string& VFName );voidSetFileName( const std::string& FName );std::string GetVirtualFileName() const; std::string GetFileName(); virtual const ubyte* GetFileData() const = 0; virtual uint64 GetFileSize() const = 0; protected: std::string FFileName; std::string FVirtualFileName; }; Along with the trivial GetFileName() and SetFileName() methods implemented here, in the following recipes we implement the GetFileData() and GetFileSize() methods. How it works... The iIStream::BlockRead() method is useful when handling non-seekable streams. For the fastest access possible, we use memory-mapped files implemented in the following recipe. The MapStream() and MapStreamFromCurrentPos() methods are there to provide access to memory-mapped files in a convenient way. These methods return a pointer to the memory where your file, or a part of it, is mapped to. The iOStream::Write() method works similar to the standard ofstream::write() function. Refer to the project 1_AbstractStreams for the full source code of this and the following recipe. There's more... The important problem while programming for multiple platforms, in our case for Windows and Linux-based Android, is the conversion of filenames. We define the following PATH_SEPARATOR constant, using OS-specific macros, to determine the path separator character in the following way: #if defined( _WIN32 ) const char PATH_SEPARATOR = '\'; #else const char PATH_SEPARATOR = '/'; #endif The following simple function helps us to make sure we use valid filenames for our operating system: inline std::string Arch_FixFileName(const std::string& VName) { std::string s( VName ); std::replace( s.begin(), s.end(), '\', PATH_SEPARATOR ); std::replace( s.begin(), s.end(), '/', PATH_SEPARATOR ); return s; } See also Implementing portable memory-mapped files Working with in-memory files Implementing portable memory-mapped files Modern operating systems provide a powerful mechanism called the memory-mapped files. In short, it allows us to map the contents of the file into the application address space. In practice, this means we can treat files as usual arrays and access them using C pointers. Getting ready To understand the implementation of the interfaces from the previous recipe we recommend to read about memory mapping. The overview of this mechanism implementation in Windows can be found on the MSDN page at http://msdn.microsoft.com/en-us/library/ms810613.aspx. To find out more about memory mapping, the reader may refer to the mmap() function documentation. How to do it... In Windows, memory-mapped files are created using the CreateFileMapping() and MapViewOfFile() API calls. Android uses the mmap() function, which works pretty much the same way. Here we declare the RawFile class implementing the iRawFile interface. RawFile holds a pointer to a memory-mapped file and its size: ubyte* FFileData; uint64 FSize; For the Windows version, we use two handles for the file and memory-mapping object, and for the Android, we use only the file handle: #ifdef _WIN32 HANDLE FMapFile; HANDLE FMapHandle; #else int FFileHandle; #endif We use the following function to open the file and create the memory mapping: bool RawFile::Open( const string& FileName,const string& VirtualFileName ) { At first, we need to obtain a valid file descriptor associated with the file: #ifdef OS_WINDOWS FMapFile = (void*)CreateFileA( FFileName.c_str(),GENERIC_READ,FILE_SHARE_READ,NULL, OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL |FILE_FLAG_RANDOM_ACCESS,NULL ); #else FFileHandle = open( FileName.c_str(), O_RDONLY ); if ( FFileHandle == -1 ) { FFileData = NULL; FSize = 0; } #endif Using the file descriptor, we can create a file mapping. Here we omit error checks for the sake of clarity. However, the example in the supplementary materials contains more error checks: #ifdef OS_WINDOWS FMapHandle = (void*)CreateFileMapping( ( HANDLE )FMapFile,NULL, PAGE_READONLY,0, 0, NULL ); FFileData = (Lubyte*)MapViewOfFile((HANDLE)FMapHandle,FILE_MAP_READ, 0,0,0 ); DWORD dwSizeLow = 0, dwSizeHigh = 0; dwSizeLow = ::GetFileSize( FMapFile, &dwSizeHigh ); FSize = ((uint64)dwSizeHigh << 32) | (uint64)dwSizeLow; #else struct stat FileInfo; fstat( FFileHandle, &FileInfo ); FSize = static_cast<uint64>( FileInfo.st_size ); FFileData = (Lubyte*) mmap(NULL, FSize, PROT_READ,MAP_PRIVATE, FFileHandle, 0); close( FFileHandle ); #endif return true; } The correct deinitialization function closes all the handles: bool RawFile::Close() { #ifdef OS_WINDOWS if ( FFileData ) UnmapViewOfFile( FFileData ); if ( FMapHandle ) CloseHandle( (HANDLE)FMapHandle ); CloseHandle( (HANDLE)FMapFile ); #else if ( FFileData ) munmap( (void*)FFileData, FSize ); #endif return true; } The main functions of the iRawFile interface, GetFileData and GetFileSize, have trivial implementation here: virtual const ubyte* GetFileData() { return FFileData; } virtual uint64 GetFileSize() { return FSize; } How it works... To use the RawFile class we create an instance and wrap it into a FileMapper class instance: clPtr<RawFile> F = new RawFile(); F->Open("SomeFileName"); clPtr<FileMapper> FM = new FileMapper(F); The FM object can be used with any function supporting the iIStream interface. The hierarchy of all our iRawFile implementations looks like what is shown in the following figure: Implementing file writers Quite frequently, our application might want to store some of its data on the disk. Another typical use case we have already encountered is the downloading of some file from the network into a memory buffer. Here, we implement two variations of the iOStream interface for the ordinary and in-memory files. How to do it... Let us derive the FileWriter class from the iOStream interface. We add the Open() and Close() member functions on top of the iOStream interface and carefully implement the Write() operation. Our output stream implementation does not use memory-mapped files and uses ordinary file descriptors, as shown in the following code: class FileWriter: public iOStream { public: FileWriter(): FPosition( 0 ) {} virtual ~FileWriter() { Close(); } bool Open( const std::string& FileName ) { FFileName = FileName; We split Android and Windows-specific code paths using defines: #ifdef _WIN32 FMapFile = CreateFile( FFileName.c_str(),GENERIC_WRITE, FILE_SHARE_READ,NULL, CREATE_ALWAYS,FILE_ATTRIBUTE_NORMAL, NULL ); return !( FMapFile == ( void* )INVALID_HANDLE_VALUE ); #else FMapFile = open( FFileName.c_str(), O_WRONLY|O_CREAT ); FPosition = 0; return !( FMapFile == -1 ); #endif } The same technique is used in the other methods. The difference between both OS systems is is trivial, so we decided to keep everything inside a single class and separate the code using defines: void Close() { #ifdef _WIN32 CloseHandle( FMapFile ); #else if ( FMapFile != -1 ) { close( FMapFile ); } #endif } virtual std::string GetFileName() const { return FFileName; } virtual uint64 GetFilePos() const { return FPosition; } virtual void Seek( const uint64 Position ) { #ifdef _WIN32 SetFilePointerEx( FMapFile,*reinterpret_cast<const LARGE_INTEGER*>(&Position ),NULL, FILE_BEGIN ); #else if ( FMapFile != -1 ) { lseek( FMapFile, Position, SEEK_SET ); } #endif FPosition = Position; } However, things may get more complex if you decide to support more operating systems. It can be a good refactoring exercise. virtual uint64 Write( const void* Buf, const uint64 Size ) { #ifdef _WIN32 DWORD written; WriteFile( FMapFile, Buf, DWORD( Size ),&written, NULL ); #else if ( FMapFile != -1 ) { write( FMapFile, Buf, Size ); } #endif FPosition += Size; return Size; } private: std::string FFileName; #ifdef _WIN32 HANDLE FMapFile; #else int FMapFile; #endif uint64 FPosition; }; How it works… Now we can also present an implementation of the iOStream that stores everything in a memory block. To store arbitrary data in a memory block, we declare the Blob class, as shown in the following code: class Blob: public iObject { public: Blob(); virtual ~Blob(); Set the blob data pointer to some external memory block: void SetExternalData( void* Ptr, size_t Sz ); Direct access to data inside this blob: void* GetData(); … Get the current size of the blob: size_t GetSize() const; Check if this blob is responsible for managing the dynamic memory it uses: bool OwnsData() const; … Increase the size of the blob and add more data to it. This method is very useful in a network downloader: bool AppendBytes( void* Data, size_t Size ); … }; There are lots of other methods in this class. You can find the full source code in the Blob.h file. We use this Blob class, and declare the MemFileWriter class, which implements our iOStream interface, in the following way: class MemFileWriter: public iOStream { public: MemFileWriter(clPtr<Blob> Container); Change the absolute position inside a file, where new data will be written to: virtual void Seek( const uint64 Position ) { if ( Position > FContainer->GetSize() ) { Check if we are allowed to resize the blob: if ( Position > FMaxSize - 1 ) { return; } And try to resize it: if ( !FContainer->SafeResize(static_cast<size_t>( Position ) + 1 )) { return; } } FPosition = Position; } Write data to the current position of this file: virtual uint64 Write( const void* Buf, const uint64 Size ) { uint64 ThisPos = FPosition; Ensure there is enough space: Seek( ThisPos + Size ); if ( FPosition + Size > FMaxSize ) { return 0; } void* DestPtr = ( void* )( &( ( ( ubyte* )(FContainer->GetData() ))[ThisPos] ) ); Write the actual data: memcpy( DestPtr, Buf, static_cast<size_t>( Size ) ); return Size; } } private: … }; We omit the trivial implementations of GetFileName(), GetFilePos(), GetMaxSize(), SetContainer(), GetContainer(), GetMaxSize(), and SetMaxSize() member functions, along with fields declarations. You will find the full source code of them in the code bundle of the book. See also Working with in-memory files Working with in-memory files Sometimes it is very convenient to be able to treat some arbitrary in-memory runtime generated data as if it were in a file. As an example, let's consider using a JPEG image downloaded from a photo hosting, as an OpenGL texture. We do not need to save it into the internal storage, as it is a waste of CPU time. We also do not want to write separate code for loading images from memory. Since we have our abstract iIStream and iRawFile interfaces, we just implement the latter to support memory blocks as the data source. Getting ready In the previous recipes, we already used the Blob class, which is a simple wrapper around a void* buffer. How to do it... Our iRawFile interface consists of two methods: GetFileData() and GetFileSize(). We just delegate these calls to an instance of Blob: class ManagedMemRawFile: public iRawFile { public: ManagedMemRawFile(): FBlob( NULL ) {} virtual const ubyte* GetFileData() const { return ( const ubyte* )FBlob->GetData(); } virtual uint64 GetFileSize() const { return FBlob->GetSize(); } void SetBlob( const clPtr<Blob>& Ptr ) { FBlob = Ptr; } private: clPtr<Blob> FBlob; }; Sometimes it is useful to avoid the overhead of using a Blob object, and for such cases we provide another class, MemRawFile, that holds a raw pointer to a memory block and optionally takes care of the memory allocation: class MemRawFile: public iRawFile { public: virtual const ubyte* GetFileData() const { return (const ubyte*) FBuffer; } virtual uint64 GetFileSize() const { return FBufferSize; } void CreateFromString( const std::string& InString ); void CreateFromBuffer( const void* Buf, uint64 Size ); void CreateFromManagedBuffer( const void* Buf, uint64 Size ); private: bool FOwnsBuffer; const void* FBuffer; uint64 FBufferSize; }; How it works... We use the MemRawFile as an adapter for the memory block extracted from a .zip file and ManagedMemRawFile as the container for data downloaded from photo sites.
Read more
  • 0
  • 0
  • 5377

article-image-working-data-components
Packt
22 Nov 2013
15 min read
Save for later

Working with Data Components

Packt
22 Nov 2013
15 min read
(For more resources related to this topic, see here.) Introducing the DataList component The DataList component displays a collection of data in the list layout with several display types and supports AJAX pagination. The DataList component iterates through a collection of data and renders its child components for each item. Let us see how to use <p:dataList>to display a list of tag names as an unordered list: <p:dataList value="#{tagController.tags}" var="tag" type="unordered" itemType="disc"> #{tag.label} </p:dataList> The preceding <p:dataList> component displays tag names as an unordered list of elements marked with disc type bullets. The valid type options are unordered, ordered, definition, and none. We can use type="unordered" to display items as an unordered collection along with various itemType options such as disc, circle, and square. By default, type is set to unordered and itemType is set to disc. We can set type="ordered" to display items as an ordered list with various itemType options such as decimal, A, a, and i representing numbers, uppercase letters, lowercase letters, and roman numbers respectively. Time for action – displaying unordered and ordered data using DataList Let us see how to display tag names as unordered and ordered lists with various itemType options. Create <p:dataList> components to display items as unordered and ordered lists using the following code: <h:form> <p:panel header="Unordered DataList"> <h:panelGrid columns="3"> <h:outputText value="Disc"/> <h:outputText value="Circle" /> <h:outputText value="Square" /> <p:dataList value="#{tagController.tags}" var="tag" itemType="disc"> #{tag.label} </p:dataList> <p:dataList value="#{tagController.tags}" var="tag" itemType="circle"> #{tag.label} </p:dataList> <p:dataList value="#{tagController.tags}" var="tag" itemType="square"> #{tag.label} </p:dataList> </h:panelGrid> </p:panel> <p:panel header="Ordered DataList"> <h:panelGrid columns="4"> <h:outputText value="Number"/> <h:outputText value="Uppercase Letter" /> <h:outputText value="Lowercase Letter" /> <h:outputText value="Roman Letter" /> <p:dataList value="#{tagController.tags}" var="tag" type="ordered"> #{tag.label} </p:dataList> <p:dataList value="#{tagController.tags}" var="tag" type="ordered" itemType="A"> #{tag.label} </p:dataList> <p:dataList value="#{tagController.tags}" var="tag" type="ordered" itemType="a"> #{tag.label} </p:dataList> <p:dataList value="#{tagController.tags}" var="tag" type="ordered" itemType="i"> #{tag.label} </p:dataList> </h:panelGrid> </p:panel> </h:form> Implement the TagController.getTags() method to return a collection of tag objects: public class TagController { private List<Tag> tags = null; public TagController() { tags = loadTagsFromDB(); } public List<Tag> getTags() { return tags; } } What just happened? We have created DataList components to display tag names as an unordered list using type="unordered" and as an ordered list using type="ordered" with various supported itemTypes values. This is shown in the following screenshot: Using DataList with pagination support DataList has built-in pagination support that can be enabled by setting paginator="true". By enabling pagination, the various page navigation options will be displayed using the default paginator template. We can customize the paginator template to display only the desired options. The paginator can be customized using the paginatorTemplate option that accepts the following keys of UI controls: FirstPageLink LastPageLink PreviousPageLink NextPageLink PageLinks CurrentPageReport RowsPerPageDropdown Note that {RowsPerPageDropdown} has its own template, and options to display is provided via the rowsPerPageTemplate attribute (for example, rowsPerPageTemplate="5,10,15"). Also, {CurrentPageReport} has its own template defined with the currentPageReportTemplate option. You can use the {currentPage}, {totalPages}, {totalRecords}, {startRecord}, and {endRecord} keywords within the currentPageReport template. The default is "{currentPage} of {totalPages}". The default paginator template is "{FirstPageLink} {PreviousPageLink} {PageLinks} {NextPageLink} {LastPageLink}". We can customize the paginator template to display only the desired options. For example: {CurrentPageReport} {FirstPageLink} {PreviousPageLink} {PageLinks} {NextPageLink} {LastPageLink} {RowsPerPageDropdown} The paginator can be positioned using the paginatorPosition attribute in three different locations: top, bottom, or both(default). The DataList component provides the following attributes for customization: rows: This is the number of rows to be displayed per page. first: This specifies the index of the first row to be displayed. The default is 0. paginator: This enables pagination. The default is false. paginatorTemplate: This is the template of the paginator. rowsPerPageTemplate: This is the template of the rowsPerPage dropdown. currentPageReportTemplate: This is the template of the currentPageReport UI. pageLinks: This specifies the maximum number of page links to display. The default value is 10. paginatorAlwaysVisible: This defines if paginator should be hidden when the total data count is less than the number of rows per page. The default is true. rowIndexVar: This specifies the name of the iterator to refer to for each row index. varStatus: This specifies the name of the exported request scoped variable to represent the state of the iteration same as in <ui:repeat> attribute varStatus. Time for action – using DataList with pagination Let us see how we can use the DataList component's pagination support to display five tags per page. Create a DataList component with pagination support along with custom paginatorTemplate: <p:panel header="DataList Pagination"> <p:dataList value="#{tagController.tags}" var="tag" id="tags" type="none" paginator="true" rows="5" paginatorTemplate="{CurrentPageReport} {FirstPageLink} {PreviousPageLink} {PageLinks} {NextPageLink} {LastPageLink} {RowsPerPageDropdown}" rowsPerPageTemplate="5,10,15"> <f:facet name="header"> Tags </f:facet> <h:outputText value="#{tag.id} - #{tag.label}" style="margin-left:10px" /> <br/> </p:dataList> </p:panel> What just happened? We have created a DataList component along with pagination support by setting paginator="true". We have customized the paginator template to display additional information such as CurrentPageReport and RowsPerPageDropdown. Also, we have used the rowsPerPageTemplate attribute to specify the values for RowsPerPageDropdown. The following screenshot displays the result: Displaying tabular data using the DataTable component DataTable is an enhanced version of the standard DataTable that provides various additional features such as: Pagination Lazy loading Sorting Filtering Row selection Inline row/cell editing Conditional styling Expandable rows Grouping and SubTable and many more In our TechBuzz application, the administrator can view a list of users and enable/disable user accounts. First, let us see how we can display list of users using basic DataTable as follows: <p:dataTable id="usersTbl" var="user" value="#{adminController.users}"> <f:facet name="header"> List of Users </f:facet> <p:column headerText="Id"> <h:outputText value="#{user.id}" /> </p:column> <p:column headerText="Email"> <h:outputText value="#{user.emailId}" /> </p:column> <p:column headerText="FirstName"> <h:outputText value="#{user.firstName}" /> </p:column> <p:column headerText="Disabled"> <h:outputText value="#{user.disabled}" /> </p:column> <f:facet name="footer"> Total no. of Users: #{fn:length(adminController.users)}. </f:facet> </p:dataTable> The following screenshot shows us the result: PrimeFaces 4.0 introduced the Sticky component and provides out-of-the-box support for DataTable to make the header as sticky while scrolling using the stickyHeader attribute: <p:dataTable var="user" value="#{adminController.users}" stickyHeader="true"> ... </p:dataTable> Using pagination support If there are a large number of users, we may want to display users in a page-by-page style. DataTable has in-built support for pagination. Time for action – using DataTable with pagination Let us see how we can display five users per page using pagination. Create a DataTable component using pagination to display five records per page, using the following code: <p:dataTable id="usersTbl" var="user" value="#{adminController.users}" paginator="true" rows="5" paginatorTemplate="{CurrentPageReport} {FirstPageLink} {PreviousPageLink} {PageLinks} {NextPageLink} {LastPageLink} {RowsPerPageDropdown}" currentPageReportTemplate="( {startRecord} - {endRecord}) of {totalRecords} Records." rowsPerPageTemplate="5,10,15"> <p:column headerText="Id"> <h:outputText value="#{user.id}" /> </p:column> <p:column headerText="Email"> <h:outputText value="#{user.emailId}" /> </p:column> <p:column headerText="FirstName"> <h:outputText value="#{user.firstName}" /> </p:column> <p:column headerText="Disabled"> <h:outputText value="#{user.disabled}" /> </p:column> </p:dataTable> What just happened? We have created a DataTable component with the pagination feature to display five rows per page. Also, we have customized the paginator template and provided an option to change the page size dynamically using the rowsPerPageTemplate attribute. Using columns sorting support DataTable comes with built-in support for sorting on a single column or multiple columns. You can define a column as sortable using the sortBy attribute as follows: <p:column headerText="FirstName" sortBy="#{user.firstName}"> <h:outputText value="#{user.firstName}" /> </p:column> You can specify the default sort column and sort order using the sortBy and sortOrder attributes on the <p:dataTable> element: <p:dataTable id="usersTbl2" var="user" value="#{adminController.users}" sortBy="#{user.firstName}" sortOrder="descending"> </p:dataTable> The <p:dataTable> component's default sorting algorithm uses a Java comparator, you can use your own customized sort method as well: <p:column headerText="FirstName" sortBy="#{user.firstName}" sortFunction="#{adminController.sortByFirstName}"> <h:outputText value="#{user.firstName}" /> </p:column> public int sortByFirstName(Object firstName1, Object firstName2) { //return -1, 0 , 1 if firstName1 is less than, equal to or greater than firstName2 respectively return ((String)firstName1).compareToIgnoreCase(((String)firstName2)); } By default, DataTable's sortMode is set to single, to enable sorting on multiple columns set sortMode to multiple. In multicolumns' sort mode, you can click on a column while the metakey (Ctrl or command) adds the column to the order group: <p:dataTable id="usersTbl" var="user" value="#{adminController.users}" sortMode="multiple"> </p:dataTable> Using column filtering support DataTable provides support for column-level filtering as well as global filtering (on all columns) and provides an option to hold the list of filtered records. In addition to the default match mode startsWith, we can use various other match modes such as endsWith, exact, and contains. Time for action – using DataTable with filtering Let us see how we can use filters with users' DataTable. Create a DataTable component and apply column-level filters and a global filter to apply filter on all columns: <p:dataTable widgetVar="userTable" var="user" value="#{adminController.users}" filteredValue="#{adminController.filteredUsers}" emptyMessage="No Users found for the given Filters"> <f:facet name="header"> <p:outputPanel> <h:outputText value="Search all Columns:" /> <p:inputText id="globalFilter" onkeyup="userTable.filter()" style="width:150px" /> </p:outputPanel> </f:facet> <p:column headerText="Id"> <h:outputText value="#{user.id}" /> </p:column> <p:column headerText="Email" filterBy="#{user.emailId}" footerText="contains" filterMatchMode="contains"> <h:outputText value="#{user.emailId}" /> </p:column> <p:column headerText="FirstName" filterBy="#{user.firstName}" footerText="startsWith"> <h:outputText value="#{user.firstName}" /> </p:column> <p:column headerText="LastName" filterBy="#{user.lastName}" filterMatchMode="endsWith" footerText="endsWith"> <h:outputText value="#{user.lastName}" /> </p:column> <p:column headerText="Disabled" filterBy="#{user.disabled}" filterOptions="#{adminController.userStatusOptions}" filterMatchMode="exact" footerText="exact"> <h:outputText value="#{user.disabled}" /> </p:column> </p:dataTable> Initialize userStatusOptions in AdminController ManagedBean. @ManagedBean @ViewScoped public class AdminController { private List<User> users = null; private List<User> filteredUsers = null; private SelectItem[] userStatusOptions; public AdminController() { users = loadAllUsersFromDB(); this.userStatusOptions = new SelectItem[3]; this.userStatusOptions[0] = new SelectItem("", "Select"); this.userStatusOptions[1] = new SelectItem("true", "True"); this.userStatusOptions[2] = new SelectItem("false", "False"); } //setters and getters } What just happened? We have used various filterMatchMode instances, such as startsWith, endsWith, and contains, while applying column-level filters. We have used the filterOptions attribute to specify the predefined filter values, which is displayed as a select drop-down list. As we have specified filteredValue="#{adminController.filteredUsers}", once the filters are applied the filtered users list will be populated into the filteredUsers property. This following is the resultant screenshot: Since PrimeFaces Version 4.0, we can specify the sortBy and filterBy properties as sortBy="emailId" and filterBy="emailId" instead of sortBy="#{user.emailId}" and filterBy="#{user.emailId}". A couple of important tips It is suggested to use a scope longer than the request such as the view scope to keep the filteredValue attribute so that the filtered list is still accessible after filtering. The filter located at the header is a global one applying on all fields; this is implemented by calling the client-side API method called filter(). The important part is to specify the ID of the input text as globalFilter, which is a reserved identifier for DataTable. Selecting DataTable rows Selecting one or more rows from a table and performing operations such as editing or deleting them is a very common requirement. The DataTable component provides several ways to select a row(s). Selecting single row We can use a PrimeFaces' Command component, such as commandButton or commandLink, and bind the selected row to a server-side property using <f:setPropertyActionListener>, shown as follows: <p:dataTable id="usersTbl" var="user" value="#{adminController.users}"> <!-- Column definitions --> <p:column style="width:20px;"> <p:commandButton id="selectButton" update=":form:userDetails" icon="ui-icon-search" title="View"> <f:setPropertyActionListener value="#{user}" target="#{adminController.selectedUser}" /> </p:commandButton> </p:column> </p:dataTable> <h:panelGrid id="userDetails" columns="2" > <h:outputText value="Id:" /> <h:outputText value="#{adminController.selectedUser.id}"/> <h:outputText value="Email:" /> <h:outputText value="#{adminController.selectedUser.emailId}"/> </h:panelGrid> Selecting rows using a row click Instead of having a separate button to trigger binding of a selected row to a server-side property, PrimeFaces provides another simpler way to bind the selected row by using selectionMode, selection, and rowKey attributes. Also, we can use the rowSelect and rowUnselect events to update other components based on the selected row, shown as follows: <p:dataTable var="user" value="#{adminController.users}" selectionMode="single" selection="#{adminController.selectedUser}" rowKey="#{user.id}"> <p:ajax event="rowSelect" listener="#{adminController.onRowSelect}" update=":form:userDetails"/> <p:ajax event="rowUnselect" listener="#{adminController.onRowUnselect}" update=":form:userDetails"/> <!-- Column definitions --> </p:dataTable> <h:panelGrid id="userDetails" columns="2" > <h:outputText value="Id:" /> <h:outputText value="#{adminController.selectedUser.id}"/> <h:outputText value="Email:" /> <h:outputText value="#{adminController.selectedUser.emailId}"/> </h:panelGrid> Similarly, we can select multiple rows using selectionMode="multiple" and bind the selection attribute to an array or list of user objects: <p:dataTable var="user" value="#{adminController.users}" selectionMode="multiple" selection="#{adminController.selectedUsers}" rowKey="#{user.id}"> <!-- Column definitions --> </p:dataTable> rowKey should be a unique identifier from your data model and should be used by DataTable to find the selected rows. You can either define this key by using the rowKey attribute or by binding a data model that implements org.primefaces.model.SelectableDataModel. When the multiple selection mode is enabled, we need to hold the Ctrl or command key and click on the rows to select multiple rows. If we don't hold on to the Ctrl or command key and click on a row and the previous selection will be cleared with only the last clicked row selected. We can customize this behavior using the rowSelectMode attribute. If you set rowSelectMode="add", when you click on a row, it will keep the previous selection and add the current selected row even though you don't hold the Ctrl or command key. The default rowSelectMode value is new. We can disable the row selection feature by setting disabledSelection="true". Selecting rows using a radio button / checkbox Another very common scenario is having a radio button or checkbox for each row, and the user can select one or more rows and then perform actions such as edit or delete. The DataTable component provides a radio-button-based single row selection using a nested <p:column> element with selectionMode="single": <p:dataTable var="user" value="#{adminController.users}" selection="#{adminController.selectedUser}" rowKey="#{user.id}"> <p:column selectionMode="single"/> <!-- Column definitions --> </p:dataTable> The DataTable component also provides checkbox-based multiple row selection using a nested <p:column> element with selectionMode="multiple": <p:dataTable var="user" value="#{adminController.users}" selection="#{adminController.selectedUsers}" rowKey="#{user.id}"> <p:column selectionMode="multiple"/> <!-- Column definitions --> </p:dataTable> In our TechBuzz application, the administrator would like to have a facility to be able to select multiple users and disable them at one go. Let us see how we can implement this using the checkbox-based multiple rows selection.
Read more
  • 0
  • 0
  • 2611

article-image-styling-user-interface
Packt
22 Nov 2013
8 min read
Save for later

Styling the User Interface

Packt
22 Nov 2013
8 min read
(For more resources related to this topic, see here.) Styling components versus themes Before we get into this article, it's important to have a good understanding of the difference between styling an individual component and creating a theme. Almost every display component in Sencha Touch has the option to set its own style. For example, a panel component can use a style in this way: { xtype: 'panel', style: 'border: none; font: 12px Arial black', html: 'Hello World' } The style can also be set as an object using: { xtype: 'panel', style : { 'border' : 'none', 'font' : '12px Arial black', 'border-left': '1px solid black' } html: 'Hello World' } You will notice that inside the style block, we have quoted both sides of the configuration setting. This is still the correct syntax for JavaScript and a very good habit to get in to for using style blocks. This is because a number of standard CSS styles use a dash as part of their name. If we do not add quotes to border-left, JavaScript will read this as border minus left and promptly collapse in a pile of errors. We can also set a style class for a component and use an external CSS file to define the class as follows: { xtype: 'panel', cls: 'myStyle', html: 'Hello World' } Your external CSS file could then control the style of the component in the following manner: .myStyle { border: none; font: 12px Arial black; } This class-based control of display is considered a best practice as it separates the style logic from the display logic. This means that when you need to change a border color, it can be done in one file instead of hunting through multiple files for individual style settings. These styling options are very useful for controlling the display of individual components. There are also certain style elements, such as border, padding, and margin, that can be set directly in the components' configuration: { xtype: 'panel', bodyMargin: '10 5 5 5', bodyBorder: '1px solid black', bodyPadding: 5, html: 'Hello World' } These configurations can accept either a number to be applied to all sides or a CSS string value, such as 1px solid black or 10 5 5 5. The number should be entered without quotes but the CSS string values need to be within quotes. These kind of small changes can be helpful in styling your application, but what if you need to do something a bit bigger? What if you want to change the color or appearance of the entire application? What if you want to create your own default style for your buttons? This is where themes and UI styles come into play. UI styling for toolbars and buttons Let's do a quick review of the basic MVC application we created, Creating a Simple Application, and use it to start our exploration of styles with toolbars and buttons. To begin, we are going to add a few things to the first panel, which has our titlebar, toolbar, and Hello World text. Adding the toolbar In app/views, you'll find Main.js. Go ahead and open that in your editor and takea look at the first panel in our items list: items: [ { title: 'Hello', iconCls: 'home', xtype: 'panel', html: 'Hello World', items: [ { xtype: 'titlebar', docked: 'top', title: 'About TouchStart' } ] }... We're going to add a second toolbar on top of the existing one. Locate the items section, and after the curly braces for our first toolbar, add the second toolbar in the following manner: { xtype: 'titlebar', docked: 'top', title: 'About TouchStart' }, { docked: 'top', xtype: 'toolbar', items: [ {text: 'My Button'} ]} Don't forget to add a comma between the two toolbars. Extra or missing commas While working in Sencha Touch, one of the most common causes of parse errors is an extra or missing comma. When you are moving the code around, always make sure you have accounted for any stray or missing commas. Fortunately for us, the Safari Error Console will usually give us a pretty good idea about the line number to look at for these types of parse errors. A more detailed list of common errors can be found at: http://javascript.about.com/od/reference/a/error.htm Now when you take a look at the first tab, you should see our new toolbar with our button to the left. Since the toolbars both have the same background, they are a bit difficult to differentiate. So, we are going to change the appearance of the bottom bar using the ui configuration option: { docked: 'top', xtype: 'toolbar', ui: 'light', items: [ {text: 'My Button'} ] } The ui configuration is the shorthand for a particular set of styles in Sencha Touch. There are several ui styles included with Sencha Touch, and later on, we will show you how to make your own. Styling buttons Buttons can also use the ui configuration setting, for which they offer several different options: normal: This is the default button back: This is a button with the left side narrowed to a point round: This is a more drastically rounded button small: This is a smaller button action: This is a brighter version of the default button (the color varies according to the active color of the theme, which we will see later) forward: This is a button with the right side narrowed to a point Buttons also have some color options built into the ui option. These color options are confirm and decline. These options are combined with the previous shape options using a hyphen; for example, confirm-small or decline-round. Let's add some new buttons and see how this looks on our screen. Locate the items list with our button in the second toolbar: items: [ {text: 'My Button'} ] Replace that old items list with the following new items list: items: [ { text: 'Back', ui: 'back' }, { text: 'Round', ui: 'round' }, { text: 'Small', ui: 'small' }, { text: 'Normal', ui: 'normal' }, { text: 'Action', ui: 'action' }, { text: 'Forward', ui: 'forward' } ] This will produce a series of buttons across the top of our toolbar. As you may notice, all of our buttons are aligned to the left. You can move buttons to the right by adding a spacer xtype in front of the buttons you want pushed to the right. Try this by adding the following between our Forward and Action buttons: { xtype: 'spacer'}, This will make the Forward button move over to the right-hand side of the toolbar: Since buttons can actually be used anywhere, we can add some to our title bar and use the align property to control where they appear. Modify the titlebar for our first panel and add an items section, as shown in the following code: { xtype: 'titlebar', docked: 'top', title: 'About TouchStart', items: [ { xtype: 'button', text: 'Left', align: 'left' }, { xtype: 'button', text: 'Right', align: 'right' } ] } Now we should have two buttons in our title bar, one on either side of the title: Let's also add some buttons to the panel container so we can see what the ui options confirm and decline look like. Locate the end of the items section of our HelloPanel container and add the following after the second toolbar: { xtype: 'button', text: 'Confirm', ui: 'confirm', width: 100 }, { xtype: 'button', text: 'Decline', ui: 'decline', width: 100 } There are two things you may notice that differentiate our panel buttons from our toolbar buttons. The first is that we declare xtype:'button' in our panel but we don't in our toolbar. This is because the toolbar assumes it will contain buttons and xtype only has to be declared if you use something other than a button. The panel does not set a default xtype attribute, so every item in the panel must declare one. The second difference is that we declare width for the buttons. If we don't declare width when we use a button in a panel, it will expand to the full width of the panel. On the toolbar, the button auto-sizes itself to fit the text. You will also see that our two buttons in the panel are mashed together. You can separate them out by adding margin: 5 to each of the button configuration sections. These simple styling options can help make your application easier to navigate and provide the user with visual clues for important or potentially destructive actions. The tab bar The tab bar at the bottom also understands the ui configuration option. In this case, the available options are light and dark. The tab bar also changes the icon appearance based on the ui option; a light toolbar will have dark icons and a dark toolbar will have light icons. These icons are actually part of a special font called Pictos. Sencha Touch started using the Pictos font in Version 2.2 instead of images icons because of compatibility issues on some mobile devices. The icon mask from previous versions of Sencha Touch is available but has been discontinued as of Version 2.2. You can see some of the icons available in the documentation for the Ext.Button component: http://docs.sencha.com/touch/2.2.0/#!/api/Ext.Button If you're curious about the Pictos font, you can learn more about it at http://pictos.cc/
Read more
  • 0
  • 0
  • 6008

article-image-code-igniter-email-and-html-table
Packt
22 Nov 2013
18 min read
Save for later

CodeIgniter Email and HTML Table

Packt
22 Nov 2013
18 min read
There might be times rather than just plain text, so you may wish to include images, text formatting, and URLs in the body of your e-mail. HTML e-mails will allow you to do this and CodeIgniter Email library can easily be set to do just that. How to do it... HTML e-mails can be sent by executing the following steps: Create a file email.php at /path/to/codeigniter/application/controllers/. Add the following code to the controller file email.php: <?php if ( ! defined('BASEPATH')) exit('No direct script access allowed'); class Email extends CI_Controller { function __construct() { parent::__construct(); $this->load->helper('url'); $this->load->library('email'); } public function index() { redirect('email/send_email'); } public function send_email() { $config['protocol'] = 'sendmail'; $config['mailpath'] = '/usr/sbin/sendmail'; $config['charset'] = 'iso-8859-1'; $config['wordwrap'] = TRUE; $config['mailtype'] = 'html'; $this->email->initialize($config); $this->email->from('from@domain.com', 'Your Name'); $this->email->to('to@domain.com'); $this->email->subject('This is a html email'); $html = 'This is an <b>HTML</b> email'; $this->email->message($html); $this->email->send(); echo $this->email->print_debugger(); } } How it works... In the constructor controller we load the Email library (highlighted in the following code), which provides support for us to send e-mails: function __construct() { parent::__construct(); $this->load->helper('url'); $this->load->library('email'); } Next, public function index() redirects us to the function public function send_mail(), which sets some initial configuration variables for CodeIgniter Email library to work with, such as the system used to send the e-mail (in this case, sendmail), the path to send e-mail on your system, the mailtype variable (text or HTML), and so on. Take a look at the following line of code: $config['mailtype'] = 'html'; Here, we're telling CodeIgniter to send the e-mail as HTML rather than as text. These configuration settings are initialized (that is, passed to the Email library) and we begin to build the e-mail by setting the to, from, subject, and message attributes: $this->email->from('from@domain.com', 'Your Name'); $this->email->to('to@domain.com'); $this->email->subject('This is a text email'); $this->email->message('And this is some content for the text email.'); Then, send the e-mail using the following code: $this->email->send(); If all works out as planned, you should see an output similar to the following code: Your message has been successfully sent using the following protocol: sendmail User-Agent: CodeIgniter Date: Fri, 4 Oct 2013 08:56:59 +0200 From: "Your Name" <from@domain.com> Return-Path: <from@domain.com> To: to@domain.com Subject: =?iso-8859-1?Q?This_is_a_html_email?= Reply-To: "from@domain.com" <from@domain.com> X-Sender: from@domain.com X-Mailer: CodeIgniter X-Priority: 3 (Normal) Message-ID: <524e66bbf282f@domain.com> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="B_ALT_524e66bbf2868" This is a multi-part message in MIME format. Your email application may not support this format. --B_ALT_524e66bbf2868 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit This is an HTML email --B_ALT_524e66bbf2868 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable This is an <b>HTML</b> email --B_ALT_524e66bbf2868-- Sending attachments with CodeIgniter Email There might be times when you wish to send an attachment along with the e-mail, such as an invoice to a customer for a recent purchase or perhaps an image. The CodeIgniter Email library can easily be set to do just that. How to do it... You can send attachments with CodeIgniter Email by executing the following steps: Create a file email.php at /path/to/codeigniter/application/controllers/. Add the following code to the controller file, email.php: <?php if ( ! defined('BASEPATH')) exit('No direct script access allowed'); class Email extends CI_Controller { function __construct() { parent::__construct(); $this->load->helper('url'); $this->load->library('email'); } public function index() { redirect('email/send_email'); } public function send_email() { $config['protocol'] = 'sendmail'; $config['mailpath'] = '/usr/sbin/sendmail'; $config['charset'] = 'iso-8859-1'; $config['wordwrap'] = TRUE; $config['mailtype'] = 'html'; $this->email->initialize($config); $this->email->from('from@domain.com', 'Your Name'); $this->email->to('to@domain.com'); $this->email->subject('This is a html email'); $html = 'This is an <b>HTML</b> email with an attachment, <i>lovely!</i>'; $this->email->message($html); $this->email->attach('/path/to/attachment'); $this->email->send(); echo $this->email->print_debugger(); } } How it works... In the constructor controller we load the Email library (highlighted in the following code), which provides support to send e-mails: function __construct() { parent::__construct(); $this->load->helper('url'); $this->load->library('email'); } Next, public function index() redirects us to the function, public function send_mail(), which sets some initial configuration variables for the CodeIgniter Email library to work with, such as the system used to send the e-mail (in this case, sendmail), the path to send mail on your system, the mailtype variable (text or HTML), and so on. These configuration settings are initialized (that is, passed to the Email library) and we begin to build the e-mail; setting the to, from, subject, and message attributes, as well as the path to the attachment we're sending in the e-mail (highlighted in the following code): $this->email->from('from@domain.com', 'Your Name'); $this->email->to('to@domain.com'); $this->email->subject('This is a html email'); $html = 'This is an <b>HTML</b> email with an attachment, <i>lovely!</i>'; $this->email->message($html); $this->email->attach('/path/to/attachment'); Then, send the e-mail using the following code: $this->email->send(); Sending bulk e-mails with CodeIgniter Email There may be times when you wish to send out bulk e-mails; perhaps to all the people who have paid to go on a tour. You may wish to send them each a personalized e-mail, and also add an attachment. You may also want to pull their e-mail preference (plain text or HTML) from the account on your database and send them the correct format of e-mail. That's what we're going to do here. Getting ready We need to know each person's preferences such as whether they want HTML e-mails or text, and also their individual reference number (or booking ID) for their trip. As per this requirement, we are going to have a database to hold all the information; so copy the following code into your database: CREATE TABLE `bookers` ( `id` int(11) NOT NULL AUTO_INCREMENT, `firstname` varchar(50) NOT NULL, `lastname` varchar(50) NOT NULL, `email` varchar(255) NOT NULL, `email_pref` varchar(4) NOT NULL, `booking_ref` varchar(10) NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=3 ; INSERT INTO `bookers` (`id`, `firstname`, `lastname`, `email`, `email_pref`, `booking_ref`) VALUES (1, 'Robert', 'Foster', 'example1@domain1.com', 'html', 'ABC123'), (2, 'Lucy', 'Welsh', 'example2@domain2.com', 'html', 'DEF456'); How to do it... Create a file email.php at /path/to/codeigniter/application/controllers/. Add the following code to the controller file, email.php: <?php if ( ! defined('BASEPATH')) exit('No direct script access allowed'); class Email extends CI_Controller { function __construct() { parent::__construct(); $this->load->helper('url'); $this->load->library('email'); } public function index() { redirect('email/send_email'); } public function send_email() { $config['protocol'] = 'sendmail'; $config['mailpath'] = '/usr/sbin/sendmail'; $config['charset'] = 'iso-8859-1'; $config['wordwrap'] = TRUE; $query = "SELECT * FROM bookers "; $result = $this->db->query($query); foreach ($result->result() as $row) { $this->email->clear(); if ($row->email_pref == 'text') { $config['mailtype'] = 'text'; $body = 'Hi ' . $row->firstname . ', Thanks you for booking with us, please find attached the itinerary for your trip. This is your booking reference number: ' . $row->booking_ref . ' Thanks for booking with us, have a lovely trip.'; } else { $config['mailtype'] = 'html'; $body = 'Hi ' . $row->firstname . ',<br /><br />Thanks you for booking with us, please find attached the itinerary for your trip. </p>This is your booking reference number: <b>' . $row->booking_ref . '</b><br /><br />Thanks for booking with us, have a lovely trip.'; } $this->email->initialize($config); $this->email->to($row->email); $this->email->from('bookings@thecodeigniterholidaycompany.com'); $this->email->subject('Holiday booking details'); $this->email->message($body); $this->email->send(); } echo $this->email->print_debugger(); } } How it works... In the constructor controller we load the Email library (highlighted in the following code), which provides support for us to send e-mails: function __construct() { parent::__construct(); $this->load->helper('url'); $this->load->library('email'); } Next, public function index() redirects us to the function, public function send_mail(), which sets some initial configuration variables for CodeIgniter Email library to work with, such as the system used to send the e-mail (in this case, sendmail), the path to send mail from your system. We then query the database for each of the customer's booking details: $query = "SELECT * FROM bookers "; $result = $this->db->query($query); foreach ($result->result() as $row) { } The query will loop through each result and send a specific e-mail based on the values retrieved from the database in each loop. Firstly, we give ourselves a clean slate by clearing all the settings and variables from a previous loop iteration by using the CodeIgniter email function: $this->email->clear(); We then look at their e-mail preference and set the e-mail sending (mailtype) variable accordingly, along with the text for the body of the e-mails. So, if someone prefers HTML, we look for that preference and define the body of the HTML e-mail, otherwise for a text e-mail, we look for the text e-mail preference and define the body for the text e-mail: if ($row->email_pref == 'text') { $config['mailtype'] = 'text'; $body = 'Hi ' . $row->firstname . ', Thank you for booking with us, please find attached the itinerary for your trip. This is your booking reference number: ' . $row->booking_ref . ' Thanks for booking with us, have a lovely trip.'; } else { $config['mailtype'] = 'html'; $body = 'Hi ' . $row->firstname . ',<br /><br />Thank you for booking with us, please find attached the itinerary for your trip. </p>This is your booking reference number: <b>' . $row->booking_ref . '</b><br /><br />Thanks for booking with us, have a lovely trip.'; } After this, we initialize the configuration variables. Those of you who have looked at the previous few recipes will notice that the initialization takes place later in the code of this recipe than in others. This is because we cannot initialize the config variables earlier as some of the variables rely on the preferences of individual customers, which are fetched from the database. So, we have to wait until each user's details are fetched from a database to initialize each iteration of the configuration settings. And finally, we send the e-mail: $this->email->send(); If all goes well, you should see an output similar to the following: Your message has been successfully sent using the following protocol: sendmail User-Agent: CodeIgniter Date: Fri, 4 Oct 2013 20:06:13 +0200 To: to@domain.com From: <bookings@thecodeigniterholidaycompany.com> Return-Path: <bookings@thecodeigniterholidaycompany.com> Subject: =?iso-8859-1?Q?Holiday_booking_details?= Reply-To: "bookings@thecodeigniterholidaycompany.com" <bookings@thecodeigniterholidaycompany.com> X-Sender: bookings@thecodeigniterholidaycompany.com X-Mailer: CodeIgniter X-Priority: 3 (Normal) Message-ID: <524f0395942a2@thecodeigniterholidaycompany.com> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="B_ALT_524f0395942bb" This is a multi-part message in MIME format. Your email application may not support this format. --B_ALT_524f0395942bb Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Hi RobertThanks you booking with us, please find attached the itinerary for your trip. This is your booking reference number: ABC123 Thanks for booking with us, have a lovely trip. --B_ALT_524f0395942bb Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Hi Robert<br /><br />Thanks you booking with us,=20 please find attached the itinerary for your trip. </p>This is your booking reference number:=20 <b>ABC123</b><br /><br /> Thanks for booking with us, have a lovely trip. --B_ALT_524f0395942bb-- Using an HTML table with DataTable DataTable is a free to use library that turns your normal looking html table into an interactive marvel with sortable and searchable columns and a whole lot more; we're going to use it with CodeIgniter, merging DataTable and CodeIgniter table functionality. It's simple to use and is able to handle most of the things you will need it for. Here, in this recipe, we're going to use it with DataTable to create an interactive HTML table that is sortable and searchable. It has pagination too! If you want database results, move on to the next recipe, Using an HTML table with DataTable and a database, where we'll look at populating a table from a database query. Getting ready For this recipe, you will need to follow the given procedure: Ensure that you've downloaded DataTable from the following link: https://datatables.net/download/ Unzip the downloaded .zip file, and move the files to a location on your web server or localhost, which will be accessible by CodeIgniter. For this recipe, I have put the folder at application/views; but you can make your own choice if you wish. How to do it... Create four files as given: /path/to/codeigniter/application/controllers/table.php /path/to/codeigniter/application/views/table_header.php /path/to/codeigniter/application/views/table_body.php /path/to/codeigniter/application/views/table_footer.php Create the controller file, table.php, and add the following code to it: <?php if ( ! defined('BASEPATH')) exit('No direct script access allowed'); class Table extends CI_Controller { function __construct() { parent::__construct(); $this->load->library('table'); } public function index() { $tmpl = array ( 'table_open' => '<table border="0" cellpadding="4" cellspacing="0" id="example">', 'heading_row_start' => '<tr>', 'heading_row_end' => '</tr>', 'heading_cell_start' => '<th>', 'heading_cell_end' => '</th>', 'row_start' => '<tr>', 'row_end' => '</tr>', 'cell_start' => '<td>', 'cell_end' => '</td>', 'row_alt_start' => '<tr>', 'row_alt_end' => '</tr>', 'cell_alt_start' => '<td>', 'cell_alt_end' => '</td>', 'table_close' => '</table>' ); $this->table->set_template($tmpl); $this->table->set_heading(array('ID', 'First Name', 'Last Name')); $this->table->add_row(array('1', 'Rob', 'Foster')); $this->table->add_row(array('2', 'Lucy', 'Welsh')); $this->table->add_row(array('3', 'George', 'Foster')); $this->table->add_row(array('4', 'Jackie', 'Foster')); $this->table->add_row(array('5', 'Antony', 'Welsh')); $this->table->add_row(array('6', 'Rowena', 'Welsh')); $this->table->add_row(array('7', 'Peter', 'Foster')); $this->table->add_row(array('8', 'Jenny', 'Foster')); $this->table->add_row(array('9', 'Oliver', 'Welsh')); $this->table->add_row(array('10', 'Felicity', 'Foster')); $this->table->add_row(array('11', 'Harrison', 'Foster')); $this->table->add_row(array('12', 'Mia', 'The Cat')); $data['table'] = $this->table->generate(); $this->load->view('tables/table_header'); $this->load->view('tables/table_body',$data); $this->load->view('tables/table_footer'); } } Create the view file, table_header.php, and add the following code to it: <html> <head> <style type="text/css" title="currentStyle"> @import "<?php echo $this->config->item('base_url') ; ?>application/views/DataTables-1.9.4/media/css/demo_page.css"; @import "<?php echo $this->config->item('base_url') ; ?>application/views/DataTables-1.9.4/media/css/jquery.dataTables.css"; </style> <script type="text/javascript" language="javascript" src="<?php echo $this->config->item('base_url') ; ?>application/views/DataTables-1.9.4/media/js/jquery.js"></script> <script type="text/javascript" language="javascript" src="<?php echo $this->config->item('base_url') ; ?>application/views/DataTables-1.9.4/media/js/jquery.dataTables.js"></script> <script type="text/javascript" charset="utf-8"> $(document).ready(function() { $('#example').dataTable(); } ); </script> </head> <body> Take a look at the <script> tag: <script type="text/javascript" charset="utf-8"> $(document).ready(function() { $('#example').dataTable(); } ); </script> The #example parameter is the ID of the table (detailed in the following How it works... section). Ensure that the value example in <script> and table markup is the same. Create the view file, table_body.php, and add the following code to it: <?php echo $table ; ?> Create the controller file, table_footer.php, and add the following code to it:   </body> </html> How it works... The constructor in the table controller loads the CodeIgniter's Table library: function __construct() { parent::__construct(); $this->load->library('table'); } The public function index() function is called. We then define how we want our HTML table markup to appear. This is where you can place any markup for the specific CSS, using which you can style the elements of the table: $tmpl = array ( 'table_open' => '<table border="0" cellpadding="4" cellspacing="0" id="example">', 'heading_row_start' => '<tr>', 'heading_row_end' => '</tr>', 'heading_cell_start' => '<th>', 'heading_cell_end' => '</th>', 'row_start' => '<tr>', 'row_end' => '</tr>', 'cell_start' => '<td>', 'cell_end' => '</td>', 'row_alt_start' => '<tr>', 'row_alt_end' => '</tr>', 'cell_alt_start' => '<td>', 'cell_alt_end' => '</td>', 'table_close' => '</table>' ); Take a closer look at the table_open element of the $tmpl array. Look for the item highlighted in the preceding code. The id="example" item is used by DataTable (in the <script> tag of the file table_header.php) to apply its CSS and functionality. You can, of course, name it anything you like, but be sure to reflect that change in the JavaScript. We then call $this->table->set_template() to apply the HTML table markup: $this->table->set_template($tmpl); We then set the table headers and apply the data for our rows. Ensure that the number of items in the table headers is the same as the number of items in the table data: $this->table->set_heading(array('ID', 'First Name', 'Last Name')); $this->table->add_row(array('1', 'Rob', 'Foster')); $this->table->add_row(array('2', 'Lucy', 'Welsh')); $this->table->add_row(array('3', 'George', 'Foster')); $this->table->add_row(array('4', 'Jackie', 'Foster')); $this->table->add_row(array('5', 'Antony', 'Welsh')); $this->table->add_row(array('6', 'Rowena', 'Welsh')); $this->table->add_row(array('7', 'Peter', 'Foster')); $this->table->add_row(array('8', 'Jenny', 'Foster')); $this->table->add_row(array('9', 'Oliver', 'Welsh')); $this->table->add_row(array('10', 'Felicity', 'Foster')); $this->table->add_row(array('11', 'Harrison', 'Foster')); $this->table->add_row(array('12', 'Mia', 'The Cat')); We then generate the table. The $this->table->generate() function will return a string of HTML, which we save in $data['table']. $data['table'] = $this->table->generate(); The $data array our view files for rendering to the browser: $this->load->view('tables/table_header'); $this->load->view('tables/table_body',$data); $this->load->view('tables/table_footer');
Read more
  • 0
  • 0
  • 13105
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-plotting-charts-images-and-maps
Packt
22 Nov 2013
9 min read
Save for later

Plotting Charts with Images and Maps

Packt
22 Nov 2013
9 min read
(For more resources related to this topic, see here.) This article explores how to work with images and maps. Python has some well-known image libraries that allow us to process images in both aesthetic and scientific ways. We will touch on PIL's capabilities by demonstrating how to process images by applying filters and by resizing them. Furthermore, we will show how to use image files as annotation for our matplotlibs' charts. To deal with data visualization of geospatial datasets, we will cover the functionality of Python's available libraries and public APIs that we can use with map-based visual representations. The final recipe shows how Python can create CAPTCHA test images. Processing images with PIL Why use Python for image processing, if we could use WIMP (http://en.wikipedia.org/wiki/WIMP_(computing)) or WYSIWYG (http://en.wikipedia.org/wiki/WYSIWYG) to achieve the same goal? This is used because we want to create an automated system to process images in real time without human support, thus, optimizing the image pipeline. Getting ready Note that the PIL coordinate system assumes that the (0,0) coordinate is in the upper-left corner. The Image module has a useful class and instance methods to perform basic operations over a loaded image object (im): im = Image.open(filename): This opens a file and loads the image into im object. im.crop(box): This crops the image inside the coordinates defined by box. box defines left, upper, right, lower pixels coordinates (for example: box = (0, 100, 100,100)). im.filter(filter): This applies a filter on the image and returns a filtered image. im.histogram(): This returns a histogram list for this image, where each item represents the number of pixels. Number of items in the list is 256 for single channel images, but if the image is not a single channel image, there can be more items in the list. For an RGB image the list contains 768 items (one set of 256 values for each channel). im.resize(size, filter): This resizes the image and uses a filter for resampling. The possible filters are NEAREST, BILINEAR, BICUBIC, and ANTIALIAS. The default is NEAREST. im.rotate(angle, filter): This rotates an image in the counter clockwise direction. im.split(): This splits bands of image and returns a tuple of individual bands. Useful for splitting an RGB image into three single band images. im.transform(size, method, data, filter): This applies transformation on a given image using data and a filter. Transformation can be AFFINE, EXTENT, QUAD, and MESH. You can read more about transformation in the official documentation. Data defines the box in the original image where the transformation will be applied. The ImageDraw module allows us to draw over the image, where we can use functions such as arc, ellipse, line, pieslice, point, and polygon to modify the pixels of the loaded image. The ImageChops module contains a number of image channel operations (hence the name Chops) that can be used for image composition, painting, special effects, and other processing operations. Channel operations are allowed only for 8-bit images. Here are some interesting channel operations: ImageChops.duplicate(image): This copies current image into a new image object ImageChops.invert(image): This inverts an image and returns a copy ImageChops.difference(image1, image2): This is useful for verification that images are the same without visual inspection The ImageFilter module contains the implementation of the kernel class that allows the creation of custom convolution kernels. This module also contains a set of healthy common filters that allows the application of well-known filters (BLUR and MedianFilter) to our image. There are two types of filters provided by the ImageFilter module: fixed image enhancement filters and image filters that require certain arguments to be defined; for example, size of the kernel to be used. We can easily get the list of all fixed filter names in IPython: In [1]: import ImageFilter In [2]: [ f for f in dir(ImageFilter) if f.isupper()] Out[2]: ['BLUR', 'CONTOUR', 'DETAIL', 'EDGE_ENHANCE', 'EDGE_ENHANCE_MORE', 'EMBOSS', 'FIND_EDGES', 'SHARPEN', 'SMOOTH', 'SMOOTH_MORE'] The next example shows how we can apply all currently supported fixed filters on any supported image: import os import sys from PIL import Image, ImageChops, ImageFilter class DemoPIL(object): def __init__(self, image_file=None): self.fixed_filters = [ff for ff in dir(ImageFilter) if ff.isupper()] assert image_file is not None assert os.path.isfile(image_file) is True self.image_file = image_file self.image = Image.open(self.image_file) def _make_temp_dir(self): from tempfile import mkdtemp self.ff_tempdir = mkdtemp(prefix="ff_demo") def _get_temp_name(self, filter_name): name, ext = os.path.splitext(os.path.basename(self.image_file)) newimage_file = name + "-" + filter_name + ext path = os.path.join(self.ff_tempdir, newimage_file) return path def _get_filter(self, filter_name): # note the use Python's eval() builtin here to return function object real_filter = eval("ImageFilter." + filter_name) return real_filter def apply_filter(self, filter_name): print "Applying filter: " + filter_name filter_callable = self._get_filter(filter_name) # prevent calling non-fixed filters for now if filter_name in self.fixed_filters: temp_img = self.image.filter(filter_callable) else: print "Can't apply non-fixed filter now." return temp_img def run_fixed_filters_demo(self): self._make_temp_dir() for ffilter in self.fixed_filters: temp_img = self.apply_filter(ffilter) temp_img.save(self._get_temp_name(ffilter)) print "Images are in: {0}".format((self.ff_tempdir),) if __name__ == "__main__": assert len(sys.argv) == 2 demo_image = sys.argv[1] demo = DemoPIL(demo_image) # will create set of images in temporary folder demo.run_fixed_filters_demo() We can run this easily from the command prompt: $ pythonch06_rec01_01_pil_demo.py image.jpeg We packed our little demo in the DemoPIL class, so we can extend it easily while sharing the common code around the demo function run_fixed_filters_demo. Common code here includes opening the image file, testing if the file is really a file, creating temporary directory to hold our filtered images, building the filtered image filename, and printing useful information to user. This way the code is organized in a better manner and we can easily focus on our demo function, without touching other parts of the code. This demo will open our image file and apply every fixed filter available in ImageFilter to it and save that new filtered image in a unique temporary directory. The location of this temporary directory is retrieved, so we can open it with our OS's file explorer and view the created images. As an optional exercise, try extending this demo class to perform other filters available in ImageFilter on the given image. How to do it... The example in this section shows how we can process all the images in a certain folder. We specify a target path, and the program reads all the image files in that target path (images folder) and resizes them to a specified ratio (0.1 in this example), and saves each one in a target folder called thumbnail_folder: import os import sys from PIL import Image class Thumbnailer(object): def __init__(self, src_folder=None): self.src_folder = src_folder self.ratio = .3 self.thumbnail_folder = "thumbnails" def _create_thumbnails_folder(self): thumb_path = os.path.join(self.src_folder, self.thumbnail_folder) if not os.path.isdir(thumb_path): os.makedirs(thumb_path) def _build_thumb_path(self, image_path): root = os.path.dirname(image_path) name, ext = os.path.splitext(os.path.basename(image_path)) suffix = ".thumbnail" return os.path.join(root, self.thumbnail_folder, name + suffix + ext) def _load_files(self): files = set() for each in os.listdir(self.src_folder): each = os.path.abspath(self.src_folder + '/' + each) if os.path.isfile(each): files.add(each) return files def _thumb_size(self, size): return (int(size[0] * self.ratio), int(size[1] * self.ratio)) def create_thumbnails(self): self._create_thumbnails_folder() files = self._load_files() for each in files: print "Processing: " + each try: img = Image.open(each) thumb_size = self._thumb_size(img.size) resized = img.resize(thumb_size, Image.ANTIALIAS) savepath = self._build_thumb_path(each) resized.save(savepath) except IOError as ex: print "Error: " + str(ex) if __name__ == "__main__": # Usage: # ch06_rec01_02_pil_thumbnails.py my_images assert len(sys.argv) == 2 src_folder = sys.argv[1] if not os.path.isdir(src_folder): print "Error: Path '{0}' does not exits.".format((src_folder)) sys.exit(-1) thumbs = Thumbnailer(src_folder) # optionally set the name of theachumbnail folder relative to *src_folder*. thumbs.thumbnail_folder = "THUMBS" # define ratio to resize image to # 0.1 means the original image will be resized to 10% of its size thumbs.ratio = 0.1 # will create set of images in temporary folder thumbs.create_thumbnails() How it works... For the given src_folder folder, we load all the files in this folder and try to load each file using Image.open(); this is the logic of the create_thumbnails() function. If the file we try to load is not an image, IOError will be thrown, and it will print this error and skip to next file in the sequence. If we want to have more control over what files we load, we should change the _load_files() function to only include files with certain extension (file type): for each in os.listdir(self.src_folder): if os.path.isfile(each) and os.path.splitext(each) is in ('.jpg','.png'): self._files.add(each) This is not foolproof, as file extension does not define file type, it just helps the operating system to attach a default program to the file, but it works in majority of the cases and is simpler than reading a file header to determine the file content (which still does not guarantee that the file really is the first couple of bytes, say it is). There's more... With PIL, although not used very often, we can easily convert images from one format to the other. This is achievable with two simple operations: first open an image in a source format using open(), and then save that image in the other format using save(). Format is defined either implicitly via filename extension (.png or .jpeg), or explicitly via the format of the argument passed to the save() function.
Read more
  • 0
  • 0
  • 12976

article-image-common-qlikview-script-errors
Packt
22 Nov 2013
3 min read
Save for later

Common QlikView script errors

Packt
22 Nov 2013
3 min read
(For more resources related to this topic, see here.) QlikView error messages displayed during the running of the script, during reload, or just after the script is run are key to understanding what errors are contained in your code. After an error is detected and the error dialog appears, review the error, and click on OK or Cancel on the Script Error dialog box. If you have the debugger open, click on Close, then click on Cancel on the Sheet Properties dialog. Re-enter the Script Editor and examine your script to fix the error. Errors can come up as a result of syntax, formula or expression errors, join errors, circular logic, or any number of issues in your script. The following are a few common error messages you will encounter when developing your QlikView script. The first one, illustrated in the following screenshot, is the syntax error we received when running the code that missed a comma after Sales. This is a common syntax error. It's a little bit cryptic, but the error is contained in the code snippet that is displayed. The error dialog does not exactly tell you that it expected a comma in a certain place, but with practice, you will realize the error quickly. The next error is a circular reference error. This error will be handled automatically by QlikView. You can choose to accept QlikView's fix of loosening one of the tables in the circular reference (view the data model in Table Viewer for more information on which table is loosened, or view the Document Properties dialog, Tables tab to find out which table is marked Loosely Coupled). Alternatively, you can choose another table to be loosely coupled in the Document Properties, Tables tab, or you can go back into the script and fix the circular reference with one of the methods. The following screenshot is a warning/error dialog displayed when you have a circular reference in a script: Another common issue is an unknown statement error that can be caused by an error in writing your script—missed commas, colons, semicolons, brackets, quotation marks, or an improperly written formula. In the case illustrated in the following screenshot, the error has encountered an unknown statement—namely, the Customers line that QlikView is attempting to interpret as Customers Load *…. The fix for this error is to add a colon after Customers in the following way: Customers: There are instances when a load script will fail silently. Attempting to store a QVD or CSV to a file that is locked by another user viewing it is one such error. Another example is when you have two fields with the same name in your load statement. The debugger can help you find the script lines in which the silent error is present. Summary In this article we learned about QlikView error messages displayed during the script execution. Resources for Article: Further resources on this subject: Meet QlikView [Article] Introducing QlikView elements [Article] Linking Section Access to multiple dimensions [Article]
Read more
  • 0
  • 0
  • 9388

Packt
22 Nov 2013
15 min read
Save for later

Unity Networking – The Pong Game

Packt
22 Nov 2013
15 min read
(For more resources related to this topic, see here.) Multiplayer is everywhere. It's a staple of AAA games and small-budget indie offerings alike. Multiplayer games tap into our most basic human desires. Whether it be teaming up with strangers to survive a zombie apocalypse, or showing off your skills in a round of "Capture the Flag" on your favorite map, no artificial intelligence in the world comes close to the feeling of playing with a living, breathing, and thinking human being. Unity3D has a sizable number of third-party networking middleware aimed at developing multiplayer games, and is arguably one of the easiest platforms to prototype multiplayer games. The first networking system most people encounter in Unity is the built-in Unity Networking API . This API simplifies a great many tasks in writing networked code by providing a framework for networked objects rather than just sending messages. This works by providing a NetworkView component, which can serialize object state and call functions across the network. Additionally, Unity provides a Master server, which essentially lets players search among all public servers to find a game to join, and can also help players in connecting to each other from behind private networks. In this article, we will cover: Introducing multiplayer Introducing UDP communication Setting up your own Master server for testing What a NetworkView is Serializing object state Calling RPCs Starting servers and connecting to them Using the Master server API to register servers and browse available hosts Setting up a dedicated server model Loading networked levels Creating a Pong clone using Unity networking Introducing multiplayer games Before we get started on the details of communication over the Internet, what exactly does multiplayer entail in a game? As far as most players are concerned, in a multiplayer game they are sharing the same experience with other players. It looks and feels like they are playing the same game. In reality, they aren't. Each player is playing a separate game, each with its own game state. Trying to ensure that all players are playing the exact same game is prohibitively expensive. Instead, games attempt to synchronize just enough information to give the illusion of a shared experience. Games are almost ubiquitously built around a client-server architecture, where each client connects to a single server. The server is the main hub of the game, ideally the machine for processing the game state, although at the very least it can serve as a simple "middleman" for messages between clients. Each client represents an instance of the game running on a computer. In some cases the server might also have a client, for instance some games allow you to host a game without starting up an external server program. While an MMO ( Massively Multiplayer Online ) might directly connect to one of these servers, many games do not have prior knowledge of the server IPs. For example, FPS games often let players host their own servers. In order to show the user a list of servers they can connect to, games usually employ another server, known as the "Master Server" or alternatively the "Lobby server". This server's sole purpose is to keep track of game servers which are currently running, and report a list of these to clients. Game servers connect to the Master server in order to announce their presence publicly, and game clients query the Master server to get an updated list of game servers currently running. Alternatively, this Master server sometimes does not keep track of servers at all. Sometimes games employ "matchmaking", where players connect to the Lobby server and list their criteria for a game. The server places this player in a "bucket" based on their criteria, and whenever a bucket is full enough to start a game, a host is chosen from these players and that client starts up a server in the background, which the other players connect to. This way, the player does not have to browse servers manually and can instead simply tell the game what they want to play. Introducing UDP communication The built-in Unity networking is built upon RakNet . RakNet uses UDP communication for efficiency. UDP ( User Datagram Protocols ) is a simple way to send messages to another computer. These messages are largely unchecked, beyond a simple checksum to ensure that the message has not been corrupted. Because of this, messages are not guaranteed to arrive, nor are they guaranteed to only arrive once (occasionally a single message can be delivered twice or more), or even in any particular order. TCP, on the other hand, guarantees each message to be received just once, and in the exact order they were sent, although this can result in increased latency (messages must be resent several times if they fail to reach the target, and messages must be buffered when received, in order to be processed in the exact order they were sent). To solve this, a reliability layer must be built on top of UDP. This is known as rUDP ( reliable UDP ). Messages can be sent unreliably (they may not arrive, or may arrive more than once), or reliably (they are guaranteed to arrive, only once per message, and in the correct order). If a reliable message was not received or was corrupt, the original sender has to resend the message. Additionally, messages will be stored rather than immediately processed if they are not in order. For example, if you receive messages 1, 2, and 4, your program will not be able to handle those messages until message 3 arrives. Allowing unreliable or reliable switching on a per-message basis affords better overall performance. Messages, such as player position, are better suited to unreliable messages (if one fails to arrive, another one will arrive soon anyway), whereas damage messages must be reliable (you never want to accidentally drop a damage message, and having them arrive in the same order they were sent reduces race conditions). In Unity, you can serialize the state of an object (for example, you might serialize the position and health of a unit) either reliably or unreliably (unreliable is usually preferred). All other messages are sent reliably. Setting up the Master Server Although Unity provide their own default Master Server and Facilitator (which is connected automatically if you do not specify your own), it is not recommended to use this for production. We'll be using our own Master Server, so you know how to connect to one you've hosted yourself. Firstly, go to the following page: http://unity3d.com/master-server/ We're going to download two of the listed server components: the Master Server and the Facilitator as shown in the following screenshot: The servers are provided in full source, zipped. If you are on Windows using Visual Studio Express, open up the Visual Studio .sln solution and compile in the Release mode. Navigate to the Release folder and run the EXE (MasterServer.exe or Facilitator.exe). If you are on a Mac, you can either use the included XCode project, or simply run the Makefile (the Makefile works under both Linux and Mac OS X). The Master Server, as previously mentioned, enables our game to show a server lobby to players. The Facilitator is used to help clients connect to each other by performing an operation known as NAT punch-through . NAT is used when multiple computers are part of the same network, and all use the same public IP address. NAT will essentially translate public and private IPs, but in order for one machine to connect to another, NAT punch-through is necessary. You can read more about it here: http://www.raknet.net/raknet/manual/natpunchthrough.html The default port for the Master Server is 23466, and for the Facilitator is 50005. You'll need these later in order to configure Unity to connect to the local Master Server and Facilitator instead of the default Unity-hosted servers. Now that we've set up our own servers, let's take a look at the Unity Networking API itself. NetworkViews and state serialization In Unity, game objects that need to be networked have a NetworkView component. The NetworkView component handles communication over the network, and even helps make networked state serialization easier. It can automatically serialize the state of a Transform, Rigidbody, or Animation component, or in one of your own scripts you can write a custom serialization function. When attached to a game object, NetworkView will generate a NetworkViewID for NetworkView. This ID serves to uniquely identify a NetworkView across the network. An object can be saved as part of a scene with NetworkView attached (this can be used for game managers, chat boxes, and so on), or it can be saved in the project as a prefab and spawned later via Network.Instantiate (this is used to generate player objects, bullets, and so on). Network.Instantiate is the multiplayer equivalent to GameObject.Instantiate —it sends a message over the network to other clients so that all clients spawn the object. It also assigns a network ID to the object, which is used to identify the object across multiple clients (the same object will have the same network ID on every client). A prefab is a template for a game object (such as the player object). You can use the Instantiate methods to create a copy of the template in the scene. Spawned network game objects can also be destroyed via Network.Destroy. It is the multiplayer counterpart of GameObject.Destroy. It sends a message to all clients so that they all destroy the object. It also deletes any RPC messages associated with that object. NetworkView has a single component that it will serialize. This can be a Transform, a Rigidbody, an Animation, or one of your own components that has an OnSerializeNetworkView function. Serialized values can either be sent with the ReliableDeltaCompressed option, where values are always sent reliably and compressed to include only changes since the last update, or they can be sent with the Unreliable option, where values are not sent reliably and always include the full values (not the change since the last update, since that would be impossible to predict over UDP). Each method has its own advantages and disadvantages. If data is constantly changing, such as player position in a first person shooter, in general Unreliable is preferred to reduce latency. If data does not often change, use the ReliableDeltaCompressed option to reduce bandwidth (as only changes will be serialized). NetworkView can also call methods across the network via Remote Procedure Calls ( RPC ). RPCs are always completely reliable in Unity Networking, although some networking libraries allow you to send unreliable RPCs, such as uLink or TNet. Writing a custom state serializer While initially a game might simply serialize Transform or Rigidbody for testing, eventually it is often necessary to write a custom serialization function. This is a surprisingly easy task. Here is a script that sends an object's position over the network: using UnityEngine; using System.Collections; public class ExampleUnityNetworkSerializePosition : MonoBehaviour { public void OnSerializeNetworkView( BitStream stream, NetworkMessageInfo info ) { // we are currently writing information to the network if( stream.isWriting ) { // send the object's position Vector3 position = transform.position; stream.Serialize( ref position ); } // we are currently reading information from the network else { // read the first vector3 and store it in 'position' Vector3 position = Vector3.zero; stream.Serialize( ref position ); // set the object's position to the value we were sent transform.position = position; } } } Most of the work is done with BitStream. This is used to check if NetworkView is currently writing the state, or if it is reading the state from the network. Depending on whether it is reading or writing, stream.Serialize behaves differently. If NetworkView is writing, the value will be sent over the network. However, if NetworkView is reading, the value will be read from the network and saved in the referenced variable (thus the ref keyword, which passes Vector3 by reference rather than value). Using RPCs RPCs are useful for single, self-contained messages that need to be sent, such as a character firing a gun, or a player saying something in chat. In Unity, RPCs are methods marked with the [RPC] attribute. This can be called by name via networkView.RPC( "methodName", … ). For example, the following script prints to the console on all machines when the space key is pressed. using UnityEngine; using System.Collections; public class ExampleUnityNetworkCallRPC : MonoBehavior { void Update() { // important – make sure not to run if this networkView is notours if( !networkView.isMine ) return; // if space key is pressed, call RPC for everybody if( Input.GetKeyDown( KeyCode.Space ) ) networkView.RPC( "testRPC", RPCMode.All ); } [RPC] void testRPC( NetworkMessageInfo info ) { // log the IP address of the machine that called this RPC Debug.Log( "Test RPC called from " + info.sender.ipAddress ); } } Also note the use of NetworkView.isMine to determine ownership of an object. All scripts will run 100 percent of the time regardless of whether your machine owns the object or not, so you have to be careful to avoid letting some logic run on remote machines; for example, player input code should only run on the machine that owns the object. RPCs can either be sent to a number of players at once, or to a specific player. You can either pass an RPCMode to specify which group of players to receive the message, or a specific NetworkPlayer to send the message to. You can also specify any number of parameters to be passed to the RPC method. RPCMode includes the following entries: All (the RPC is called for everyone) AllBuffered (the RPC is called for everyone, and then buffered for when new players connect, until the object is destroyed) Others (the RPC is called for everyone except the sender) OthersBuffered (the RPC is called for everyone except the sender, and then buffered for when new players connect, until the object is destroyed) Server (the RPC is sent to the host machine) Initializing a server The first thing you will want to set up is hosting games and joining games. To initialize a server on the local machine, call Network.InitializeServer. This method takes three parameters: the number of allowed incoming connections, the port to listen on, and whether to use NAT punch-through. The following script initializes a server on port 25000 which allows 8 clients to connect: using UnityEngine; using System.Collections; public class ExampleUnityNetworkInitializeServer : MonoBehavior { void OnGUI() { if( GUILayout.Button( "Launch Server" ) ) { LaunchServer(); } } // launch the server void LaunchServer() { // Start a server that enables NAT punchthrough, // listens on port 25000, // and allows 8 clients to connect Network.InitializeServer( 8, 25005, true ); } // called when the server has been initialized void OnServerInitialized() { Debug.Log( "Server initialized" ); } } You can also optionally enable an incoming password (useful for private games) by setting Network.incomingPassword to a password string of the player's choice, and initializing a general-purpose security layer by calling Network.InitializeSecurity(). Both of these should be set up before actually initializing the server. Connecting to a server To connect to a server you know the IP address of, you can call Network.Connect. The following script allows the player to enter an IP, a port, and an optional password and attempts to connect to the server: using UnityEngine; using System.Collections; public class ExampleUnityNetworkingConnectToServer : MonoBehavior { private string ip = ""; private string port = ""; private string password = ""; void OnGUI() { GUILayout.Label( "IP Address" ); ip = GUILayout.TextField( ip, GUILayout.Width( 200f ) ); GUILayout.Label( "Port" ); port = GUILayout.TextField( port, GUILayout.Width( 50f ) ); GUILayout.Label( "Password (optional)" ); password = GUILayout.PasswordField( password, '*',GUILayout.Width( 200f ) ); if( GUILayout.Button( "Connect" ) ) { int portNum = 25005; // failed to parse port number – a more ideal solution is tolimit input to numbers only, a number of examples can befound on the Unity forums if( !int.TryParse( port, out portNum ) ) { Debug.LogWarning( "Given port is not a number" ); } // try to initiate a direct connection to the server else { Network.Connect( ip, portNum, password ); } } } void OnConnectedToServer() { Debug.Log( "Connected to server!" ); } void OnFailedToConnect( NetworkConnectionError error ) { Debug.Log( "Failed to connect to server: " +error.ToString() ); } } Connecting to the Master Server While we could just allow the player to enter IP addresses to connect to servers (and many games do, such as Minecraft), it's much more convenient to allow the player to browse a list of public servers. This is what the Master Server is for. Now that you can start up a server and connect to it, let's take a look at how to connect to the Master Server you downloaded earlier. First, make sure both the Master Server and Facilitator are running. I will assume you are running them on your local machine (IP is 127.0.0.1), but of course you can run these on a different computer and use that machine's IP address. Keep in mind, if you want the Master Server publicly accessible, it must be installed on a machine with a public IP address (it cannot be in a private network). Let's configure Unity to use our Master Server rather than the Unity-hosted test server. The following script configures the Master Server and Facilitator to connect to a given IP (by default 127.0.0.1): using UnityEngine; using System.Collections; public class ExampleUnityNetworkingConnectToMasterServer : MonoBehaviour { // Assuming Master Server and Facilitator are on the same machine public string MasterServerIP = "127.0.0.1"; void Awake() { // set the IP and port of the Master Server to connect to MasterServer.ipAddress = MasterServerIP; MasterServer.port = 23466; // set the IP and port of the Facilitator to connect to Network.natFacilitatorIP = MasterServerIP; Network.natFacilitatorPort = 50005; } }
Read more
  • 0
  • 0
  • 13655

article-image-augmented-reality
Packt
22 Nov 2013
6 min read
Save for later

Augmented Reality

Packt
22 Nov 2013
6 min read
(For more resources related to this topic, see here.) A quick overview of AR concepts As AR has become increasingly popular in the media over the last few years, unfortunately, several distorted notions of Augmented Reality have evolved. Anything that is somehow related to the real world and involves some computing, such as standing in front of a shop and watching 3D models wear the latest fashions, has become AR. Augmented Reality emerged from research labs a few decades ago and different definitions of AR have been produced. As more and more research fields (for example, computer vision, computer graphics, human-computer interaction, medicine, humanities, and art) have investigated AR as a technology, application, or concept, multiple overlapping definitions now exist for AR. Rather than providing you with an exhaustive list of definitions, we will present some major concepts present in any AR application. Sensory augmentation The term Augmented Reality itself contains the notion of reality. Augmenting generally refers to the aspect of influencing one of your human sensory systems, such as vision or hearing, with additional information. This information is generally defined as digital or virtual and will be produced by a computer. The technology currently uses displays to overlay and merge the physical information with the digital information. To augment your hearing, modified headphones or earphones equipped with microphones are able to mix sound from your surroundings in realtime with sound generated by your computer. Displays The TV screen at home is the ideal device to perceive virtual content, streamed from broadcasts or played from your DVD. Unfortunately, most common TV screens are not able to capture the real world and augment it. An Augmented Reality display needs to simultaneously show the real and virtual worlds. One of the first display technologies for AR was produced by Ivan Sutherlandin 1964 (named "The Sword of Damocles"). The system was rigidly mounted on the ceiling and used some CRT screens and a transparent display to be able to create the sensation of visually merging the real and virtual. Since then, we have seen different trends in AR display, going from static to wearable and handheld displays. One of the major trends is the usage of optical see-through (OST) technology. The idea is to still see the real world through a semitransparent screen and project some virtual content on the screen. The merging of the real and virtual worlds does not happen on the computer screen, but directly on the retina of your eye, as depicted in the following figure: The other major trend in AR display is what we call video see-through (VST) technology. You can imagine perceiving the world not directly, but through a video on a monitor. The video image is mixed with some virtual content (as you will see in a movie) and sent back to some standard display, such as your desktop screen, your mobile phone, or the upcoming generation of head-mounted displays as shown in the following figure: In this book, we will work on Android-driven mobile phones and, therefore, discuss only VST systems; the video camera used will be the one on the back of your phone. Registration in 3D With a display (OST or VST) in your hands, you are already able to superimpose things from your real world, as you will see in TV advertisements with text banners at the bottom of the screen. However, any virtual content (such as text or images will remain fixed in its position on the screen. The superposition being really static, your AR display will act as a head-up display (HUD), but won't really be an AR as shown in the following figure: Google Glass is an example of an HUD. While it uses a semitransparent screen like an OST, the digital content remains in a static position. AR needs to know more about real and virtual content. It needs to know where things are in space (registration) and follow where they are moving (tracking). Registration is basically the idea of aligning virtual and real content in the same space. If you are into movies or sports, you will notice that 2D or 3D graphics are superimposed onto scenes of the physical world quite often. In ice hockey, the puck is often highlighted with a colored trail. In movies such as Walt Disney'sTRON (1982 version), the real and virtual elements are seamlessly blended. However, AR differs from those effects as it is based on all of the following aspects (proposed by Ronald T. Azumain 1997): It's in 3D: In the olden days, some of the movies were edited manually to merge virtual visual effects with real content. A well-known example is Star Wars, where all the lightsaber effects have been painted by hand by hundreds of artists and, thus, frame by frame. Nowadays, more complex techniques support merging digital 3D content (such as characters or cars) with the video image (and is called match moving). AR is inherently always doing that in a 3D space. The registration happens in real time: In a movie, everything is prerecorded and generated in a studio; you just play the media. In AR, everything is in real time, so your application needs to merge, at each instance, reality and virtuality. It's interactive: In a movie, you only look passively at the scene from where it has been shot. In AR, you can actively move around, forward, and backward and turn your AR display—you will still see an alignment between both worlds. Interaction with the environment Building a rich AR application needs interaction between environments; otherwise you end up with pretty, 3D graphics that can turn boring quite fast. AR interaction refers to selecting and manipulating digital and physical objects and navigating in the augmented scene. Rich AR applications allow you to use objects which can be on your table, to move some virtual characters, use your hands to select some floating virtual objects while walking on the street, or speak to a virtual agent appearing on your watch to arrange a meeting later in the day. We will look at how some of the standard mobile interaction techniques can also be applied to AR. We will also dig into specific techniques involving the manipulation of the real world. Summary Thus we have learned about the AR concepts through this article. Resources for Article: Further resources on this subject: Marker-based Augmented Reality on iPhone or iPad [Article] Creating Dynamic UI with Android Fragments [Article] Introducing an Android platform [Article]
Read more
  • 0
  • 0
  • 30198
article-image-exploring-streams
Packt
22 Nov 2013
15 min read
Save for later

Exploring streams

Packt
22 Nov 2013
15 min read
(For more resources related to this topic, see here.) According to Bjarne Stoustrup in his book The C++ Programming Language, Third Edition: Designing and implementing a general input/output facility for a programming language is notoriously difficult... An I/O facility should be easy, convenient, and safe to use; efficient and flexible; and, above all, complete. It shouldn't surprise anyone that a design team, focused on providing efficient and easy I/O, has delivered such a facility through Node. Through a symmetrical and simple interface, which handles data buffers and stream events so that the implementer does not have to, Node's Stream module is the preferred way to manage asynchronous data streams for both internal modules and, hopefully, for the modules developers will create. A stream in Node is simply a sequence of bytes. At any time, a stream contains a buffer of bytes, and this buffer has a zero or greater length: Because each character in a stream is well defined, and because every type of digital data can be expressed in bytes, any part of a stream can be redirected, or "piped", to any other stream, different chunks of the stream can be sent do different handlers, and so on. In this way stream input and output interfaces are both flexible and predictable and can be easily coupled. Digital streams are well described using the analogy of fluids, where individual bytes (drops of water) are being pushed through a pipe. In Node, streams are objects representing data flows that can be written to and read from asynchronously. The Node philosophy is a non-blocking flow, I/O is handled via streams, and so the design of the Stream API naturally duplicates this general philosophy. In fact, there is no other way of interacting with streams except in an asynchronous, evented manner—you are prevented, by design, from blocking I/O. Five distinct base classes are exposed via the abstract Stream interface: Readable, Writable, Duplex, Transform, and PassThrough. Each base class inherits from EventEmitter, which we know of as an interface to which event listeners and emitters can be bound. As we will learn, and here will emphasize, the Stream interface is an abstract interface. An abstract interface functions as a kind of blueprint or definition, describing the features that must be built into each constructed instance of a Stream object. For example, a readable stream implementation is required to implement a public read method which delegates to the interface's internal _read method. In general, all stream implementations should follow these guidelines: As long as data exists to send, write to a stream until that operation returns false, at which point the implementation should wait for a drain event, indicating that the buffered stream data has emptied Continue to call read until a null value is received, at which point wait for a readable event prior to resuming reads Several Node I/O modules are implemented as streams. Network sockets, file readers and writers, stdin and stdout, zlib, and so on. Similarly, when implementing a readable data source, or data reader, one should implement that interface as a Stream interface. It is important to note that as of Node 0.10.0 the Stream interface changed in some fundamental ways. The Node team has done its best to implement backwards-compatible interfaces, such that (most) older programs will continue to function without modification. In this article we will not spend any time discussing the specific features of this older API, focusing on the current (and future) design. The reader is encouraged to consult Node's online documentation for information on migrating older programs. Implementing readable streams Streams producing data that another process may have an interest in are normally implemented using a Readable stream. A Readable stream saves the implementer all the work of managing the read queue, handling the emitting of data events, and so on. To create a Readable stream: var stream = require('stream'); var readable = new stream.Readable({ encoding : "utf8", highWaterMark : 16000, objectMode: true }); As previously mentioned, Readable is exposed as a base class, which can be initialized through three options: encoding: Decode buffers into the specified encoding, defaulting to UTF-8. highWaterMark: Number of bytes to keep in the internal buffer before ceasing to read from the data source. The default is 16 KB. objectMode: Tell the stream to behave as a stream of objects instead of a stream of bytes, such as a stream of JSON objects instead of the bytes in a file. Default false. In the following example we create a mock Feed object whose instances will inherit the Readable stream interface. Our implementation need only implement the abstract _read method of Readable, which will push data to a consumer until there is nothing more to push, at which point it triggers the Readable stream to emit an "end" event by pushing a null value: var Feed = function(channel) { var readable = new stream.Readable({ encoding : "utf8" }); var news = [ "Big Win!", "Stocks Down!", "Actor Sad!" ]; readable._read = function() { if(news.length) { return readable.push(news.shift() + "n"); } readable.push(null); }; return readable; } Now that we have an implementation, a consumer might want to instantiate the stream and listen for stream events. Two key events are readable and end. The readable event is emitted as long as data is being pushed to the stream. It alerts the consumer to check for new data via the read method of Readable. Note again how the Readable implementation must provide a private _read method, which services the public read method exposed to the consumer API. The end event will be emitted whenever a null value is passed to the push method of our Readable implementation. Here we see a consumer using these methods to display new stream data, providing a notification when the stream has stopped sending data: var feed = new Feed(); feed.on("readable", function() { var data = feed.read(); data && process.stdout.write(data); }); feed.on("end", function() { console.log("No more news"); }); Similarly, we could implement a stream of objects through the use of the objectMode option: var readable = new stream.Readable({ objectMode : true }); var prices = [ { price : 1 }, { price : 2 } ]; ... readable.push(prices.shift()); // } { prices : 1 } // } { prices : 2 } Here we see that each read event is receiving an object, rather than a buffer or string. Finally, the read method of a Readable stream can be passed a single argument indicating the number of bytes to be read from the stream's internal buffer. For example, if it was desired that a file should be read one byte at a time, one might implement a consumer using a routine similar to: readable.push("Sequence of bytes"); ... feed.on("readable", function() { var character; while(character = feed.read(1)) { console.log(character); }; }); // } S // } e // } q // } ... Here it should be clear that the Readable stream's buffer was filled with a number of bytes all at once, but was read from discretely. Pushing and pulling We have seen how a Readable implementation will use push to populate the stream buffer for reading. When designing these implementations it is important to consider how volume is managed, at either end of the stream. Pushing more data into a stream than can be read can lead to complications around exceeding available space (memory). At the consumer end it is important to maintain awareness of termination events, and how to deal with pauses in the data stream. One might compare the behavior of data streams running through a network with that of water running through a hose. As with water through a hose, if a greater volume of data is being pushed into the read stream than can be efficiently drained out of the stream at the consumer end through read, a great deal of back pressure builds, causing a data backlog to begin accumulating in the stream object's buffer. Because we are dealing with strict mathematical limitations, read simply cannot be compelled to release this pressure by reading more quickly—there may be a hard limit on available memory space, or other limitation. As such, memory usage can grow dangerously high, buffers can overflow, and so forth. A stream implementation should therefore be aware of, and respond to, the response from a push operation. If the operation returns false this indicates that the implementation should cease reading from its source (and cease pushing) until the next _read request is made. In conjunction with the above, if there is no more data to push but more is expected in the future the implementation should push an empty string (""), which adds no data to the queue but does ensure a future readable event. While the most common treatment of a stream buffer is to push to it (queuing data in a line), there are occasions where one might want to place data on the front of the buffer (jumping the line). Node provides an unshift operation for these cases, which behavior is identical to push, outside of the aforementioned difference in buffer placement. Writable streams A Writable stream is responsible for accepting some value (a stream of bytes, a string) and writing that data to a destination. Streaming data into a file container is a common use case. To create a Writable stream: var stream = require('stream'); var readable = new stream.Writable({ highWaterMark : 16000, decodeStrings: true }); The Writable streams constructor can be instantiated with two options: highWaterMark: The maximum number of bytes the stream's buffer will accept prior to returning false on writes. Default is 16 KB decodeStrings: Whether to convert strings into buffers before writing. Default is true. As with Readable streams, custom Writable stream implementations must implement a _write handler, which will be passed the arguments sent to the write method of instances. One should think of a Writable stream as a data target, such as for a file you are uploading. Conceptually this is not unlike the implementation of push in a Readable stream, where one pushes data until the data source is exhausted, passing null to terminate reading. For example, here we write 100 bytes to stdout: var stream = require('stream'); var writable = new stream.Writable({ decodeStrings: false }); writable._write = function(chunk, encoding, callback) { console.log(chunk); callback(); } var w = writable.write(new Buffer(100)); writable.end(); console.log(w); // Will be `true` There are two key things to note here. First, our _write implementation fires the callback function immediately after writing, a callback that is always present, regardless of whether the instance write method is passed a callback directly. This call is important for indicating the status of the write attempt, whether a failure (error) or a success. Second, the call to write returned true. This indicates that the internal buffer of the Writable implementation has been emptied after executing the requested write. What if we sent a very large amount of data, enough to exceed the default size of the internal buffer? Modifying the above example, the following would return false: var w = writable.write(new Buffer(16384)); console.log(w); // Will be 'false' The reason this write returns false is that it has reached the highWaterMark option—default value of 16 KB (16 * 1024). If we changed this value to 16383, write would again return true (or one could simply increase its value). What to do when write returns false? One should certainly not continue to send data! Returning to our metaphor of water in a hose: when the stream is full, one should wait for it to drain prior to sending more data. Node's Stream implementation will emit a drain event whenever it is safe to write again. When write returns false listen for the drain event before sending more data. Putting together what we have learned, let's create a Writable stream with a highWaterMark value of 10 bytes. We will send a buffer containing more than 10 bytes (composed of A characters) to this stream, triggering a drain event, at which point we write a single Z character. It should be clear from this example that Node's Stream implementation is managing the buffer overflow of our original payload, warning the original write method of this overflow, performing a controlled depletion of the internal buffer, and notifying us when it is safe to write again: var stream = require('stream'); var writable = new stream.Writable({ highWaterMark: 10 }); writable._write = function(chunk, encoding, callback) { process.stdout.write(chunk); callback(); } writable.on("drain", function() { writable.write("Zn"); }); var buf = new Buffer(20, "utf8"); buf.fill("A"); console.log(writable.write(buf.toString())); // false The result should be a string of 20 A characters, followed by false, then followed by the character Z. The fluid data in a Readable stream can be easily redirected to a Writable stream. For example, the following code will take any data sent by a terminal (stdin is a Readable stream) and pass it to the destination Writable stream, stdout: process.stdin.pipe(process.stdout); Whenever a Writable stream is passed to a Readable stream's pipe method, a pipe event will fire. Similarly, when a Writable stream is removed as a destination for a Readable stream, the unpipe event fires. To remove a pipe, use the following: unpipe(destination stream)   Duplex streams A duplex stream is both readable and writeable. For instance, a TCP server created in Node exposes a socket that can be both read from and written to: var stream = require("stream"); var net = require("net"); net .createServer(function(socket) { socket.write("Go ahead and type something!"); socket.on("readable", function() { process.stdout.write(this.read()) }); }) .listen(8080); When executed, this code will create a TCP server that can be connected to via Telnet: telnet 127.0.0.1 8080 Upon connection, the connecting terminal will print out Go ahead and type something! —writing to the socket. Any text entered in the connecting terminal will be echoed to the stdout of the terminal running the TCP server (reading from the socket). This implementation of a bi-directional (duplex) communication protocol demonstrates clearly how independent processes can form the nodes of a complex and responsive application, whether communicating across a network or within the scope of a single process. The options sent when constructing a Duplex instance merge those sent to Readable and Writable streams, with no additional parameters. Indeed, this stream type simply assumes both roles, and the rules for interacting with it follow the rules for the interactive mode being used. As a Duplex stream assumes both read and write roles, any implementation is required to implement both _write and _read methods, again following the standard implementation details given for the relevant stream type. Transforming streams On occasion stream data needs to be processed, often in cases where one is writing some sort of binary protocol or other "on the fly" data transformation. A Transform stream is designed for this purpose, functioning as a Duplex stream that sits between a Readable stream and a Writable stream. A Transform stream is initialized using the same options used to initialize a typical Duplex stream. Where Transform differs from a normal Duplex stream is in its requirement that the custom implementation merely provide a _transform method, excluding the _write and _read method requirement. The _transform method will receive three arguments, first the sent buffer, an optional encoding argument, and finally a callback which _transform is expected to call when the transformation is complete: _transform = function(buffer, encoding, cb) { var transformation = "..."; this.push(transformation) cb(); } Let's imagine a program that wishes to convert ASCII (American Standard Code for Information Interchange) codes into ASCII characters, receiving input from stdin. We would simply pipe our input to a Transform stream, then piping its output to stdout: var stream = require('stream'); var converter = new stream.Transform(); converter._transform = function(num, encoding, cb) { this.push(String.fromCharCode(new Number(num)) + "n") cb(); } process.stdin.pipe(converter).pipe(process.stdout); Interacting with this program might produce an output resembling the following: 65 A 66 B 256 A 257 a   Using PassThrough streams This sort of stream is a trivial implementation of a Transform stream, which simply passes received input bytes through to an output stream. This is useful if one doesn't require any transformation of the input data, and simply wants to easily pipe a Readable stream to a Writable stream. PassThrough streams have benefits similar to JavaScript's anonymous functions, making it easy to assert minimal functionality without too much fuss. For example, it is not necessary to implement an abstract base class, as one does with for the _read method of a Readable stream. Consider the following use of a PassThrough stream as an event spy: var fs = require('fs'); var stream = new require('stream').PassThrough(); spy.on('end', function() { console.log("All data has been sent"); }); fs.createReadStream("./passthrough.js").pipe(spy).pipe(process.std out); Summary As we have learned, Node's designers have succeeded in creating a simple, predictable, and convenient solution to the very difficult problem of enabling efficient I/O between disparate sources and targets. Its abstract Stream interface facilitates the instantiation of consistent readable and writable interfaces, and the extension of this interface into HTTP requests and responses, the filesystem, child processes, and other data channels makes stream programming with Node a pleasant experience. Resources for Article: Further resources on this subject: So, what is Node.js? [Article] Getting Started with Zombie.js [Article] So, what is KineticJS? [Article]
Read more
  • 0
  • 0
  • 13701

article-image-data-acquisition-and-mapping
Packt
22 Nov 2013
9 min read
Save for later

Data Acquisition and Mapping

Packt
22 Nov 2013
9 min read
Machine listening We can acquire data from various sources. In a visualization context, however, we may encounter situations wherein we will need to control some elements of an animation with respect to some particular characteristic of a signal, for example, their amplitude or their frequency. Yet, these kinds of information are attributes of the signal, rather than parts of it. In other words, we need something to happen not with respect to some existent data (that is, our signal in this context) but with respect to certain characteristics of a data flow. Consider that an audio signal is completely unaware of how loud it is or of what its frequency is. Remember that audio signals are merely streams of numbers and that sounds are merely fluctuations of air pressure. The reason we understand sounds as having loudness or pitch, is because our auditory apparatus analyzes them and provides the brain with information on certain sonic qualities. Further, more sophisticated perceptual and cognitive processes perform additional kinds of analyses to extract as well as attribute information and meanings, so that we perceptually decipher what we hear. Likewise, we can say that a signal is periodic and has a certain frequency, only if we somehow analyze it. Remember that the output of a sinusoidal wave at a frequency of 200 Hz is just a flow of numbers between ±1. The datum 200 is not part of this signal, so the only way to make something happen with respect to this number is to actually generate it by means of analyzing the audio signal against its frequency. The task of retrieving statistical and other kinds of information from audio signals is generally referred to as machine listening. Machine listening is, in essence, to analyze signals in order to generate information that represent certain qualities of these signals. To properly understand and evaluate the kind of information we may get from some machine listening algorithm, it is worth distinguishing briefly the different kinds of properties a signal may have. Acoustic properties refer to physical properties of sound, and consequently of audio signals, particularly qualities such as amplitude, frequency, and spectrum. Psychoacoustic properties refer to low-level perceptional properties of audio signals, such as loudness, pitch, and timbre. Psychoacoustic properties are fundamentally different than their acoustic equivalents, the latter being intrinsically linked to perception. For instance, loudness refers to how loud something sounds, while the amplitude stands for the actual amount of the displacement of the air particles that occurs in the physical space. It has to be stressed that the various psychoacoustic qualities do relate and depend upon the acoustic properties of sound; nonetheless, the relationships are very complex and not that straightforward as they may appear to be. For example, loudness does not depend exclusively upon amplitude, but it also depends upon frequency, spectral content, and even upon a series of psychological and other factors. We can also speak of several families of higher-level perceptional properties, such as musical ones (scale, tonality, rhythm, genre, expressivity, and so on), cognitive ones (semantics, symbolical signification, and so on), and psychological ones (irritability, entertainability, ability to cause relaxation, and so on). Again, such properties may depend or relate to some extent to the acoustic or psychoacoustic qualities of sound; yet the inter-relationships may be extremely complex and even not fully understood in certain cases. Machine listening algorithms are not limited only to simple acoustic properties of a signal; sophisticated algorithms have been proposed for more complex problems as well such as musical style recognition and rhythm extraction. As far as musical qualities are concerned, the more specialized term musical information retrieval is sometimes encountered too. In SuperCollider, we can easily perform basic audio analyses to retrieve information on both physical as well as certain perceptional properties of audio signals using the available machine listening UGens, the most important of which will be discussed immediately. Music Information Retrieval (MIR) is an interdisciplinary field of science dealing with how to retrieve and classify information from music Tracking amplitude and loudness We can also use the Peak UGen, which will return the maximum peak amplitude every time it receives a trigger or the PeakFollower UGen which smoothly decays from the maximum value by some specified decay time. To track the minimum or the maximum value of a signal, we can use the RunningMin or RunningMax UGens. To track Root Mean Square (RMS) amplitude, we can use the RunningSum UGen. The following example shows how to use these UGens: (// tracking amplitude { var sound = SinOsc.ar(mul:LFNoise2.kr(1).range(0,1)); // source RunningSum.rms(sound,100).poll(label:'rms'); // rms Amplitude.kr(sound).poll(label:'peak'); // peak Peak.kr(sound, Impulse.kr(1)).poll(label:'peak_trig'); // peak when triggered PeakFollower.kr(sound).poll(label:'peak_dec'); // peak with decay RunningMin.kr(sound).poll(label:'min'); // minimum RunningMax.kr(sound).poll(label:'max'); // maximum Out.ar(0,sound); // write to output }.play; ) Sometimes we may want something to happen when a signal is silent or at least when it is below a certain level. In such cases, we can use DetecteSilence. There also exists a Loudness UGent which will estimate loudness in Sones (the measure of loudness). It is designed to analyze spectra and requires an FFT window of size 1024 for sampling rates of 44100 or 48000 and of the size 2048 for 88200 or 96000, respectively. For example: ( // track loudness { var sound, loudness; sound = SinOsc.ar(LFNoise2.ar(1).range(100,10000), mul:LFNoise0.ar(1).range(0,1)); // source loudness = FFT(LocalBuf(1024),sound); // sampling rates of 44.1/48K // loudness = FFT(LocalBuf(1024),sound); // sampling rates of 88.2/96K loudness = Loudness.kr(loudness).poll(label:loudness); Out.ar(0, sound); }.play; ) Tracking frequency As far as frequency is concerned, there are a number of relevant UGens, each of them implemented differently. The most simple one is ZeroCrossing, which will estimate the frequency by keeping track of how often an input signal crosses the horizontal axis, which represents 0 in terms of amplitude. Pitch is a more accurate frequency tracker, which also allows for some tweaking. Note that, regardless of its name, it performs frequency tracking rather than pitch tracking, the latter also depends on a series of other factors. More advanced frequency trackers are Tartini (which is based on the method used in the homonymous open source pitch tracker) and Qitch (which has to be used along with one of the special auxiliary WAV files it is distributed with). Tartini and Qitch are not included in the standard SuperCollider distribution but on the SC3Plugins extension bundle (available at http://sc3-plugins.sourceforge.net/). Pitch, Tartini, and Qitch will all return an array of instances of OutProxy obtaining both the estimated frequency as well as a flag of 1 or 0 to denote whether they successfully tracked some frequency or not. When attempting to track frequency we should always bear in mind that the former being a complicated process, not all trackers would work equally well for all kinds of signals. For example: ( // frequency tracking var qitchBuffer = Buffer.read (Server.default,"/Users/marinos/Library/Application Support/ SuperCollider/Extensions/SC3plugins/PitchDetection/extraqitchfiles/ QspeckernN2048SR44100.wav"); // path to auxiliary wav file for Qitch { // a complex signal var sound = Saw.ar(LFNoise2.ar(1).range(500,1000).poll(label: ActualFrequency)) + WhiteNoise.ar(0.4); ZeroCrossing.ar(sound).poll(label:ZeroCross); Pitch.kr(sound).poll(label:Pitch); Tartini.kr(sound).poll(label:Tartini); Qitch.kr(sound,qitchBuffer).poll(label:Qitch); Out.ar(0,sound!2); }.play; ) For this signal, Qitch is probably the most reasonable choice, judging by the output on my machine: ActualFrequency: 864.222 ZeroCross: 6368.27 Pitch: 171.704 Pitch: 1 Tartini: 95.0917 Tartini: 1 Qitch: 845.466 Qitch: 1 Timbre analysis and feature detection Timbre is a psycho-acoustic quality, and refers to what makes sounds distinct even if they have the same loudness and pitch. Of course this is a broad oversimplification of a very complex subject; in reality there isn't even a consensus on what exactly timbre stands for. While timbre has been proposed to depend on several qualities, in a machine listening context timbre recognition is almost exclusively based on analyzing spectra. Herein, we will focus on how to broadly detect several spectral features, rather than timbre per se, which is a rather indefinite quality. By the term feature we refer to anything that could be characteristic about a signal's spectral characteristics. In SuperCollider there is a plethora of relevant UGens, both in the standard distribution as well as in extension libraries. Of the most useful are SpecCentroid and ScpeFlatness used to calculate the spectral centroid and the spectral flatness, respectively. The former roughly stands for the most perceptually prominent frequency range in our signal while the latter is an indicator of how complicated our signal is (for example, for a sinusoid it would be 0 while for white noise close to 1). The SpecPcile UGen will calculate the cumulative distribution of a spectrum, and given a percentile of spectral energy as an argument, will return that frequency from which the given percentile of spectral energy lies below. In the SC3Plugins extensions bundle, we will also find the FFTCrest UGen, which will calculate the spectral crest of a signal, which, in short, indicates how flat or peaky a signal is, and the SensoryDissonance UGen, which will attempt to calculate how dissonant a signal is (with 1 being totally dissonant and 0 being totally consonant). The FFTSpread UGen measures the spectral spread of a signal, that is how wide or narrow its spectrum is and FFTSlope calculates the slope of the linear correlation line derived from the spectral magnitudes. Finally, the Goertzel UGen calculates the magnitude and phase at a single specified frequency. For example: ( // feature extraction { var sound = SinOsc.ar(240,mul:0.5) + Resonz.ar(ClipNoise.ar,2000,0.6,mul:SinOsc.kr(0.05).range(0,0.5)) + Saw.ar(2000,mul:SinOsc.kr(0.1).range(0,0.3)); var fft = FFT(LocalBuf(2048),sound); // a complex signal SpecCentroid.kr(fft).poll(label:Centroid); SpecFlatness.kr(fft).poll(label:Flatness); SpecPcile.kr(fft,0.8).poll(label:Percentile); FFTCrest.kr(fft,1800,2200).poll(label:Crest); SensoryDissonance.kr(fft).poll(label:Dissonance); Out.ar(0,sound!2); }.play; ) Summary In this article, we discussed about machine listening techniques and ways to retrieve information from audio signals. Resources for Article: Further resources on this subject: Data Profiling with IBM Information Analyzer [Article] Preparation Analysis of Data Source [Article] Spatial Data Services [Article]
Read more
  • 0
  • 0
  • 1323

article-image-preparation-analysis-data-source
Packt
22 Nov 2013
6 min read
Save for later

Preparation Analysis of Data Source

Packt
22 Nov 2013
6 min read
(For more resources related to this topic, see here.) List of data sources Here, the wide varieties of data sources that are supported in the PowerPivot interface are given in brief. The vital part is to install providers such as OLE DB and ODBC that support the existing data source, because when installing the PowerPivot add-in, it will not install the provider too and some providers might already be installed with other applications. For instance, if there is a SQL Server installed in the system, the OLE DB provider will be installed with the SQL Server, so that later on it wouldn't be necessary to install OLE DB while adding data sources by using the SQL Server as a data source. Hence, make sure to verify the provider before it is added as a data source. Perhaps the provider is required only for relational database data sources. Relational database By using RDBMS, you can import tables and views into the PowerPivot workbook. The following is a list of various data sources: Microsoft SQL Server Microsoft SQL Azure Microsoft SQL Server Parallel Data Warehouse Microsoft Access Oracle Teradata Sybase Informix IBM DB2 Other providers (OLE DB / ODBC) Multidimensional sources Multidimensional data sources can only be added from Microsoft Analysis Services (SSAS). Data feeds The three types of data feeds are as follows: SQL Server Reporting Service (SSRS) Azure DataMarket dataset Other feeds such as atom service documents and single data feed Text files The two types of text files are given as follows: Excel file Text file Purpose of importing data from a variety of sources In order to make a decision about a particular subject area, you should analyze all the required data that is relevant to the subject area. If the data is stored in a variety of data sources, importing the data from different data sources has to be done. If all the data is only in one data source, only the data needs to be imported from the required tables for the subject and then various types of analysis can be done. The reason why users need to import data from different data sources is that they would then have an ample amount of data when they need to make any analysis. Another generic reason would be to cross-analyze data from different business systems such as Customer Relationship Management (CRM) and Campaign Management System (CMS). Data sourcing from only one source wouldn't be as sophisticated as an analysis done from different data sources as the amount of data from which the analysis was done for multisourced data is more detailed than the data from only a single source. It also might reveal conflicts between multiple data sources. In real time, usually in the e-commerce industry, blogs, and forum websites wouldn't ask for more details about customers at the time of registration, because the time consumed for long registrations would discourage the user, leading to cancellation the of the order. For instance, the customer table that would be stored in the database of an e-commerce industry would contain the following attributes: Customer FirstName LastName E-mail BirthDate Zip Code Gender However, this kind of industry needs to know their customers more in order to increase their sales. Since the industry only saves a few attributes about the customer during registration, it is difficult to track down the customers and it is even more difficult to advertise according to individual customers. Therefore, in order to find some relevant data about the customers, the e-commerce industries try to make another data source using the Internet or other types of sources. For instance, by using the Postcode given by the customer during registration, the user can get Country | State | City from various websites and then use the information obtained to make a table format either in Excel or CSV, as follows: Location Postcode City State Country So, finally the user would have two sources—one source is from the user's RDBMS database and the other source is from the Excel file created later—both of these can be used in order to make a customer analysis based on their location. General overview of ETL The main goal is to facilitate the development of data migration applications by applying data transformations. Extraction Transform Load (ETL) comprises of the first step in a data warehouse process. It is the most important and the hardest part, because it determines the quality of the data warehouse and the scope of the analyses the users will be able to build upon it. Let us discuss in detail what ETL is. The first substep is extraction. As the users would want to regroup all the information a company has, they will have to collect the data from various heterogeneous sources (operational data such as databases and/or external data such as CSV files) and various models (relational, geographical, web pages, and so on). The difficulty starts here. As data sources are heterogeneous, formats are usually different, and the users would have different types of data models for similar information (different names and/or formats) and different keys for the same objects. These are some of the main problems. The aim of the transformation task is to correct such problems, as much as possible. Let us now see what a transformation task is. The users would need to transform heterogeneous data of different sources into homogeneous data. Here are some examples of what they can do: Extraction of the data sources Identification of relevant data sources Filtering of non-admissible data Modification of formats or values Summary This article shows the users how to prepare data for analysis. We also covered different types of data sources that can be imported into the PowerPivot interface. Some information about the provider and a general overview of the ETL process has also been given. Users now know how the ETL process works in the PowerPivot interface; also, users were shown an introduction and the advantages of DAX. Resources for Article: Further resources on this subject: Creating a pivot table [Article] What are SSAS 2012 dimensions and cube? [Article] An overview of Oracle Hyperion Interactive Reporting [Article]
Read more
  • 0
  • 0
  • 1468
article-image-executing-desired-state-configuration-advanced
Packt
22 Nov 2013
3 min read
Save for later

Executing Desired State Configuration (Advanced)

Packt
22 Nov 2013
3 min read
(For more resources related to this topic, see here.) DSC approaches declarative syntax that describes what needs to be done rather than covering imperative syntax that specifies how a task can be performed. Getting ready To use DSC, first define a desired configuration. Like functions, configurations in DSC can be defined in the Windows PowerShell language by using the Configuration keyword and stored in script (.ps1) or module (.psm1) files. Also, similar to functions, configurations need to be defined and then run. How to do it... To use a configuration, invoke the Configuration block the same way you would invoke a Windows PowerShell function, passing in any expected parameters you have defined (two in the preceding sample). For example, in this case, the MyWebConfig configuration can be invoked as follows: PS C :>MyWebConf -MachineName $env:COMPUTERNAME –WebsitePath \PSShareMyWebSites This will create a folder with the same name as your configuration name and will contain our MOF output file. The following command creates an MOF file known as the configuration instance document. Path represents the target directory where your MOF files are located. Wait causes the execution of the DSC resources to run in the background, that is, an interactive process. PS C :>Start-DscConfiguration -Path . MyWebConf –Wait –Verbose How it works... Each Configuration block must have at least one Node block. Each Node block can have one or more resource provider blocks. You can use the same role provider more than once in the same Node block. In addition to new language keywords, DSC includes the following set of CMDLETs for managing configurations: Start-DscConfiguration: This CMDLET deploys a configuration to one or more target nodes and applies the configuration on those nodes by using the local configuration manager Get-DscConfiguration: This CMDLET returns the current configuration from one or more target machines: PS C :>$Sess = New-CimSession -ComputerName localhost PS C :>Get-DscConfiguration –CimSession $Sess Restore-DscConfiguration: This CMDLET restores the current configuration from one or more target machines: PS C :>$Sess = New-CimSession -ComputerName localhost PS C :>Restore-DscConfiguration –CimSession $Sess There's more… There is one more CMDLET that helps to detect the configuration drift: Test-DscConfiguration: This CMDLET checks for one or more target nodes and returns a Boolean value indicating whether the current desired state matches the actual state. Have a look at the following command: PS C :>$session = New-CimSession -ComputerName localhost PS C :>Test-DscConfiguration –CimSession $session This will either return True when the current and actual configuration matches or False if there's a mismatch Summary In this article, we learned about how to execute the new feature introduced with the release of Windows PowerShell v4.0—Desired State Configuration. Resources for Article: Further resources on this subject: Exchange Server 2010 Windows PowerShell: Working with Address Lists [Article] So, what is PowerShell 3.0 WMI? [Article] Inventorying Servers with PowerShell [Article]
Read more
  • 0
  • 0
  • 1658

article-image-configuring-roles-high-availability
Packt
22 Nov 2013
8 min read
Save for later

Configuring Roles for High Availability

Packt
22 Nov 2013
8 min read
(For more resources related to this topic, see here.) Now it's down to the real deal—the bits and bytes. In this article, we will go through different roles and services and how they can be deployed as highly available for the clients. We will also walk through some examples of how a simple site design might look like for different scenarios. First, let's take a look at roles within Configuration Manager that cannot be deployed as highly available and what options we have. It is important to note that when installing a site system role on another server Configuration Manager, the installation files are installed on the first available NTFS disk with maximum available disk space. If you wish to exclude a disk, create an empty file called no_sms_on_drive.sms on the root folder of the drive. Also, Configuration Manager uses the site server computer account to try to install the role on a remote server (so, this account needs to be added to local administrator on the server) unless we specify a domain account (which also needs to be a local admin) when we run the setup. Site Server The first and foremost important role that cannot be deployed as highly available is the Site Server role. It does not support network load balancing or other clustering features available in Windows Server. In order to make sure that you can restore the site in case of disaster, you have to use the built-in backup tasks. Endpoint Protection The Endpoint Protection role allows for management of endpoint protection agents. If this role is unavailable for the clients, they will still continue to run as before but you cannot update the policy centrally. This role is not seen as critical for Configuration Manager and cannot be deployed as highly available. If something happens to your endpoint server, just reinstall the role on another server. It is important to know that this role can only be deployed once at the top of your hierarchy; so, if we have a CAS, we would place this role there. Same is true if we have a standalone primary site with secondary sites. Asset Intelligence Synchronization Point The Asset Intelligence Synchronization Point role connects to System Center online (a cloud service from Microsoft) in order to update the asset catalog with the latest information. If this role becomes unavailable, the site will just store data locally until it becomes online and is able to synchronize again. This role cannot be deployed as highly available, and it is important to note that this role can only be installed on the CAS role (if present) or on a standalone primary site as well. Fallback Status Point The Fallback Status Point role is used to monitor client deployments and the clients that are unable to connect to their management point, or if there is something wrong with the management point. All communication between clients and this role happens over regular HTTP requests and therefore this role should be placed on a standalone server. All status messages that get sent to the Fallback Status Point role will be registered in the logfiles fspisapi.log and fspmgr.log. This role is not considered critical and cannot be deployed as highly available. It is important to note that clients can only be assigned to one fallback point and the assignment happens during client installation. Out of Band Service Point The Out of Band Service Point role is used to manage and provision clients which have Intel vPRO or Intel AMT functionality, which allows Configuration Manager to manage clients before the operating system is active. This role cannot be deployed as highly available. Secondary sites Secondary sites by default contain one distribution point and one management point, and as mentioned previously, should only be used when you need control on how data is moved up the hierarchy. A secondary site does not support more than one management point. Data for a secondary site is primarily located in the primary site database. You can recover a secondary site by reinstalling it from its primary site. Windows Intune Connector The Windows Intune Connector role is a new role with Service Pack 1. It allows for mobile management from Configuration Manager using Windows Intune. This role can only be installed once and it needs to be in the top-level site in the hierarchy. So, now that we have covered the roles that cannot be deployed as highly available, let us go forward with the roles that we can deploy as highly available. Management Point The Management Point role is used by clients for policy downloads, data reporting, and is the primary role for client-to-site communication. When a client is assigned to a site (depending on how you have defined the client installation parameters), it will download a list of all the available management points in the site and get assigned to an appropriate one. The clients will connect to another management point if one of the following happens: Client restarts Network changes 25 hours have passed The contact with its primary management point is lost It will also check if the management point is Intranet-only or is available from the Internet, and if it communicates using HTTP or HTTPS. (Remember that if the management point is set up on the Internet, only HTTPS traffic is allowed.) Now besides manual installation, clients will look for a management point within one of the following locations if something from one of the points mentioned above happens: When installing a client manually, you can append the parameters CCMSetup.exe /mp:MPFQDN SMSSITECODE=SITECODE FSP=FSPFQDN. to define which management point to contact, which site to register to, and which FSP to contact. Site information: After a client has successfully signed up with a site, it will download a list of available management points which it stores in its cache. If it does not have any other management point available, it will continue looking in other locations. Active Directory: This only applies if you have extended the schema. Configuration Manager can publish management points directly under System Management Container with the site code. (This can be disabled under Administration | Sites | Site Properties | Configure Site Components | Management Point.) Extending the schema expands the Active Directory schema with specific Configuration Manager attributes. This allows clients to locate specific Configuration Manager server roles such as Management Points, boundaries, and its site from Active Directory. This is an irreversible action, but it allows for clients to easily locate site resources. The site code and the management points that are available within the site will be stored under Active Directory Users and Computer | Domain | System | System Management. As you can see in the following screenshot, the clients will look here if there is another management point available within this container: DNS: Clients will write a DNS query for service locator point record within the current domain. All management points will appear within the DNS zone as a prefix _mssms_mp_SITECODE, with a port number, and which host it is pointing to. These DNS records can also be automatically be published by Configuration Manager and can be disabled from the same location as Active Directory publishing. The following screenshot shows the DNS zone showing management point: WINS: If your organization is using WINS, Configuration Manager can also publish site information there as well. If a client manages to find another management point within one of the locations mentioned in the preceding list, it will try to establish a connection with it. By installing multiple management points within a site, you will gain redundant communication points for the clients within a site. Since clients have multiple places to get the information, you have multiple fallback points. There is no extra configuration required for the management points to become highly available after they are installed. It is also important to note that by default the management point talks directly to the site database server, to query for data and to publish data. If we want to reduce the CPU processing against the site database, we can set up a management point against a SQL replica. This requires that we set up a site server database with the same SQL replication installed for the site replica database. When we set up this management point, we will always query the replica database for information, but since the replication happens one-way (site to replica) the management point will always need to contact the site database to write data to the site. But we can set up multiple management points to talk to the SQL replica server, this will help to offload the site-server database. Summary In this article, we went through different roles of Configuration Manager and how we can deploy them as highly available. Resources for Article: Further resources on this subject: Client-Side Endpoint Protection Tasks in Microsoft SCEP 2012 [Article] Testing Workflows for Microsoft Dynamics AX 2009 Administration [Article] Microsoft LightSwitch Application using SQL Azure Database [Article]
Read more
  • 0
  • 0
  • 1453
Modal Close icon
Modal Close icon