Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-managing-it-portfolio-using-troux-enterprise-architecture
Packt
12 Aug 2010
16 min read
Save for later

Managing the IT Portfolio using Troux Enterprise Architecture

Packt
12 Aug 2010
16 min read
(For more resources on Troux, see here.) Managing the IT Portfolio using Troux Enterprise Architecture Almost every company today is totally dependent on IT for day-to-day operations. Large companies literally spend billions on IT-related personnel, software, equipment, and facilities. However, do business leaders really know what they get in return for these investments? Upper management knows that a successful business model depends on information technology. Whether the company is focused on delivery of services or development of products, management depends on its IT team to deliver solutions that meet or exceed customer expectations. However, even though companies continue to invest heavily in various technologies, for most companies, knowing the return-on-investment in technology is difficult or impossible. When upper management asks where the revenues are for the huge investments in software, servers, networks, and databases, few IT professionals are able to answer. There are questions that are almost impossible to answer without guessing, such as: Which IT projects in the portfolio of projects will actually generate revenue? What are we getting for spending millions on vendor software? When will our data center run out of capacity? This article will explore how IT professionals can be prepared when management asks the difficult questions. By being prepared, IT professionals can turn conversations with management about IT costs into discussions about the value IT provides. Using consolidated information about the majority of the IT portfolio, IT professionals can work with business leaders to select revenue-generating projects, decrease IT expenses, and develop realistic IT plans. The following sections will describe what IT professionals can do to be ready with accurate information in response to the most challenging questions business leaders might ask. Management repositories IT has done a fine job of delivering solutions for years. However, pressure to deliver business projects quickly has created a mentality in most IT organizations of "just put it in and we will go back and do the clean up later." This has led to a layering effect where older "legacy" technology remains in place, while new technology is adopted. With this complex mix of legacy solutions and emerging technology, business leaders have a hard time understanding how everything fits together and what value is provided from IT investments. Gone are the days when the Chief Information Officer (CIO) could say "just trust me" when business people asked questions about IT spending. In addition, new requirements for corporate compliance combined with the expanding use of web-based solutions makes managing technology more difficult than ever. With the advent of Software-as-a-Service (SaaS) or cloud computing, the technical footprint, or ecosystem, of IT has extended beyond the enterprise itself. Virtualization of platforms and service-orientation adds to the mind-numbing mix of technologies available to IT. However, there are many systems available to help companies manage their technological portfolio. Unfortunately, multiple teams within the business and within IT see the problem of managing the IT portfolio differently. In many companies, there is no centralized effort to gather and store IT portfolio information. Teams with a need for IT asset information tend to purchase or build a repository specific to their area of responsibility. Some examples of these include: Business goals repository Change management database Configuration management database Business process management database Fixed assets database Metadata repository Project portfolio management database Service catalog Service registry While each of these repositories provides valuable information about IT portfolios, they are each optimized to meet a specific set of requirements. The following table shows the main types of information stored in each of these repositories along with a brief statement about its functional purpose: Repository Main content Main purpose Business goals Goal statements and assignments Documents business goals and who is responsible Change management database Change request tickets, application owners Captures change requests and who can authorize change Configuration management database Identifies actual hardware and software in use across the enterprise Supports Information Technology Infrastructure Library (ITIL) processes Business process management database Business processes, information flows, and process owners Used to develop applications and document business processes Fixed assets database Asset identifiers for hardware and software, asset life, purchase cost, and depreciation amounts Documents cost and depreciable life of IT assets Metadata repository Data about the company databases and files Documents the names, definitions, data types, and locations of the company data Project portfolio management database Project names, classifications, assignments, business value and scope Used to manage IT workload and assess value of IT projects to the business Service catalog Defines hardware and compatible software available for project use Used to manage hardware and software implementations assigned to the IT department Service registry Names and details of reusable software services Used to manage, control, and report on reusable software It is easy to see that while each of these repositories serves a specific purpose, none supports an overarching view across the others. For example, one might ask: How many SQL Server databases do we have installed and what hardware do they run on? To answer this question, IT managers would have to extract data from the metadata repository and combine it with data from the Configuration Management Database (CMDB). The question could be extended: How much will it cost in early expense write-offs if we retire the SQL Server DB servers into a new virtual grid of servers? To answer this question, IT managers need to determine not only how many servers host SQL Server, but how old they are, what they cost at purchase time, and how much depreciation is left on them. Now the query must span at least three systems (CMDB, fixed assets, and metadata repository). The accuracy of the answer will also depend on the relative validity of the data in each repository. There could be overlapping data in some, and outright errors in others. Changing the conversation When upper management asks difficult questions, they are usually interested in cost, risk management, or IT agility. Not knowing a great deal about IT, they are curious about why they need to spend millions on technology and what they get for their investments. The conversation ends up being primarily about cost and how to reduce expenses. This is not a good position to be in if you are running a support function like Enterprise Architecture. How can you explain IT investments in a way that management can understand? If you are not prepared with facts, management has no choice but to assume that costs are out of control and they can be reduced, usually by dramatic amounts. As a good corporate citizen, it is your job to help reduce costs. Like everyone in management, getting the most out of the company's assets is your responsibility. However, as we in IT know, it's just as important to be ready for changes in technology and to be on top of technology trends. As technology leaders, it is our job to help the company stay current through investments that may pay off in the future rather than show an immediate return. The following diagram shows various management functions and technologies that are used to manage the business of IT: The dimensions of these tools and processes span systems that run the business to change the business and from the ones using operational information to using strategic information. Various technologies that support data about IT assets are shown. These include: Business process analytics and management information Service-oriented architecture governance Asset-liability management Information technology systems management Financial management information Project portfolio and management information The key to changing the conversation about IT is having the ability to bring the information of these disciplines into a single view. The single view provides the ability to actually discuss IT in a strategic way. Gathering data and reporting on the actual metrics of IT, in a way business leaders can understand, supports strategic planning. The strategic planning process combined with fact-based metrics establishes credibility with upper management and promotes improved decision making on a daily basis. Troux Technologies Solving the IT-business communication problem has been difficult until recently. Troux Technologies (www.troux.com) developed a new open-architected repository and software solution, called the Troux Transformation Platform, to help IT manage the vast array of technology deployed within the company. Troux customers use the suite of applications and advanced integration platform within the product architecture to deliver bottom-line results. By locating where IT expenses are redundant, or out-of-step with business strategy, Troux customers experience significant cost savings. When used properly, the platform also supports improved IT efficiency, quicker response to business requirements, and IT risk reduction. In today's globally-connected markets, where shocks and innovations happen at an unprecedented rate, antiquated approaches to Strategic IT Planning and Enterprise Architecture have become a major obstruction. The inability of IT to plan effectively has driven business leaders to seek solutions available outside the enterprise. Using SaaS or Application Service Providers (ASPs) to meet urgent business objectives can be an effective means to meet short-term goals. However, to be complete, even these solutions usually require integration with internal systems. IT finds itself dealing with unspecified service-level requirements, developing integration architectures, and cleaning up after poorly planned activities by business leaders who don't understand what capabilities exist within the software running inside the company. A global leader in Strategic IT Planning and Enterprise Architecture software, Troux has created an Enterprise Architecture repository that IT can use to put itself at the center of strategic planning. Troux has been successful in implementing its repository at a number of companies. A partial list of Troux's customers can be found on the website. There are other enterprise-level repository vendors on the market. However, leading analysts, such as The Gartner Group and Forrester Research, have published recent studies ranking Troux as a leader in the IT strategy planning tools space. Troux Transformation Platform Troux's sophisticated integration and collaboration capabilities support multiple business initiatives such as handling mergers, aligning business and IT plans, and consolidating IT assets. The business-driven platform provides new levels of visibility into the complex web of IT resources, programs, and business strategy so the business can see instantly where IT spending and programs are redundant or out-of-step with business strategy. The business suite of applications helps IT to plan and execute faster with data assimilated from various trusted sources within the company. The platform provides information necessary to relevant stakeholders such as Business Analysts, Enterprise Architects, The Program Management Office, Solutions Architects, and executives within the business and IT. The transformation platform is not only designed to address today's urgent cost-restructuring agendas, but it also introduces an ongoing IT management discipline, allowing EA and business users to drive strategic growth initiatives. The integration platform provides visibility and control to: Uncover and fix business/IT disconnects: This shows how IT directly supports business strategies and capabilities, and ensures that mismatched spending can be eliminated. Troux Alignment helps IT think like a CFO and demonstrate control and business purpose for the billions that are spent on IT assets, by ensuring that all stakeholders have valid and relevant IT information. Identify and eliminate redundant IT spending: This uncovers the many untapped opportunities with Troux Optimization to free up needless spend, and apply it either to the bottom line or to support new business initiatives. Speed business response and simplify IT: This speeds the creation and deployment of a set of standard, reusable building blocks that are proven to work in agile business cycles. Troux Standards enables the use of IT standards in real time, thereby streamlining the process of IT governance. Accelerate business transformation for government agencies: This helps federal agencies create an actionable Enterprise Architecture and comply with constantly changing mandates. Troux eaGov automatically identifies opportunities to reduce costs to business and IT risks, while fostering effective initiative planning and execution within or across agencies. Support EA methodology: Companies adopting The Open Group Architecture Framework (TOGAF™) can use the Troux for TOGAF solution to streamline their efforts. Unlock the full potential of IT portfolio investment: Unifies Strategic IT Planning, EA, and portfolio project management through a common IT governance process. The Troux CA Clarity Connection enables the first bi-directional integration in the market between CA Clarity Project Portfolio Management (PPM) and the Troux EA repository for enhanced IT investment portfolio planning, analysis, and control. Understand your deployed IT assets: Using the out-of-the-box connection to HP's Universal Configuration Management Database (uCMDB), link software and hardware with the applications they support. All of these capabilities are enabled through an open-architected platform that provides uncomplicated data integration tools. The platform provides Architecture-modeling capabilities for IT Architects, an extensible database schema (or meta-model), and integration interfaces that are simple to automate and bring online with minimal programming efforts. Enterprise Architecture repository The Troux Transformation Platform acts as the consolidation point across all the various IT management databases and even some management systems outside the control of IT. By collecting data from across various areas, new insights are possible, leading to reductions in operating costs and improvements in service levels to the business. While it is possible to combine these using other products on the market or even develop a home-grown EA repository, Troux has created a very easy-to-use API for data collection purposes. In addition, Troux provides a database meta-model for the repository that is extensible. Meta-model extensibility makes the product adaptable to the other management systems across the company. Troux also supports a configurable user interface allowing for a customized view into the repository. This capability makes the catalog appear as if it were a part of the other control systems already in place at the company. Additionally, Troux provides an optional set of applications that support a variety of roles, out of the box, with no meta-model extensions or user interface configurations required. These include: Troux Standards: This application supports the IT technology standards and lifecycle governance process usually conducted by the Enterprise Architecture department. Troux Optimization: This application supports the Application portfolio lifecycle management process conducted by the Enterprise Program Management Office (EPMO) and/or Enterprise Architecture. Troux Alignment: This application supports the business and IT assets and application-planning processes conducted by IT Engineering, Corporate Finance, and Enterprise Architecture. Even these three applications that are available out-of-the-box from Troux can be customized by extending their underlying meta-models and customizing the user interface. The EA repository provides output that is viewable online. Standard reports are provided or custom reports can be developed as per the specific needs of the user community. Departments within or even outside of IT can use the customized views, standard reports, and custom reports to perform analyses. For example, the Enterprise Program Management Office (EPMO) can produce reports that link projects with business goals. The EPMO can review the project portfolio of the company to identify projects that do not support company goals. Decisions can be made about these projects, thereby stopping them, slowing them down, or completing them faster. Resources can be moved from the stopped or completed low-value projects to the higher-value projects, leading to increased revenue or reduced costs for the company. In a similar fashion, the Internal Audit department can check on the level of compliance to company IT standards or use the list of applications stored within the catalog to determine the best audit schedule to follow. Less time can be spent auditing applications with minimal impact on company operations or on applications and projects targeted as low value. Application development can use data from the catalog to understand the current capabilities of the existing applications of the company. As staff changes or "off-shore" resources are applied to projects, knowing what existing systems do in advance of a new project can save many hours of work. Information can be extracted from the EA repository directly into requirements documentation, which is always the starting point for new applications, as well as maintenance projects on existing applications. One study performed at a major financial services company showed that over 40% of project development time was spent in the upfront work of documenting and explaining current application capabilities to business sponsors of projects. By supplying development teams with lists of application capabilities early in the project life cycle, time to gather and document requirements can be reduced significantly. Of course, one of the biggest benefactors of the repository is the EA group. In most companies, EA's main charter is to be the steward of information about applications, databases, hardware, software, and network architecture. EA can perform analyses using the data from the repository leading to recommendations for changes by middle and upper management. In addition, EA is responsible for collecting, setting, and managing the IT standards for the company. The repository supports a single source for IT standards, whether they are internal or external standards. The standards portion of the repository can be used as the centerpiece of IT governance. The function of the Architecture Review Board (ARB) is fully supported by Troux Standards. Capacity Planning and IT Engineering functions will also gain substantially through the use of an EA repository. The useful life of IT assets can be analyzed to create a master plan for technical refresh or reengineering efforts. The annual spend on IT expenses can be reduced dramatically through increased levels of virtualization of IT assets, consolidation of platforms, and even consolidation of whole data centers. IT Engineering can review what is currently running across the company and recommend changes to reduce software maintenance costs, eliminate underutilized hardware, and consolidate federated databases. Lastly, IT Operations can benefit from a consolidated view into the technical footprint running at any point in time. Even when system availability service levels call for near-real-time error correction, it may take hours for IT Operations personnel to diagnose problems. They tend not to have a full understanding of what applications run on what servers, which firewalls support which networks, and which databases support which applications. Problem determination time can be reduced by providing accurate technical architecture information to those focused on keeping systems running and meeting business service-level requirements. Summary This article identified the problem IT has with understanding what technologies it has under management. While many solutions are in place in many companies to gain a better view into the IT portfolio, none are designed to show the impact of IT assets in the aggregate. Without the capabilities provided by an EA repository, IT management has a difficult time answering tough questions asked by business leaders. Troux Technologies offers a solution to this problem using the Troux Transformation Platform. The platform acts as a master metadata repository and becomes the focus of many efforts that IT may run to reduce significant costs and improve business service levels. Further resources on this subject: Troux Enterprise Architecture: Managing the EA function [article]
Read more
  • 0
  • 0
  • 9108

article-image-practical-how-recipes-android
Packt
27 Jan 2016
20 min read
Save for later

Practical How-To Recipes for Android

Packt
27 Jan 2016
20 min read
In this article by Rick Boyer and Kyle Merrifield Mew, the author of Android Application Development Cookbook - Second Edition, we'll take a look at the following recipes: Making a Flashlight with a Heads-up notification Scaling down large images to avoid out-of-memory exceptions How to get the last location Push notification using Google Cloud Messaging (For more resources related to this topic, see here.) Making a Flashlight with a Heads-up notification Android 5.0—Lollipop (API 21)—introduced a new type of notification called the Heads-up notification. Many people do not care about this new notification as it can be extremely intrusive. This is because the notification forces its way on top of other apps. (Take a look at the following screenshot.) Keep this in mind when using this type of notification. We're going to demonstrate the Heads-up notification with a Flashlight as this demonstrates a good use case scenario. Here's a screenshot showing the Heads-up notification that we'll create: If you have a device running Android 6.0, you may have noticed the new Flashlight settings option. As a demonstration, we're going to create something similar in this recipe. Getting ready Create a new project in Android Studio and call it FlashlightWithHeadsUp. When prompted for the API level, we need API 23 (or higher) for this project. Select Empty Activity when prompted for Activity Type. How to do it... Our activity layout will consist of just ToggleButton to control the flashlight mode. We'll use the setTorchMode() code and add a Heads-up notification. We'll need permission to use the vibrate option; so, start by opening the Android Manifest and follow these steps: Add the following permission: <uses-permission android_name="android.permission.VIBRATE"/> Specify that we only want a single instance of MainActivity by adding android:launchMode="singleInstance" to the <MainActivity> element. It will look like this: <activity android_name=".MainActivity"     android_launchMode="singleInstance"> With the changes made to the Manifest, open the activity_main.xml layout, and replace the existing <TextView> element with this <ToggleButton> code: <ToggleButton     android_id="@+id/buttonLight"     android_layout_width="wrap_content"     android_layout_height="wrap_content"     android_text="Flashlight"     android_layout_centerVertical="true"     android_layout_centerHorizontal="true"     android_onClick="clickLight"/> Now, open ActivityMain.java and add the following global variables: private static final String ACTION_STOP="STOP"; private CameraManager mCameraManager; private String mCameraId=null; private ToggleButton mButtonLight; Add the following code to onCreate() to set up the camera: mButtonLight = (ToggleButton)findViewById(R.id.buttonLight); mCameraManager = (CameraManager) this.getSystemService(Context.CAMERA_SERVICE); mCameraId = getCameraId(); if (mCameraId==null) {     mButtonLight.setEnabled(false); } else {     mButtonLight.setEnabled(true); } Add the following method to handle the response when the user presses the notification: @Override protected void onNewIntent(Intent intent) {     super.onNewIntent(intent);     if (ACTION_STOP.equals(intent.getAction())) {         setFlashlight(false);     } } Add the method to get the camera ID: private String getCameraId()  {     try {         String[] ids = mCameraManager.getCameraIdList();         for (String id : ids) {             CameraCharacteristics c = mCameraManager.getCameraCharacteristics(id);             Boolean flashAvailable = c.get(CameraCharacteristics.FLASH_INFO_AVAILABLE);             Integer facingDirection = c.get(CameraCharacteristics.LENS_FACING);             if (flashAvailable != null && flashAvailable && facingDirection != null && facingDirection == CameraCharacteristics.LENS_FACING_BACK) {                 return id;             }         }     } catch (CameraAccessException e) {         e.printStackTrace();     }     return null; } Add these two methods to handle the flashlight mode: public void clickLight(View view) {     setFlashlight(mButtonLight.isChecked());     if (mButtonLight.isChecked()) {         showNotification();     } }   private void setFlashlight(boolean enabled) {     mButtonLight.setChecked(enabled);     try {         mCameraManager.setTorchMode(mCameraId, enabled);     } catch (CameraAccessException e) {         e.printStackTrace();     } } Finally, add this method to create the notification: private void showNotification() {     Intent activityIntent = new Intent(this,MainActivity.class);     activityIntent.setAction(ACTION_STOP);     PendingIntent pendingIntent = PendingIntent.getActivity(this,0,activityIntent,0);     final Builder notificationBuilder = new Builder(this)             .setContentTitle("Flashlight")             .setContentText("Press to turn off the flashlight")             .setSmallIcon(R.mipmap.ic_launcher)             .setLargeIcon(BitmapFactory.decodeResource(getResources(), R.mipmap.ic_launcher))             .setContentIntent(pendingIntent)             .setVibrate(new long[]{DEFAULT_VIBRATE})             .setPriority(PRIORITY_MAX);     NotificationManager notificationManager = (NotificationManager) this.getSystemService(Context.NOTIFICATION_SERVICE);     notificationManager.notify(0, notificationBuilder.build()); } You're ready to run the application on a physical device. As seen in the preceding steps, you'll need an Android 6.0 (or higher) device, with an outward facing camera flash. How it works... Since this recipe uses the same flashlight code, we'll jump into the showNotification() method. Most of the notification builder calls are the same as the ones seen in previous examples, but there are two significant differences: .setVibrate() .setPriority(PRIORITY_MAX) Notifications will not be escalated to Heads-up notifications unless the priority is high (or above) and uses either vibrate or sound. Take a look at this from the developer documentation (http://developer.android.com/reference/android/app/Notification.html#headsUpContentView): "At its discretion, the system UI may choose to show this as a heads-up notification". We create the PendingIntent method as we've done previously, but, here, we set the action using this code: activityIntent.setAction(ACTION_STOP); We set the app to only allow a single instance in the AndroidManifest method as we don't want to start a new instance of the app when the user presses the notification. The PendingIntent method we created sets the action, which we can check out in the onNewIntent() callback. If the user opens the app without pressing the notification, they can still disable the flashlight using the ToggleButton. There's more... We can use a custom layout with notifications. Use the following method on the builder to specify its layout: headsupContentView() Scaling down large images to avoid out-of-memory exceptions Working with images can be very memory-intensive, often resulting in your application crashing due to an out-of-memory exception. This is especially true for pictures taken with the device camera as they often have a much higher resolution than the device itself. Since loading a higher-resolution image than the UI supports doesn't provide any visual benefit, this recipe will demonstrate how to take smaller samples of the image for display. We'll use BitmapFactory to first check the image size, and we'll then load a scaled down image. Here's a screenshot from this recipe, showing a thumbnail of a very large image: Getting ready Create a new project in Android Studio and call it LoadLargeImage. Use the default Phone & Tablet options, and select Empty Activity when prompted for the Activity Type. We'll need a large image for this recipe, so we've referred to https://pixabay.com/ for an image. Since the image itself doesn't matter, we downloaded the first image that shown at the time. (The full size of image is 6000 x 4000 and 3.4 MB.) How to do it... As stated previously, we need a large image to demonstrate the scaling. Once you have the image, follow these steps: Copy the image to res/drawable as image_large.jpg (use the appropriate extension if you choose a different file type) Open activity_main.xml and replace the existing TextView with the following ImageView: <ImageView     android_id="@+id/imageViewThumbnail"     android_layout_width="100dp"     android_layout_height="100dp"     android_layout_centerInParent="true" /> Now, open MainActivity.java and add this method, which we'll explain as follows: public Bitmap loadSampledResource(int imageID, int targetHeight, int targetWidth) {     final BitmapFactory.Options options = new BitmapFactory.Options();     options.inJustDecodeBounds = true;     BitmapFactory.decodeResource(getResources(), imageID, options);     final int originalHeight = options.outHeight;     final int originalWidth = options.outWidth;     int inSampleSize = 1;     while ((originalHeight / (inSampleSize *2)) > targetHeight && (originalWidth / (inSampleSize *2)) > targetWidth) {         inSampleSize *= 2;     }     options.inSampleSize=inSampleSize;     options.inJustDecodeBounds = false;     return BitmapFactory.decodeResource(getResources(), imageID, options); } Add the following code to the existing onCreate() method: ImageView imageView = (ImageView)findViewById(R.id.imageViewThumbnail); imageView.setImageBitmap(loadSampledResource(R.drawable.image_large, 100, 100)); Run the application on a device or emulator. How it works... The purpose of the loadSampledResource() method is to load a smaller image to reduce the memory consumption of the image. If we attempted to load the full image chosen from https://pixabay.com/, the app would require over 3 MB of RAM to load. That's more memory than most devices can handle (at the moment, anyway), and even if it could be loaded completely, it would provide no visual benefit to our thumbnail view. To avoid an out-of-memory situation, we use the inSampleSize property of BitmapFactory. You will find options to reduce or subsample the image. (If we set inSampleSize=2, it will reduce the image in half. If we use inSampleSize=4, it will reduce the image by ¼.) To calculate inSampleSize, we first need to know the image size. We can use the inJustDecodeBounds property, as follows: options.inJustDecodeBounds = true; This tells BitmapFactory to get the image dimensions without actually storing the contents of the image. Once we know the image size, we calculate the sample using this code: while ((originalHeight / (inSampleSize *2)) > targetHeight &&             (originalWidth / (inSampleSize *2)) > targetWidth) {         inSampleSize *= 2;     } The purpose of this code is to determine the largest sample size that does not reduce the image below the target dimensions. To do this, we double the sample size, and check whether the size exceeds the target size dimensions. If it doesn't, we save the doubled sample size and repeat the process. Once the reduced size falls below the target dimensions, we use the last saved inSampleSize. From the inSampleSize documentation: Note that the decoder uses a final value that's based on powers of 2; any other value will be rounded down to the nearest power of 2. Once we have the sample size, we set the inSampleSize property. We also set inJustDecodeBounds to false in order to make it load normally. Here is the code to do this: options.inSampleSize = inSampleSize; options.inJustDecodeBounds = false; It's important to note that this recipe illustrates the concept of applying a task in your own application. Loading and processing images can be a long operation, which could cause your application to stop responding. This is not a good thing and could cause Android to show the Application Not Responding (ANR) dialog. It is recommended that you perform long tasks on a background thread to keep your UI thread responsive. There's more... It's important to note that the targetHeight and targetWidth parameters we pass to the loadSampledResource()method do not actually set the size of the image. If you run the application using the same sized image we used earlier, the sample size will be 32, resulting in a loaded image that is 187 x 125 in size. If your layout needs an image of a specific size, either set the size in the layout file; otherwise, you can modify the size directly using the Bitmap class. See also The inSampleSize() documentation at https://developer.android.com/reference/android/graphics/BitmapFactory.Options.html#inSampleSize How to get the last Location We'll start this with a simple recipe that is commonly needed: how to get the last known location. This is an easy-to-use API with very little overhead resource drain (which means that your app won't be responsible for killing the battery life). This recipe also provides a good introduction to setting up the Google Location APIs. Getting ready Create a new project in Android Studio and call it GetLastLocation. Use the default Phone & Tablet options, and select Empty Activity when prompted for the Activity Type. How to do it... First, we'll add the necessary permissions to the Android Manifest. We'll then create a layout with a Button and TextView. Finally, we'll create GoogleAPIClient to access the previous location. Open the Android Manifest and follow these steps: Add the following permission: <uses-permission android_name="android.permission.ACCESS_COARSE_LOCATION"/> Open the build.gradle file (Module: app), as shown in this screenshot: Add the following statement to the dependencies section: compile 'com.google.android.gms:play-services:8.4.0' Open activity_main.xml and replace the existing TextView with the following XML: <TextView     android_id="@+id/textView"     android_layout_width="wrap_content"     android_layout_height="wrap_content" /> <Button     android_id="@+id/button"     android_layout_width="wrap_content"     android_layout_height="wrap_content"     android_text="Get Location" android_layout_centerInParent="true"     android_onClick="getLocation"/> Open MainActivity.java and add the following global variables: GoogleApiClient mGoogleApiClient; TextView mTextView; Button mButton; Add the class for ConnectionCallbacks: GoogleApiClient.ConnectionCallbacks mConnectionCallbacks = new GoogleApiClient.ConnectionCallbacks() {     @Override     public void onConnected(Bundle bundle) {         mButton.setEnabled(true);     }     @Override     public void onConnectionSuspended(int i) {} }; Add the class to handle the OnConnectionFailedListener callback: GoogleApiClient.OnConnectionFailedListener mOnConnectionFailedListener = new GoogleApiClient.OnConnectionFailedListener() {     @Override     public void onConnectionFailed(ConnectionResult connectionResult) {         Toast.makeText(MainActivity.this, connectionResult.toString(), Toast.LENGTH_LONG).show();     } }; Add the following code to the existing onCreate() method: mTextView = (TextView) findViewById(R.id.textView); mButton = (Button) findViewById(R.id.button); mButton.setEnabled(false); setupGoogleApiClient(); Add the method to set up GoogleAPIClient: protected synchronized void setupGoogleApiClient() {     mGoogleApiClient = new GoogleApiClient.Builder(this)             .addConnectionCallbacks(mConnectionCallbacks)             .addOnConnectionFailedListener(mOnConnectionFailedListener)             .addApi(LocationServices.API)             .build();     mGoogleApiClient.connect(); } Add the method for the button click: public void getLocation(View view) {     try {         Location lastLocation = LocationServices.FusedLocationApi.getLastLocation(                 mGoogleApiClient);         if (lastLocation != null) {             mTextView.setText(                     DateFormat.getTimeInstance().format(lastLocation.getTime()) + "n" +                             "Latitude="+lastLocation.getLatitude()+"n"+                             "Longitude="+lastLocation.getLongitude());         } else {             Toast.makeText(MainActivity.this, "null", Toast.LENGTH_LONG).show();         }     }     catch (SecurityException e) {} } You're ready to run the application on a device or emulator. How it works... Before we can call the getLastLocation() method, we need to set up GoogleApiClient. We call GoogleApiClient.Builder in our setupGoogleApiClient() method, and then connect to the library. When the library is ready, it calls our ConnectionCallbacks.onConnected() method. For demonstration purposes, this is where we enable the button. We used a button to show that we can call getLastLocation() on demand; it's not a one-time call. The system is responsible for updating the location and may return the same previous location on repeated calls. (This can be seen in the timestamp—it's the location timestamp, not the timestamp that appears when the button is pressed.) This approach of calling the location non-demand can be useful in situations where you only need the location when something happens in your app (such as geocoding an object). Since the system is responsible for the updates of the location, your app will not be responsible for draining your battery due to location updates. The accuracy of the Location object we receive is based on our permission setting. We used ACCESS_COARSE_LOCATION, but if we want higher accuracy, we can request ACCESS_FINE_LOCATION instead using the following permission: <uses-permission android_name="android.permission.ACCESS_FINE_LOCATION"/> Lastly, to keep the code focused on GoogleApiClient, we just wrap getLastLocation() with SecurityException. In a production application, you should check and request the permission. There's more... If a problem occurs when establishing a connection with GoogleApiClient, OnConnectionFailedListener is called. In this example, we will display a toast. Testing the location can be a challenge since it's difficult to actually move the device when testing and debugging it. Fortunately, we have the ability to simulate GPS data with the emulator. (It is possible to create mock locations on a physical device as well, but it's not as easy.) Mock Locations There are three ways to simulate locations using the emulator: Android Studio DDMS The Geo command via Telnet To set a mock location in Android Studio, follow these steps: Go to the Tools | Android | Android Device Monitor menu. Select the Emulator Control tab in the Devices window. Enter GPS coordinates under Location Controls. Here's a screenshot showing Location Controls: Important: Simulating the location works by sending GPS data. Therefore, for your app to receive the mock location, it will need to receive GPS data. Testing lastLocation() may not send the mock GPS data since it doesn't rely solely on GPS to determine the location of the device. (We can't force the system to use any specific location sensor; we can only make a request. The system will choose the optimum solution to deliver results.) See also How to set up Google Play Services at https://developers.google.com/android/guides/setup FusedLocationProviderApi at https://developers.google.com/android/reference/com/google/android/gms/location/FusedLocationProviderApi Push notification using Google Cloud Messaging Google Cloud Messaging (GCM), Google's version of a push notification, allows your application to receive messages. The idea is similar to SMS messages but much more flexible. There are three components of GCM: Your server (this is where you initiate the message) Google's GCM server An Android device (though GCM is also available on other platforms) When the user starts the application, your code needs to connect to the GCM server and obtain a device token, and then send this token to your server. Your server is responsible for initiating the message and passing it to the GCM server. Your server needs to track the device tokens to be sent when initiating the message (your server tells the GCM server which device tokens to send). You can implement your own server or chose to use one of many services available (the Simple Testing Option section offers an option to verify whether your code works). This recipe will walk you through the steps needed to add GCM using the current (version 8.3) Google Services library. Before getting to the steps, it's worth noting that GCM is supported all the way back to API 8 as long as the user has a Google account. A Google account is not required after installing Android 4.0.4. Getting ready Create a new project in Android Studio and call it GCM. Use the default Phone & Tablet options, and select Empty Activity when prompted for the Activity Type. Google Cloud Messaging uses the Google Services Plugin, which requires a Google Services Configuration File, available from the Google Developer Console. To create the configuration file, you will need the following information: The name of your application package When you have the information, log into https://developers.google.com/mobile/add, and follow the wizard to enable Google Cloud Messaging for your app Note that if you download the source files, you will need to create a new package name when following the steps as the existing package name has already been registered. How to do it... After completing the preceding section, follow these steps: Copy the google-services.json file you downloaded in the Getting Ready section to your app folder (<project folder>GCMapp). Open the project Gradle build file called build.gradle (project: GCM). Add the following to the build script dependencies section: classpath 'com.google.gms:google-services:1.5.0-beta2' Open the Gradle app module build file, called build.gradle (module: app), and add the following statement to the beginning of the file (above the android section): apply plugin: 'com.google.gms.google-services' In the same module build file, as seen in step 3, add the following statement to the dependencies section: compile 'com.google.android.gms:play-services-auth:8.3.0' Open the Android Manifest and add the following permissions: <uses-permission android_name="android.permission.WAKE_LOCK" /> <permission android_name="<packageName >.permission.C2D_MESSAGE"   android_protectionLevel="signature" /> <uses-permission android_name="<packageName >.permission.C2D_MESSAGE" /> Within the <application> element, add the following <receiver> and <service> declarations (these should be at the same level as <activity>): <receiver     android_name="com.google.android.gms.gcm.GcmReceiver"     android_exported="true"     android_permission="com.google.android.c2dm.permission.SEND" > <intent-filter> <action android_name="com.google.android.c2dm.intent.RECEIVE" /> <category android_name="<packageName>" /> <action android_name="com.google.android.c2dm.intent.REGISTRATION" /> </intent-filter> </receiver> <service     android_name=".GCMService"     android_exported="false" > <intent-filter> <action android_name="com.google.android.c2dm.intent.GCM_RECEIVED_ACTION"/> <action android_name="com.google.android.c2dm.intent.RECEIVE" /> </intent-filter> </service> <service     android_name=".GCMInstanceService"     android_exported="false"> <intent-filter> <action android_name="com.google.android.gms.iid.InstanceID" /> </intent-filter> </service> <service     android_name=".GCMRegistrationService"     android_exported="false"> </service> Create a new Java class, called GCMRegistrationService, that extends IntentService, as follows: public class GCMRegistrationService extends IntentService {     private final String SENT_TOKEN="SENT_TOKEN";     public GCMRegistrationService() {         super("GCMRegistrationService");     }     @Override     protected void onHandleIntent(Intent intent) {         super.onCreate();         SharedPreferences sharedPreferences = PreferenceManager.getDefaultSharedPreferences(this);         try {             InstanceID instanceID = InstanceID.getInstance(this);             String token = instanceID.getToken(getString(R.string.gcm_defaultSenderId),                     GoogleCloudMessaging.INSTANCE_ID_SCOPE, null);             Log.i("GCMRegistrationService", "GCM Registration Token: " + token);             //sendTokenToServer(token);             sharedPreferences.edit().putBoolean(SENT_TOKEN, true).apply();         } catch (Exception e) {             sharedPreferences.edit().putBoolean(SENT_TOKEN, false).apply();         }     } } Create a new Java class, called GCMInstanceServicethat, that extends InstanceIDListenerServiceas, as follows: public class GCMInstanceService extends InstanceIDListenerService {     @Override          public void onTokenRefresh() {         Intent intent = new Intent(this, GCMRegistrationService.class);         startService(intent);     } } Create a new Java class, called GCMServicethat, that extends GcmListenerServiceas, as follows: public class GCMService extends GcmListenerService {     @Override     public void onMessageReceived(String from, Bundle data) {         super.onMessageReceived(from, data);         Log.i("GCMService", "onMessageReceived(): " + data.toString());     } } Add the following code to the existing onCreate() callback: Intent intent = new Intent(this, GCMRegistrationService.class); startService(intent); You're ready to run the application on a device or emulator. How it works... Most of the actual GCM code is encapsulated within the Google APIs, simplifying their implementation. We just have to set up the project to include the Google Services and give our app the required permissions. Important: When adding the permissions in steps 5 and 6, replace the <packageName> placeholder with your application's package name. The most complicated aspect of GCM is probably the multiple services that are required. Even though the code in each service is minimal, each service has a specific task. There are two main aspects of GCM: Registering the app with the GCM server Receiving messages This is the code to register with the GCM server: String token = instanceID.getToken(getString(R.string.gcm_defaultSenderId),         GoogleCloudMessaging.INSTANCE_ID_SCOPE, null); We don't call getToken() in the Activity because it could block the UI thread. Instead, we call GCMRegistrationService, which handles the call in a background thread. After you receive the device token, you need to send it to your server as it is needed when initiating a message. Receiving a GCM message is handled in GCMService, which extends GcmListenerService. Since the Google API already handles most of the work, all we have to do respond to the onMessageReceived() callback. There's more... To make it easier to type, we left out an important Google Services API verification, which should be included in any production application. Instead of calling GCMRegistrationService directly, as we did in onCreate() previously, first check whether the Google API Service is available. Here's an example that shows how to call the isGooglePlayServicesAvailable() method: private boolean isGooglePlayServicesAvailable() {     GoogleApiAvailability googleApiAvailability = GoogleApiAvailability.getInstance();     int resultCode = googleApiAvailability.isGooglePlayServicesAvailable(this);     if (resultCode != ConnectionResult.SUCCESS) {         if (googleApiAvailability.isUserResolvableError(resultCode)) {             googleApiAvailability.getErrorDialog(this, resultCode, PLAY_SERVICES_RESOLUTION_REQUEST)                     .show();         } else {             Toast.makeText(MainActivity.this, "Unsupported Device", Toast.LENGTH_SHORT).show();             finish();         }         return false;     }     return true; } Then, change the onCreate() code to call this method first: if (isGooglePlayServicesAvailable()) {     Intent intent = new Intent(this, GCMRegistrationService.class);     startService(intent); } Simple testing option To verify whether your code is working correctly, a testing application was created and posted on Google Play. This app will run on both a physical device and an emulator. The Google Play listing also includes a link to download the source code to run the project directly, making it easier to enter the required fields. Take a look at GCM (Push Notification) Tester at https://play.google.com/store/apps/details?id=com.eboyer.gcmtester. See also Google Cloud Messaging at https://developers.google.com/android/reference/com/google/android/gms/gcm/GoogleCloudMessaging GCM Connection Server at https://developers.google.com/cloud-messaging/server Summary In this article, we learned how to make a Flashlight with a Heads-up notification, scaling down large images to avoid out-of-memory exceptions, how to get last location and using push notification with GCM. Resources for Article: Further resources on this subject: Introduction to GameMaker: Studio [article] Working with Xamarin.Android [article] The Art of Android Development Using Android Studio [article]
Read more
  • 0
  • 0
  • 9104

article-image-working-data-access-and-file-formats-using-nodejs
Packt
04 Sep 2014
27 min read
Save for later

Working with Data Access and File Formats Using Node.js

Packt
04 Sep 2014
27 min read
In this article by Surendra Mohan, the author of Node.js Essentials, we will cover the following concepts: Reading and writing files using Node.js MySQL database handling using Node.js Working with data formats using Node.js Let's get started! (For more resources related to this topic, see here.) Reading and writing files The easiest and most convenient way of reading a file in a PHP application is by using the PHP file_get_contents() API function. Let's look into the following example PHP code snippet, wherein we intend to read a sample text file named sampleaf.txt that resides in the same directory as that of our PHP file (sampleaf.php): <?php $text = file_get_contents('sampleaf.txt'); print $text; ?> In the preceding code snippet, if the source file, sampleaf.txt, exists or can be read, the content of this file is assigned to the $text variable (long string type); otherwise, it will result in a Boolean value as false. All PHP API functions are blocking in nature, so is the file_get_contents() API function. Thus, the PHP code that is supposed to be executed after the file_get_contents() API function call gets blocked until the former code either executes successfully or completely fails. There is no callback mechanism available for this PHP API. Let's convert the preceding PHP code snippet into its corresponding Node.js code. Because the readFileSync() API function in the fs module is the closest Node.js equivalent to that of the PHP file_get_contents() API function, let's use it. Our Node.js code equivalent looks something like the following code snippet (sampleaf.njs): var fs = require('fs'); var text = false; try { text = fs.readFileSync(__dirname+'/'+'sampleaf.txt', 'utf8'); } catch (err) { // No action } console.log(text); Node.js functions come in both asynchronous as well as synchronous forms, asynchronous being the default. In our preceding Node.js code, we have appended the Sync term in our Node.js fs.readFile() API function, which is asynchronous in nature, and gets converted to its synchronous version once Sync gets appended to the end of it. The asynchronous version is nonblocking in nature, and depends upon the callbacks to take care of the Node.js API function call results. On the other hand, the synchronous version is blocking in nature (same as of our PHP code), which results in blocking of the Node.js code that is supposed to be executed after the API function till it completely succeeds or fails. If we look into the arguments passed with the Node.js fs.readFileSync() API function, we find the source file sampleaf.txt (prepended with the _dirname variable) that needs to be read, and utf8 stating the encoding we intend to use. The _dirname variable holds the directory name where our Node.js code file, sampleaf.njs, resides. The use of the _dirname variable instructs the Node.js fs.readFileSync() API function to locate the source file in the directory returned by this variable. If this variable is missing, our API function will try to find the source file in the directory where the Node.js server was started. By default, the second argument doesn't encode, which results in the function to return a raw buffer of bytes instead of a string. In order to enable the function to return a string, we pass the utf8 string (for UTF-8 encoding) as the second parameter to the function. Because we are dealing with the synchronous version of the API function, we handled Node.js exceptions by using the Node.js try and catch keywords. In this case, if the try block code gets executed successfully, the catch block code will be ignored and never get executed; otherwise, the try block code will immediately stop executing, thereby helping the catch block code to get executed. While replacing the PHP file_get_contents() API function with its corresponding Node.js API function, it is recommended to use the Node.js fs.readFile() API function instead of fs.readFileSync() due to the fact that synchronous API functions (in our case, fs.readFileSync()) are blocking in nature, whereas asynchronous API functions (in our case, fs.readFile()) are not. So, let's try converting the preceding PHP code snippet to its corresponding asynchronous Node.js code by using the fs.readFile() API function. We write the following Node.js code snippet and save it in a new file with the filename sampleafa.njs: var fs = require('fs'); var text = false; fs.readFile(__dirname+'/'+'sampleaf.txt', 'utf8', function(err, fo) { if (!err) { text = fo; } console.log(text); }); Our preceding asynchronous Node.js fs.readFile() API function accepts a callback function as its third argument that can return both the data as well as error, whichever is applicable. There is another way to read a file in Node.js. However, it doesn't match with any features available in PHP so far. We do so by creating a Node.js stream that will help us read the file. While the stream is read, events such as data, error, and close, are sent along with the stream. In such scenarios, we need to set up event handlers that would take care of such events. The PHP file() API function The file() API function helps us read content of a file and returns it as an indexed array. The content in the array are stored in such a way that each value of the array holds a single line of the file. If we want to print the first line of our source file (sampleaf.txt) in PHP, we write the following code snippet that includes the End Of Line (EOL) character sequence at the end of the line: $x = file('sampleaf.txt'); print $x[0]; If we are using PHP5 version, we get an opportunity to include the second and optional parameter (the flags parameter) to our file() API function. The flags parameter provides us with three options that can be either used individually or can be combined together using the OR operator (|), and they are as follows: FILE_IGNORE_NEW_LINES FILE_SKIP_EMPTY_LINES FILE_USE_INCLUDE_PATH The FILE_IGNORE_NEW_LINES flag option is normally used, and it instructs the PHP file() API function to eradicate EOL characters from the end of each line. Let's rework on our preceding PHP code snippet such that it prints the first line of the sampleaf.txt file, but eradicates the EOL character sequence at the end of each value in the array. So, our modified PHP code snippet will look like the following: $x = file('sampleaf.txt', FILE_IGNORE_NEW_LINES); print $x[0]; Now it's time to convert the preceding PHP code snippet into its corresponding Node.js code snippet. Converting the PHP file() API function is a bit complicated as compared to that of converting the PHP file_get_contents() API function. The following code demonstrates the converted Node.js code snippet corresponding to our PHP code snippet: var fs = require('fs'); var FILE_IGNORE_NEW_LINES = 0x2; var x = false; var flag = FILE_IGNORE_NEW_LINES; fs.readFile(__dirname+'/'+'sampleaf.txt', 'utf8', function(err, data) { if (!err) { x = data.replace(/rn?/g,'n'); x = x.split('n'); x.neol = x.length - 1; if ((x.length > 0) && (x[x.length-1] === '')) { x.splice(x.length-1, 1); } if ((flag & FILE_IGNORE_NEW_LINES) === 0) { for (var i=0; i < x.neol; ++i) { x[i] += 'n'; } } delete x.neol; } console.log(x[0]); }); In the preceding Node.js code, the !err condition within the if statement is the real culprit that makes this code actually complicated. Let's now walk through the preceding Node.js code snippet, especially the ones we have embedded in the if statement: First of all, we converted EOL characters for the Linux, Windows, and Mac text files into the end-of-line character (n) for the operating system our Node.js server is current running, using the following code chunk: x = data.replace(/rn?/g,'n'); Then, we converted the string to an array of lines that complies with the PHP file() API function standards, using the following line of code: x = x.split('n'); Then, we handled the last line of the file by implementing the following code snippet: x.neol = x.length - 1; if ((x.length > 0) && (x[x.length-1] === '')) { x.splice(x.length-1, 1); } Finally, in the following code snippet, we check whether FILE_IGNORE_NEW_LINES has been specified or not. If it hasn't been specified, the EOL character will be added to the end of the lines: if ((flag & FILE_IGNORE_NEW_LINES) === 0) { for (var i=0; i < x.neol; ++i) { x[i] += 'n'; } } File handling APIs The core set of file handling APIs in PHP and Node.js are shaped based on the C language file handling API functions. For instance, the PHP fopen() API function opens a file in different modes, such as read, write, and append. The Node.js open() API function is the equivalent to this PHP fopen() API function, and both of these API functions are shaped with the fopen() from the C language. Let's consider the following PHP code snippet that opens a file for reading purposes and reads the first 500 bytes of content from the file: $fo = fopen('sampleaf.txt', 'r'); $text = fread($fo, 500); fclose($fo); In case the file size is less than 500 bytes, our preceding PHP code snippet will read the entire file. In Node.js, the Node.js fs.read() API function is used to read from the file and uses the buffer built-in module to hold an ordered collection of bytes. Likewise, the Node.js fs.close() API function is used to close the file once it is read as intended. In order to convert the preceding PHP code snippet, we write the following Node.js code snippet: var fs = require('fs'); var Buffer = require('buffer').Buffer; fs.open(__dirname+'/'+'sampleaf.txt', 'r', function(err, fo) { var text = ''; var b = new Buffer(500); fs.read(fo, b, 0, b.length, null, function(err, bytesRead, buf){ var bufs = buf.slice(0, bytesRead); text += bufs.toString(); fs.close(fo, function() { console.log(text); }); }); }); In our Node.js code, besides the usual callback functions, we have used a couple of buffer variables, such as the b and bufs variables that adds some complexity to our code. The b variable holds the data that is read using the Node.js fs.read() API function. The bufs variable holds the actual bytes that are read, wherein the unused bytes of the b variable are sliced off. The buf argument is an alias of the b variable. Both PHP and the Node.js maintain a file pointer that indicates the next bytes that should be read from the file. We can cross-check the end of the file using the PHP feof() API function. In order to implement the PHP feof() API function, we write the following PHP code snippet: $fo = fopen('sampleaf.txt', 'r'); $text = ''; while (!feof($fo)) { $text .= fread($fo, 500); } fclose($fo); print $text; Node.js doesn't have anything that is equivalent to the PHP feof() API function. Instead, we use the bytesRead argument that is passed to the callback function and is compared with the number of bytes requested in order to read the file. We land to the following Node.js code snippet when we convert our preceding and modified PHP code snippet: var fs = require('fs'); var Buffer = require('buffer').Buffer; fs.open(__dirname+'/'+'sampleaf.txt', 'r', function(err, fo) { var text = ''; var b = new Buffer(500); var fread = function() { fs.read(fo, b, 0, b.length, null, function(err, bytesRead, buf) { var eof = (bytesRead != b.length); if (!eof) { text += buf.toString(); fread(); } else { if (bytesRead > 0) { var bufs = buf.slice(0, bytesRead); text += bufs.toString(); } fs.close(fo, function() { console.log(text); }); } }); }; fread(); }) Due to callbacks in Node.js, the fread function variable must be defined in such a way that it can be called if the file size is greater than the b buffer variable. The fread function variable is triggered continuously till the end of the file. By the end of the file, the partially occupied buffer is proceeded, and then the Node.js fs.close() API function is triggered. We also use the linearity concept, where the Node.js console.log() API function call is embedded in the callback of the Node.js fs.close() API function. MySQL access In the previous section, we learned how to access the data from files using PHP code and exercised how to convert such PHP code to its corresponding Node.js code. As an alternative to what we discussed earlier, we can even access the necessary data from our database, and write to it as a record or set of records. Because the database server can be accessed remotely, PHP and Node.js are capable enough to connect to the intended database, regardless of whether it is running on the same server or remote server. You must be aware that data in a database is arranged in rows and columns. This makes it easy to organize and store data such as usernames. On the other hand, it is quite complex to organize and store certain types of data, such as image files or any other media files. In this section, we will use the MySQL database with PHP, and learn how to convert our PHP code that uses the MySQL database into its equivalent Node.js code based on different scenarios. The reason behind choosing the MySQL database for our exercise is that it is quite popular in the database and hosting market. Moreover, PHP applications have a special bond with MySQL database. We assume that the MySQL server has already been installed, so that we can create and use the MySQL database with the PHP and Node.js code during our exercise. In order to access our database through a PHP or Node.js application, our web application server (where PHP or Node.js is running) needs some tweaking, so that necessary accesses are granted to the database. If you are running the Apache2 web server on a Linux environment, the phpx-myql extension needs to be installed, where x denotes the PHP version you are using. For instance, when using PHP 5.x, the required and related extension that needs to be installed would be php5-mysql. Likewise, the php4-mysql and php6-mysql extensions are necessary for PHP versions 4.x and 6.x, respectively. On the other hand, if you are using the Apache2 web server on a Windows environment, you need to install the PHP-to-MySQL extension during the Apache2 web server installation. Database approaches Node.js doesn't have a built-in module that can help a Node.js application access the MySQL database. However, we have a number of modules that are provided by the Node.js npm package and can be installed in order to achieve database access in variety of approaches. Using the MySQL socket protocol MySQL socket protocol is one of the easiest approaches that can be implemented in Node.js using the Node.js npm package. This npm package uses the built-in net module to open a network socket for the MySQL server to connect with the application and exchange packets in a format that is supported and expected by the MySQL server. The Node.js npm package bluffs and surpasses other MySQL drivers (shipped with the MySQL server installer), unaware of the fact that it is communicating to the Node.js driver instead of the default driver that has been built in C language. There are a number of ways MySQL socket protocol can be implemented in Node.js. The most popular implementation is using the Node.js node-mysql npm package. In order to install this npm package, you can either retrieve it from its GitHub repository at http://github.com/felixge/node-mysql or run npm install mysql on the command line. An alternative to the Node.js implementation of this protocol is the Node.js mysql-native npm package. In order to install this package, you can either retrieve it from its GitHub repository at http://github.com/sidorares/nodejs-mysql-native, or run npm install mysql-native on the command line. In order to play around with database records, the SQL language that needs to be applied constitutes of commands such as SELECT, INSERT, UPDATE, and DELETE along with other commands. However, Node.js stores data in the database as properties on a Node.js object. Object-relational mapping (ORM or O/R mapping) is a set of planned actions to read and write objects (in our case, Node.js objects) to a SQL-based database (in our case, the MySQL database). This ORM is implemented on the top of other database approaches. Thus, the Node.js ORM npm package can use any other Node.js npm packages (for instance, node-mysql and mysql-native) to access and play around with the database. Normally, ORM npm packages use SQL statements during implementation; however, it provides a better and logical set of API functions to do the database access and data exchange job. The following are a couple of Node.js npm modules that provide object-relational mapping support to Node.js: The Node.js persistencejs npm module: This is an asynchronous JavaScript-based ORM library. We recommend you to refer its GitHub repository documentation at https://github.com/coresmart/persistencejs, in case you wish to learn about it. The Node.js sequelize npm module : This is a JavaScript-based ORM library that provides access to databases such as MySQL, SQLite, PostgreSQL, and so on, by mapping database records to objects and vice versa. If you want to learn more about the sequelize library, we recommend that you refer to its documentation at http://sequelizejs.com/. Normally, an object-relational mapping layer makes the Node.js world quite simple, convenient, and developer friendly. Using the node-mysql Node.js npm package In this section, we will learn how to implement the Node.js node-mysql npm package, which is the most popular way of accessing a MySQL database using Node.js. In order to use the node-mysql package, we need to install it. This Node.js npm module can be installed in the same way as we install other Node.js npm packages. So, to install it, we run the following command: npm install mysql As soon as the node-mysql package gets installed, we need to make this package available for use by using the Node.js require() API function. To do so, we create a mysql variable to access the Node.js node-mysql module, as demonstrated in the following line of Node.js code: var mysql = require('mysql'); Before you can use database records to read or write, it is mandatory to connect your PHP or Node.js application to this database. In order to connect our PHP application to our MySQL database, we use three sets of the PHP API function (mysql, mysqli, and PDO) that use the PDO_MySQL driver. Let's write the following PHP code snippet: $sql_host = '192.168.0.100'; $sql_user = 'adminuser'; $sql_pass = 'password'; $conn = mysql_connect($sql_host, $sql_user, $sql_pass); In the preceding code snippet, the $conn variable holds the database collection. In order to establish the database connection, we used the PHP mysql_connect() API function that accepts three arguments: database server as the IP address or DNS (in our case, the IP address is 192.168.0.100), database username (in our case, adminuser), and the password associated to the database user (in our case, password). When working with the node-mysql Node.js npm package, the Node.js createClient() API function is used as the equivalent to the PHP mysql_connect() API function. Unlike the PHP API function, the Node.js API function accesses a Node.js object with the three properties as its parameters. Moreover, we want our Node.js code to load the mysql Node.js npm package. Thus, we use Node.js require() to achieve this. Let's write the following Node.js code snippet that is equivalent to our preceding PHP code snippet: Var mysql = require('mysql'); var sql_host = '192.168.0.100'; var sql_user = 'adminuser'; var sql_pass = 'password'; var sql_conn = {host: sql_host, user: sql_user, password: sql_pass}; var conn = mysql.createClient(sql_conn); We can even merge the last two statements (highlighted) in a single statement. Thus, we replace the highlighted statements with the following one: var conn = mysql.createClient({host: sql_host, user: sql_user, password: sql_pass}); In the case of both the PHP $conn and Node.js conn variables, a meaningful value is assigned to these variables if the MySQL server is accessible; otherwise, they are assigned a false value. Once the database is no longer needed, it needs to be disconnected from our PHP and Node.js code. Using PHP, the MySQL connection variable (in our case, $conn) needs to be closed using the PHP mysql_close() API function by implementing the following PHP code statement: $disconn = mysql_close($conn); The PHP mysql_close() API function returns a Boolean value that indicates whether the connection has been closed successfully or failed. In the case of Node.js, we use the destroy() method on the conn object in order to close the database connection using the following Node.js code statement: conn.destroy(); Once our applications get connected to the desired MySQL database on the MySQL server, the database needs to be selected. In case of PHP, the PHP mysql_select_db() API function is used to do this job. The following PHP code snippet demonstrates how we select the desired database: $sql_db = 'desiredDB'; $selectedDB = mysql_select_db($sql_db, $conn); While converting the PHP code into its equivalent Node.js code, it should be refactored to explicitly pass the PHP $conn variable to all the mysql API functions. As soon as the database is selected, we use the PHP mysql_query() API function to play around with the data of the selected database. In the case of Node.js, we use Node.js query() methods, which is equivalent to its corresponding PHP code statement. In order to select a database, the SQL USE command is used as demonstrated in the following code line: USE desiredDB; In the preceding statement, if we are using a single database, the semicolon (;) is optional. It is used as a separator in case you plan to use more than one database. When working with Node.js, the USE desiredDB SQL command needs to be sent using the Node.js query() method. Let's write the following Node.js code snippet that selects the desiredDB database from our MySQL server via the Node.js conn variable: var sql_db = 'desiredDB'; var sql_db_select = 'USE '+sql_db; conn.query(sql_db_select, function(err) { if (!err) { // Selects the desired MySQL database, that is, desiredDB } else { // MySQL database selection error } }); We can even merge the highlighted statements in the preceding code snippet into one statement. Our Node.js code snippet will look something like the following once these statements get merged: var sql_db = 'desiredDB'; conn.query('USE '+sql_db, function(err) { if (!err) { // Selects the desired MySQL database, that is, desiredDB } else { // MySQL database selection error } }); By now, we are able to connect our PHP and Node.js applications to the MySQL server and select the desired MySQL database to play around with. Our data in the selected database can be accessed (in terms of reading and writing) using popular SQL commands, such as CREATE TABLE, DROP TABLE, SELECT, INSERT, UPDATE, and DELETE. Creating a table Let's consider the CREATE TABLE SQL command. In PHP, the CREATE TABLE SQL command is triggered using the PHP myql_query() API function, as demonstrated in the following PHP code snippet: $sql_prefix = 'desiredDB_'; $sql_cmd = 'CREATE TABLE `'.$sql_prefix.'users` (`id` int AUTO_INCREMENT KEY, `user` text)'; $tabCreated = mysql_query($sql_cmd, $conn); In the preceding PHP code snippet, we created a table, desiredDB_users, which consists of two columns: id and user. The id column holds an integer value and possesses the SQL AUTO_INCREMENT and KEY options. These SQL options indicate that the MySQL server should set a unique value for each row that is associated to the id column, and that the user has no control over this value. The user column holds string values and is set with the value indicated by the requester when a new row is inserted into our MySQL database. Let's write the following Node.js code snippet, which is equivalent to our preceding PHP code: var sql_prefix = 'desiredDB_'; var sql_cmd = 'CREATE TABLE `'+sql_prefix+'users` (`id` int AUTO_INCREMENT KEY, `user` text)'; var tabCreated = false; conn.query(sql_cmd, function(err, rows, fields) { if (!e) { tabCreated = true; } }); Here, the err parameter that is placed in our query() method's callback function indicates whether any error has been triggered or not. Deleting a table Let's now learn how to delete a table and attempt to delete the same table, users, we just created. This activity is quite similar to creating a table, which we recently discussed. In PHP, we use the DROP TABLE SQL command to achieve our purpose as demonstrated in the following PHP code snippet: $sql_prefix = 'desiredDB_'; $sql_cmd = 'DROP TABLE `'.$sql_prefix.'users`'; $tabDropped = mysql_query($sql_cmd, $conn); Converting the preceding PHP code snippet into its corresponding Node.js code snippet, we follow the same basis as we discussed while creating the table. Our converted Node.js code snippet will look like the following code snippet: var sql_prefix = 'desiredDB_'; var sql_cmd = 'DROP TABLE `'+sql_prefix+'users`'; var tabDropped = false; conn.query(sql_cmd, function(err, rows, fields) { if (!err) { tabDropped = true; } }); Using a SELECT statement Coming to the SQL SELECT statement, it is used to read data from the database tables and functions in a bit different way. In PHP, the PHP mysql_query() API function triggers the statement and returns a result object that is stored in the PHP $sql_result variable. In order to access the actual data, the PHP mysql_fetch_assoc() API function is implemented that runs in a loop in order to fetch the data from more than one row. We assume that we have retained our users table and all the records that we had deleted recently. Our PHP code snippet will look like the following one: $sql_prefix = 'desiredDB_'; $sql_cmd = 'SELECT user FROM `'.$sql_prefix.'users`'; $sql_result = mysql_query($sql_cmd, $conn); while ($row = mysql_fetch_assoc($sql_result)) { $user = $row['user']; print $user; } It is always good to extract the PHP $row variable into an array-free variable (in our case, $user), due to the fact that doing so eliminates complexity when converting PHP code to its corresponding Node.js code. When converting the preceding PHP code snippet into a Node.js code snippet, we use the Node.js query() method to trigger the statement that returns the data as arguments to the callback function. The rows parameter to the callback function holds a two-dimensional array of data, that is, an indexed array of rows along with the array of values associated to each row. Our Node.js code snippet for the preceding PHP code snippet will look like the following one: var sql_prefix = 'desiredDB_'; var sql_cmd = 'SELECT user FROM `'.$sql_prefix.'users`'; conn.query(sql_cmd, function(err, rows, fields) { if (!err) { for (var i=0; i < rows.length; ++i) { var row = rows[i]; var user = row['user']; console.log(user); } } }); In the preceding Node.js code snippet, we could have defined row[i]['user'] so as to show that the Node.js rows variable is a two-dimensional array. Using the UPDATE statement Now, let's try out implementing the SQL UPDATE statement that is used to modify any data of a table. In PHP, we trigger the SQL UPDATE statement by using the PHP mysql_query() API function, as demonstrated in the following code snippet: $sql_prefix = 'desiredDB_'; $sql_cmd = 'UPDATE `'.$sql_prefix.'users` SET `user`="mohan" WHERE `user`="surendra"'; $tabUpdated = mysql_query($sql_cmd, $conn); if ($tabUpdated) { $rows_updated = mysql_affected_rows($conn); print 'Updated '.$rows_updated.' rows.'; } Here, the PHP mysql_affected_rows() API function returns the number of rows that have been modified due to the SQL UPDATE statement. In Node.js, we use the same SQL UPDATE statement. Additionally, we use the affectedRows property of the row's object that holds the same value which is returned out of the PHP mysql_affected_rows() API function. Our Node.js converted code snippet will look like the following one: var sql_prefix = 'desiredDB_'; var sql_cmd = 'UPDATE `'+sql_prefix+'users` SET `user`="mohan" WHERE `user`="surendra"'; conn.query(sql_cmd, function(err, rows, fields) { if (!err) { var rows_updated = rows.affectedRows; console.log('Updated '+rows_updated+' rows.'); } }); Using the INSERT statement Now it is time to write the PHP code to insert data into a table, and then convert the code to its equivalent Node.js code. In order to insert data into a table, we use the SQL INSERT statement. In PHP, the SQL INSERT statement is triggered using the PHP mysql_query() API function, as demonstrated in the following PHP code snippet: $sql_prefix = 'desiredDB_'; $sql_cmd = 'INSERT INTO `'.$sql_prefix.'users` (`id`, `user`) VALUES (0, "surmohan")'; $tabInserted = mysql_query($sql_cmd, $conn); if ($tabInserted) { $inserted_id = mysql_insert_id($conn); print 'Successfully inserted row with id='.$inserted_id.'.'; } Here, the PHP mysql_insert_id() API function returns the value of id that is associated to the newly inserted data. In Node.js, we use the same SQL INSERT statement. Additionally, we use the insertId property of the row's object that holds the same value that is returned from the PHP mysql_insert_id() API function. The Node.js code snippet that is equivalent to the preceding PHP code snippet looks like the following one: var sql_prefix = 'desiredDB_'; var sql_cmd = 'INSERT INTO `'+sql_prefix+'users` (`id`, `user`) VALUES (0, "surmohan")'; conn.query(sql_cmd, function(err, rows, fields) { if (!err) { var inserted_id = rows.insertId; console.log(''Successfully inserted row with id='+inserted_id+'.'); } }); Using the DELETE statement Finally, we have reached our last activity of this section, that is, use of the SQL DELETE statement. Like the SQL statements we discussed earlier in this section, the SQL DELETE statement is also triggered using the PHP mysql_query() API function as demonstrated in the following PHP code snippet: $sql_prefix = 'desiredDB_'; $sql_cmd = 'DELETE FROM `'.$sql_prefix.'users` WHERE `user`="surmohan"'; $tabDeleted = mysql_query($sql_cmd, $conn); if ($tabDeleted) { $rows_deleted = mysql_affected_rows($conn); print 'Successfully deleted '.$rows_deleted.' rows.'; } In Node.js, we use the same SQL DELETE statement. We also use the affectedRows property that serves us in the same way as discussed while dealing with the SQL UPDATE statement. The equivalent Node.js code snippet will look like the following one: var sql_prefix = 'desiredDB_'; var sql_cmd = 'DELETE FROM `'+sql_prefix+'users` WHERE `user`="surmohan"'; conn.query(sql_cmd, function(err, rows, fields) { if (!err) { var rows_deleted = rows.affectedRows; console.log('Successfully deleted '+rows_deleted +' rows.'); } });
Read more
  • 0
  • 0
  • 9104

article-image-storage-configurations
Packt
07 Sep 2015
21 min read
Save for later

Storage Configurations

Packt
07 Sep 2015
21 min read
In this article by Wasim Ahmed, author of the book Proxmox Cookbook, we will cover topics such as local storage, shared storage, Ceph storage, and a recipe which shows you how to configure the Ceph RBD storage. (For more resources related to this topic, see here.) A storage is where virtual disk images of virtual machines reside. There are many different types of storage systems with many different features, performances, and use case scenarios. Whether it is a local storage configured with direct attached disks or a shared storage with hundreds of disks, the main responsibility of a storage is to hold virtual disk images, templates, backups, and so on. Proxmox supports different types of storages, such as NFS, Ceph, GlusterFS, and ZFS. Different storage types can hold different types of data. For example, a local storage can hold any type of data, such as disk images, ISO/container templates, backup files and so on. A Ceph storage, on the other hand, can only hold a .raw format disk image. In order to provide the right type of storage for the right scenario, it is vital to have a proper understanding of different types of storages. The full details of each storage is beyond the scope of this article, but we will look at how to connect them to Proxmox and maintain a storage system for VMs. Storages can be configured into two main categories: Local storage Shared storage Local storage Any storage that resides in the node itself by using directly attached disks is known as a local storage. This type of storage has no redundancy other than a RAID controller that manages an array. If the node itself fails, the storage becomes completely inaccessible. The live migration of a VM is impossible when VMs are stored on a local storage because during migration, the virtual disk of the VM has to be copied entirely to another node. A VM can only be live-migrated when there are several Proxmox nodes in a cluster and the virtual disk is stored on a shared storage accessed by all the nodes in the cluster. Shared storage A shared storage is one that is available to all the nodes in a cluster through some form of network media. In a virtual environment with shared storage, the actual virtual disk of the VM may be stored on a shared storage, while the VM actually runs on another Proxmox host node. With shared storage, the live migration of a VM becomes possible without powering down the VM. Multiple Proxmox nodes can share one shared storage, and VMs can be moved around since the virtual disk is stored on different shared storages. Usually, a few dedicated nodes are used to configure a shared storage with their own resources apart from sharing the resources of a Proxmox node, which could be used to host VMs. In recent releases, Proxmox has added some new storage plugins that allow users to take advantage of some great storage systems and integrating them with the Proxmox environment. Most of the storage configurations can be performed through the Proxmox GUI. Ceph storage Ceph is a powerful distributed storage system, which provides RADOS Block Device (RBD) object storage, Ceph filesystem (CephFS), and Ceph Object Storage. Ceph is built with a very high-level of reliability, scalability, and performance in mind. A Ceph cluster can be expanded to several petabytes without compromising data integrity, and can be configured using commodity hardware. Any data written to the storage gets replicated across a Ceph cluster. Ceph was originally designed with big data in mind. Unlike other types of storages, the bigger a Ceph cluster becomes, the higher the performance. However, it can also be used in small environments just as easily for data redundancy. A lower performance can be mitigated using SSD to store Ceph journals. Refer to the OSD Journal subsection in this section for information on journals. The built-in self-healing features of Ceph provide unprecedented resilience without a single point of failure. In a multinode Ceph cluster, the storage can tolerate not just hard drive failure, but also an entire node failure without losing data. Currently, only an RBD block device is supported in Proxmox. Ceph comprises a few components that are crucial for you to understand in order to configure and operate the storage. The following components are what Ceph is made of: Monitor daemon (MON) Object Storage Daemon (OSD) OSD Journal Metadata Server (MSD) Controlled Replication Under Scalable Hashing map (CRUSH map) Placement Group (PG) Pool MON Monitor daemons form quorums for a Ceph distributed cluster. There must be a minimum of three monitor daemons configured on separate nodes for each cluster. Monitor daemons can also be configured as virtual machines instead of using physical nodes. Monitors require a very small amount of resources to function, so allocated resources can be very small. A monitor can be set up through the Proxmox GUI after the initial cluster creation. OSD Object Storage Daemons (OSDs) are responsible for the storage and retrieval of actual cluster data. Usually, each physical storage device, such as HDD or SSD, is configured as a single OSD. Although several OSDs can be configured on a single physical disc, it is not recommended for any production environment at all. Each OSD requires a journal device where data first gets written and later gets transferred to an actual OSD. By storing journals on fast-performing SSDs, we can increase the Ceph I/O performance significantly. Thanks to the Ceph architecture, as more and more OSDs are added into the cluster, the I/O performance also increases. An SSD journal works very well on small clusters with about eight OSDs per node. OSDs can be set up through the Proxmox GUI after the initial MON creation. OSD Journal Every single piece of data that is destined to be a Ceph OSD first gets written in a journal. A journal allows OSD daemons to write smaller chunks to allow the actual drives to commit writes that give more time. In simpler terms, all data gets written to journals first, then the journal filesystem sends data to an actual drive for permanent writes. So, if the journal is kept on a fast-performing drive, such as SSD, incoming data will be written at a much higher speed, while behind the scenes, slower performing SATA drives can commit the writes at a slower speed. Journals on SSD can really improve the performance of a Ceph cluster, especially if the cluster is small, with only a few terabytes of data. It should also be noted that if there is a journal failure, it will take down all the OSDs that the journal is kept on the journal drive. In some environments, it may be necessary to put two SSDs to mirror RAIDs and use them as journaling. In a large environment with more than 12 OSDs per node, performance can actually be gained by collocating a journal on the same OSD drive instead of using SSD for a journal. MDS The Metadata Server (MDS) daemon is responsible for providing the Ceph filesystem (CephFS) in a Ceph distributed storage system. MDS can be configured on separate nodes or coexist with already configured monitor nodes or virtual machines. Although CephFS has come a long way, it is still not fully recommended to use in a production environment. It is worth mentioning here that there are many virtual environments actively running MDS and CephFS without any issues. Currently, it is not recommended to configure more than two MDSs in a Ceph cluster. CephFS is not currently supported by a Proxmox storage plugin. However, it can be configured as a local mount and then connected to a Proxmox cluster through the Directory storage. MDS cannot be set up through the Proxmox GUI as of version 3.4. CRUSH map A CRUSH map is the heart of the Ceph distributed storage. The algorithm for storing and retrieving user data in Ceph clusters is laid out in the CRUSH map. CRUSH allows a Ceph client to directly access an OSD. This eliminates a single point of failure and any physical limitations of scalability since there are no centralized servers or controllers to manage data in and out. Throughout Ceph clusters, CRUSH maintains a map of all MONs and OSDs. CRUSH determines how data should be chunked and replicated among OSDs spread across several local nodes or even nodes located remotely. A default CRUSH map is created on a freshly installed Ceph cluster. This can be further customized based on user requirements. For smaller Ceph clusters, this map should work just fine. However, when Ceph is deployed with very big data in mind, this map should be customized. A customized map will allow better control of a massive Ceph cluster. To operate Ceph clusters of any size successfully, a clear understanding of the CRUSH map is mandatory. For more details on the Ceph CRUSH map, visit http://ceph.com/docs/master/rados/operations/crush-map/ and http://cephnotes.ksperis.com/blog/2015/02/02/crushmap-example-of-a-hierarchical-cluster-map. As of Proxmox VE 3.4, we cannot customize the CRUSH map throughout the Proxmox GUI. It can only be viewed through a GUI and edited through a CLI. PG In a Ceph storage, data objects are aggregated in groups determined by CRUSH algorithms. This is known as a Placement Group (PG) since CRUSH places this group in various OSDs depending on the replication level set in the CRUSH map and the number of OSDs and nodes. By tracking a group of objects instead of the object itself, a massive amount of hardware resources can be saved. It would be impossible to track millions of individual objects in a cluster. The following diagram shows how objects are aggregated in groups and how PG relates to OSD: To balance available hardware resources, it is necessary to assign the right number of PGs. The number of PGs should vary depending on the number of OSDs in a cluster. The following is a table of PG suggestions made by Ceph developers: Number of OSDs Number of PGs Less than 5 OSDs 128 Between 5-10 OSDs 512 Between 10-50 OSDs 4096 Selecting the proper number of PGs is crucial since each PG will consume node resources. Too many PGs for the wrong number of OSDs will actually penalize the resource usage of an OSD node, while very few assigned PGs in a large cluster will put data at risk. A rule of thumb is to start with the lowest number of PGs possible, then increase them as the number of OSDs increases. For details on Placement Groups, visit http://ceph.com/docs/master/rados/operations/placement-groups/. There's a great PG calculator created by Ceph developers to calculate the recommended number of PGs for various sizes of Ceph clusters at http://ceph.com/pgcalc/. Pools Pools in Ceph are like partitions on a hard drive. We can create multiple pools on a Ceph cluster to separate stored data. For example, a pool named accounting can hold all the accounting department data, while another pool can store the human resources data of a company. When creating a pool, assigning the number of PGs is necessary. During the initial Ceph configuration, three default pools are created. They are data, metadata, and rbd. Deleting a pool will delete all stored objects permanently. For details on Ceph and its components, visit http://ceph.com/docs/master/. The following diagram shows a basic Proxmox+Ceph cluster: The preceding diagram shows four Proxmox nodes, three Monitor nodes, three OSD nodes, and two MDS nodes comprising a Proxmox+Ceph cluster. Note that Ceph is on a different network than the Proxmox public network. Depending on the set replication number, each incoming data object needs to be written more than once. This causes high bandwidth usage. By separating Ceph on a dedicated network, we can ensure that a Ceph network can fully utilize the bandwidth. On advanced clusters, a third network is created only between Ceph nodes for cluster replication, thus improving network performance even further. As of Proxmox VE 3.4, the same node can be used for both Proxmox and Ceph. This provides a great way to manage all the nodes from the same Proxmox GUI. It is not advisable to put Proxmox VMs on a node that is also configured as Ceph. During day-to-day operations, Ceph nodes do not consume large amounts of resources, such as CPU or memory. However, when Ceph goes into rebalancing mode due to OSD or node failure, a large amount of data replication occurs, which takes up lots of resources. Performance will degrade significantly if resources are shared by both VMs and Ceph. Ceph RBD storage can only store .raw virtual disk image files. Ceph itself does not come with a GUI to manage, so having the option to manage Ceph nodes through the Proxmox GUI makes administrative tasks mush easier. Refer to the Monitoring the Ceph storage subsection under the How to do it... section of the Connecting the Ceph RBD storage recipe later in this article to learn how to install a great read-only GUI to monitor Ceph clusters. Connecting the Ceph RBD storage In this recipe, we are going to see how to configure a Ceph block storage with a Proxmox cluster. Getting ready The initial Ceph configuration on a Proxmox cluster must be accomplished through a CLI. After the Ceph installation, initial configurations and one monitor creation for all other tasks can be accomplished through the Proxmox GUI. How to do it... We will now see how to configure the Ceph block storage with Proxmox. Installing Ceph on Proxmox Ceph is not installed by default. Prior to configuring a Proxmox node for the Ceph role, Ceph needs to be installed and the initial configuration must be created through a CLI. The following steps need to be performed on all Proxmox nodes that will be part of the Ceph cluster: Log in to each node through SSH or a console. Configure a second network interface to create a separate Ceph network with a different subnet. Reboot the nodes to initialize the network configuration. Using the following command, install the Ceph package on each node: # pveceph install –version giant Initializing the Ceph configuration Before Ceph is usable, we have to create the initial Ceph configuration file on one Proxmox+Ceph node. The following steps need to be performed only on one Proxmox node that will be part of the Ceph cluster: Log in to the node using SSH or a console. Run the following command create the initial Ceph configuration: # pveceph init –network <ceph_subnet>/CIDR Run the following command to create the first monitor: # pveceph createmon Configuring Ceph through the Proxmox GUI After the initial Ceph configuration and the creation of the first monitor, we can continue with further Ceph configurations through the Proxmox GUI or simply run the Ceph Monitor creation command on other nodes. The following steps show how to create Ceph Monitors and OSDs from the Proxmox GUI: Log in to the Proxmox GUI as a root or with any other administrative privilege. Select a node where the initial monitor was created in previous steps, and then click on Ceph from the tabbed menu. The following screenshot shows a Ceph cluster as it appears after the initial Ceph configuration: Since no OSDs have been created yet, it is normal for a new Ceph cluster to show PGs stuck and unclean error Click on Disks on the bottom tabbed menu under Ceph to display the disks attached to the node, as shown in the following screenshot: Select an available attached disk, then click on the Create: OSD button to open the OSD dialog box, as shown in the following screenshot: Click on the Journal Disk drop-down menu to select a different device or collocate the journal on the same OSD by keeping it as the default. Click on Create to finish the OSD creation. Create additional OSDs on Ceph nodes as needed. The following screenshot shows a Proxmox node with three OSDs configured: By default, Proxmox has created OSDs with an ext3 partition. However, sometimes, it may be necessary to create OSDs with different partition types due to a requirement or for performance improvement. Enter the following command format through the CLI to create an OSD with a different partition type: # pveceph createosd –fstype ext4 /dev/sdX The following steps show how to create Monitors through the Proxmox GUI: Click on Monitor from the tabbed menu under the Ceph feature. The following screenshot shows the Monitor status with the initial Ceph Monitor we created earlier in this recipe: Click on Create to open the Monitor dialog box. Select a Proxmox node from the drop-down menu. Click on the Create button to start the monitor creation process. Create a total of three Ceph monitors to establish a Ceph quorum. The following screenshot shows the Ceph status with three monitors and OSDs added: Note that even with three OSDs added, the PGs are still stuck with errors. This is because by default, the Ceph CRUSH is set up for two replicas. So far, we've only created OSDs on one node. For a successful replication, we need to add some OSDs on the second node so that data objects can be replicated twice. Follow the steps described earlier to create three additional OSDs on the second node. After creating three more OSDs, the Ceph status should look like the following screenshot: Managing Ceph pools It is possible to perform basic tasks, such as creating and removing Ceph pools through the Proxmox GUI. Besides these, we can see check the list, status, number of PGs, and usage of the Ceph pools. The following steps show how to check, create, and remove Ceph pools through the Proxmox GUI: Click on the Pools tabbed menu under Ceph in the Proxmox GUI. The following screenshot shows the status of the default rbd pool, which has replica 1, 256 PG, and 0% usage: Click on Create to open the pool creation dialog box. Fill in the required information, such as the name of the pool, replica size, and number of PGs. Unless the CRUSH map has been fully customized, the ruleset should be left at the default value 0. Click on OK to create the pool. To remove a pool, select the pool and click on Remove. Remember that once a Ceph pool is removed, all the data stored in this pool is deleted permanently. To increase the number of PGs, run the following command through the CLI: #ceph osd pool set <pool_name> pg_num <value> #ceph osd pool set <pool_name> pgp_num <value> It is only possible to increase the PG value. Once increased, the PG value can never be decreased. Connecting RBD to Proxmox Once a Ceph cluster is fully configured, we can proceed to attach it to the Proxmox cluster. During the initial configuration file creation, Ceph also creates an authentication keyring in the /etc/ceph/ceph.client.admin.keyring directory path. This keyring needs to be copied and renamed to match the name of the storage ID to be created in Proxmox. Run the following commands to create a directory and copy the keyring: # mkdir /etc/pve/priv/ceph # cd /etc/ceph/ # cp ceph.client.admin.keyring /etc/pve/priv/ceph/<storage>.keyring For our storage, we are naming it rbd.keyring. After the keyring is copied, we can attach the Ceph RBD storage with Proxmox using the GUI: Click on Datacenter, then click on Storage from the tabbed menu. Click on the Add drop-down menu and select the RBD storage plugin. Enter the information as described in the following table: Item Type of value Entered value ID The name of the storage. rbd Pool The name of the Ceph pool. rbd Monitor Host The IP address and port number of the Ceph MONs. We can enter multiple MON hosts for redundancy. 172.16.0.71:6789;172.16.0.72:6789; 172.16.0.73:6789 User name The default Ceph administrator. Admin Nodes The Proxmox nodes that will be able to use the storage. All Enable The checkbox for enabling/disabling the storage. Enabled Click on Add to attach the RBD storage. The following screenshot shows the RBD storage under Summary: Monitoring the Ceph storage Ceph itself does not come with any GUI to manage or monitor the cluster. We can view the cluster status and perform various Ceph-related tasks through the Proxmox GUI. There are several third-party software that allow Ceph-only GUI to manage and monitor the cluster. Some software provide management features, while others provide read-only features for Ceph monitoring. Ceph Dash is such a software that provides an appealing read-only GUI to monitor the entire Ceph cluster without logging on to the Proxmox GUI. Ceph Dash is freely available through GitHub. There are other heavyweight Ceph GUI dashboards, such as Kraken, Calamari, and others. In this section, we are only going to see how to set up the Ceph Dash cluster monitoring GUI. The following steps can be used to download and start Ceph Dash to monitor a Ceph cluster using any browser: Log in to any Proxmox node, which is also a Ceph MON. Run the following commands to download and start the dashboard: # mkdir /home/tools # apt-get install git # git clone https://github.com/Crapworks/ceph-dash # cd /home/tools/ceph-dash # ./ceph_dash.py Ceph Dash will now start listening on port 5000 of the node. If the node is behind a firewall, open port 5000 or any other ports with port forwarding in the firewall. Open any browser and enter <node_ip>:5000 to open the dashboard. The following screenshot shows the dashboard of the Ceph cluster we have created: We can also monitor the status of the Ceph cluster through a CLI using the following commands: To check the Ceph status: # ceph –s To view OSDs in different nodes: # ceph osd tree To display real-time Ceph logs: # ceph –w To display a list of Ceph pools: # rados lspools To change the number of replicas of a pool: # ceph osd pool set size <value> Besides the preceding commands, there are many more CLI commands to manage Ceph and perform advanced tasks. The Ceph official documentation has a wealth of information and how-to guides along with the CLI commands to perform them. The documentation can be found at http://ceph.com/docs/master/. How it works… At this point, we have successfully integrated a Ceph cluster with a Proxmox cluster, which comprises six OSDs, three MONs, and three nodes. By viewing the Ceph Status page, we can get lot of information about a Ceph cluster at a quick glance. From the previous figure, we can see that there are 256 PGs in the cluster and the total cluster storage space is 1.47 TB. A healthy cluster will have the PG status as active+clean. Based on the nature of issue, the PGs can have various states, such as active+unclean, inactive+degraded, active+stale, and so on. To learn details about all the states, visit http://ceph.com/docs/master/rados/operations/pg-states/. By configuring a second network interface, we can separate a Ceph network from the main network. The #pveceph init command creates a Ceph configuration file in the /etc/pve/ceph.conf directory path. A newly configured Ceph configuration file looks similar to the following screenshot: Since the ceph.conf configuration file is stored in pmxcfs, any changes made to it are immediately replicated in all the Proxmox nodes in the cluster. As of Proxmox VE 3.4, Ceph RBD can only store a .raw image format. No templates, containers, or backup files can be stored on the RBD block storage. Here is the content of a storage configuration file after adding the Ceph RBD storage: rbd: rbd monhost 172.16.0.71:6789;172.16.0.72:6789;172.16.0.73:6789 pool rbd content images username admin If a situation dictates the IP address change of any node, we can simply edit this content in the configuration file to manually change the IP address of the Ceph MON nodes. See also To learn about Ceph in greater detail, visit http://ceph.com/docs/master/ for the official Ceph documentation Also, visit https://indico.cern.ch/event/214784/session/6/contribution/68/material/slides/0.pdf to find out why Ceph is being used at CERN to store the massive data generated by the Large Hadron Collider (LHC) Summary In this article, we came across with different configurations for a variety of storage categories and got hands-on practice with various stages in configuring the Ceph RBD storage. Resources for Article: Further resources on this subject: Deploying New Hosts with vCenter [article] Let's Get Started with Active Di-rectory [article] Basic Concepts of Proxmox Virtual Environment [article]
Read more
  • 0
  • 0
  • 9102

article-image-making-better-faq-page
Packt
24 Jul 2014
17 min read
Save for later

Making a Better FAQ Page

Packt
24 Jul 2014
17 min read
(For more resources related to this topic, see here.) Marking up the FAQ page We'll get started by taking some extra care and attention with the way we mark up our FAQ list. As with most things that deal with web development, there's no right way of doing anything, so don't assume this approach is the only correct one. Any markup that makes sense semantically and makes it easy to enhance your list with CSS and JavaScript is perfectly acceptable. Time for action – setting up the HTML file Perform the following steps to get the HTML file set up for our FAQ page: We'll get started with our sample HTML file, the jQuery file, the scripts.js file, and the styles.css file. In this case, our HTML page will contain a definition list with the questions inside the <dt> tags and the answers wrapped in the <dd> tags. By default, most browsers will indent the <dd> tags, which means the questions hang into the left margin, making them easy to scan. Inside the <body> tag of your HTML document, add a heading and a definition list as shown in the following code: <h1>Frequently Asked Questions</h1> <dl> <dt>What is jQuery?</dt> <dd> <p>jQuery is an awesome JavaScript library</p> </dd> <dt>Why should I use jQuery?</dt> <dd> <p>Because it's awesome and it makes writing JavaScript faster and easier</p> </dd> <dt>Why would I want to hide the answers to my questions?</dt> <dd> <p>To make it easier to peruse the list of available questions - then you simply click to see the answer you're interested in reading.</p> </dd> <dt>What if my answers were a lot longer and more complicated than these examples?</dt> <dd> <p>The great thing about the &lt;dd&gt; element is that it's a block level element that can contain lots of other elements.</p> <p>That means your answer could contain:</p> <ul> <li>Unordered</li> <li>Lists</li> <li>with lots</li> <li>of items</li> <li>(or ordered lists or even another definition list)</li> </ul> <p>Or it might contain text with lots of <strong>special</strong> <em>formatting</em>.</p> <h2>Other things</h2> <p>It can even contain headings. Your answers could take up an entire screen or more all on their own - it doesn't matter since the answer will be hidden until the user wants to see it.</p> </dd> <dt>What if a user doesn't have JavaScript enabled?</dt> <dd> <p>You have two options for users with JavaScript disabled - which you choose might depend on the content of your page.</p> <p>You might just leave the page as it is - and make sure the &lt;dt&gt; tags are styled in a way that makes them stand out and easy to pick up when you're scanning down through the page. This would be a great solution if your answers are relatively short.</p> <p>If your FAQ page has long answers, it might be helpful to put a table of contents list of links to individual questions at the top of the page so users can click it to jump directly to the question and answer they're interested in.This is similar to what we did in the tabbed example, but in this case, we'd usejQuery to hide the table of contents when the page loaded since users with JavaScript wouldn't need to see the table of contents.</p> </dd> </dl> You can adjust the style of the page however you'd like by adding in some CSS styles. The following screenshot shows how the page is styled: For users with JavaScript disabled, this page works fine as is. The questions hang into the left margin and are bolder and larger than the rest of the text on the page, making them easy to scan. What just happened? We set up a basic definition list to hold our questions and answers. The default style of the definition list lends itself nicely to making the list of questions scannable for site visitors without JavaScript. We can enhance that further with our own custom CSS code to make the style of our list match our site. As this simple collapse-and-show (or accordion) action is such a common one, two new elements have been proposed for HTML5: <summary> and <details> that will enable us to build accordions in HTML without the need for JavaScript interactivity. However, at the time of writing this, the new elements are only supported in Webkit browsers, which require some finagling to get them styled with CSS, and are also not accessible. Do keep an eye on these new elements to see if more widespread support for them develops. You can read about the elements in the HTML5 specs (http://www.whatwg.org/specs/web-apps/current-work/multipage/interactive-elements.html). If you'd like to understand the elements better, the HTML5 Doctor has a great tutorial that explains their use and styling at http://html5doctor.com/the-details-and-summary-elements/. Time for action – moving around an HTML document Perform the following steps to move from one element to another in JavaScript: We're going to keep working with the files we set up in the previously. Open up the scripts.js file that's inside your scripts folder. Add a document ready statement, then write a new empty function called dynamicFAQ, as follows: $(document).ready(function(){ }); function dynamicFAQ() { // Our function will go here } Let's think through how we'd like this page to behave. We'd like to have all the answers to our questions hidden when the page is loaded. Then, when a user finds the question they're looking for, we'd like to show the associated answer when they click on the question. This means the first thing we'll need to do is hide all the answers when the page loads. Get started by adding a class jsOff to the <body> tag, as follows: <body class="jsOff"> Now, inside the document ready statement in scripts.js, add the line of code that removes the jsOff class and adds a class selector of jsOn: $(document).ready(function(){ $('body').removeClass('jsOff').addClass('jsOn'); }); Finally, in the styles.css file, add this bit of CSS to hide the answers for the site visitors who have JavaScript enabled: .jsOn dd { display: none; } Now if you refresh the page in the browser, you'll see that the <dd> elements and the content they contain are no longer visible (see the following screenshot): Now, we need to show the answer when the site visitor clicks on a question. To do that, we need to tell jQuery to do something whenever someone clicks on one of the questions or the <dt> tags. Inside the dynamicFAQ function, add a line of code to add a click event handler to the <dt> elements, as shown in the following code: function dynamicFAQ() { $('dt').on('click', function(){ //Show function will go here }); } When the site visitor clicks on a question, we want to get the answer to that question and show it because our FAQ list is set up as follows: <dl> <dt>Question 1</dt> <dd>Answer to Question 1</dd> <dt>Question 2</dt> <dd>Answer to Question 2</dd> ... </dl> We know that the answer is the next node or element in the DOM after our question. We'll start from the question. When a site visitor clicks on a question, we can get the current question by using jQuery's $(this) selector. The user has just clicked on a question, and we say $(this) to mean the question they just clicked on. Inside the new click function, add $(this) so that we can refer to the clicked question, as follows: $('dt').on('click', function(){ $(this); }); Now that we have the question that was just clicked, we need to get the next thing, or the answer to that question so that we can show it. This is called traversing the DOM in JavaScript. It just means that we're moving to a different element in the document. jQuery gives us the next method to move to the next node in the DOM. We'll select our answer by inserting the following code: $('dt').on('click', function(){ $(this).next(); }); Now, we've moved from the question to the answer. Now all that's left to do is show the answer. To do so, add a line of code as follows: $('dt').on('click', function(){ $(this).next().show(); }); If you refresh the page in the browser, you might be disappointed to see that nothing happens when we click the questions. Don't worry—that's easy to fix. We wrote a dynamicFAQ() function, but we didn't call it. Functions don't work until they're called. Inside the document ready statement, call the function as follows: $(document).ready(function(){ $('body').removeClass('jsOff').addClass('jsOn'); dynamicFAQ(); }); Now, if we load the page in the browser, you can see that all of our answers are hidden until we click on the question. This is nice and useful, but it would be even nicer if the site visitor could hide the answer again when they're done reading it to get it out of their way. Luckily, this is such a common task, jQuery makes this very easy for us. All we have to do is replace our call to the show method with a call to the toggle method as follows: $('dt').on('click', function(){ $(this).next().toggle(); }); Now when you refresh the page in the browser, you'll see that clicking on the question once shows the answer and clicking on the question a second time hides the answer again. What just happened? We learned how to traverse the DOM—how to get from one element to another. Toggling the display of elements on a page is a common JavaScript task, so jQuery already has built-in methods to handle it and make it simple and straightforward to get this up and running on our page. That was pretty easy—just a few lines of code. Sprucing up our FAQ page That was so easy, in fact, that we have plenty of time left over to enhance our FAQ page to make it even better. This is where the power of jQuery becomes apparent—you can not only create a show/hide FAQ page, but you can make it a fancy one and still meet your deadline. How's that for impressing a client or your boss? Time for action – making it fancy Perform the following steps to add some fancy new features to the FAQ page: Let's start with a lit le CSS code to change the cursor to a pointer and add a little hover effect to our questions to make it obvious to site visitors that the questions are clickable. Open up the styles.css file that's inside the styles folder and add the following bit of CSS code: .jsOn dt { cursor: pointer; } .jsOn dt:hover { color: #ac92ec; } We're only applying these styles for those site visitors that have JavaScript enabled. These styles definitely help to communicate to the site visitor that the questions are clickable. You might also choose to change something other than the font color for the hover effect. Feel free to style your FAQ list however you'd like. Have a look at the following screenshot: Now that we've made it clear that our <dt> elements can be interacted with, let's take a look at how to show the answers in a nicer way. When we click on a question to see the answer, the change isn't communicated to the site visitor very well; the jump in the page is a little disconcerting and it takes a moment to realize what just happened. It would be nicer and easier to understand if the questions were to slide into view. The site visitor could literally see the question appearing and would understand immediately what change just happened on the screen. jQuery makes that easy for us. We just have to replace our call to the toggle method with a call to the slideToggle method: $('dt').on('click', function(){ $(this).next().slideToggle(); }); Now if you view the page in your browser, you can see that the questions slide smoothly in and out of view when the question is clicked. It's easy to understand what's happening when the page changes, and the animation is a nice touch. Now, there's just one lit le detail we've still got to take care of. Depending on how you've styled your FAQ list, you might see a lit le jump in the answer at the end of the animation. This is caused by some extra margins around the <p> tags inside the <dd> element. They don't normally cause any issues in HTML, and browsers can figure how to display them correctly. However, when we start working with animation, sometimes this becomes a problem. It's easy to fix. Just remove the top margin from the <p> tags inside the FAQ list as follows: .content dd p { margin-top: 0; } If you refresh the page in the browser, you'll see that the little jump is now gone and our animation smoothly shows and hides the answers to our questions. What just happened? We replaced our toggle method with the slideToggle method to animate the showing and hiding of the answers. This makes it easier for the site visitor to understand the change that's taking place on the page. We also added some CSS to make the questions appear to be clickable to communicate the abilities of our page to our site visitors. We're almost there! jQuery made animating that show and hide so easy that we've still got time left over to enhance our FAQ page even more. It would be nice to add some sort of indicator to our questions to show that they're collapsed and can be expanded, and to add some sort of special style to our questions once they're opened to show that they can be collapsed again. Time for action – adding some final touches Perform the following steps to add some finishing touches to our FAQ list: Let's start with some simple CSS code to add a small arrow icon to the left side of our questions. Head back into style.css and modify the styles a bit to add an arrow as follows: .jsOn dt:before { border: 0.5em solid; border-color: transparent transparent transparent #f2eeef; content: ''; display: inline-block; height: 0; margin-right: 0.5em; vertical-align: middle; width: 0; } .jsOn dt:hover:before { border-left-color: #ac92ec; } You might be wondering about this sort of odd bit of CSS. This is a technique to create triangles in pure CSS without having to use any images. If you're not familiar with this technique, I recommend checking out appendTo's blog post that explains pure CSS triangles at http://appendto.com/2013/03/pure-css-triangles-explained/. We've also included a hover style so that the triangle will match the text color when the site visitor hovers his/her mouse over the question. Note that we're using the jsOn class so that arrows don't get added to the page unless the site visitors have JavaScript enabled. See the triangles created in the following screenshot: Next, we'll change the arrow to a different orientation when the question is opened. We'll create a new CSS class open and use it to de fine some new styles for our CSS arrow using the following code: .jsOn dt.open:before { border-color: #f2eeef transparent transparent transparent; border-bottom-width: 0; } .jsOn dt.open:hover:before { border-left-color: transparent; border-top-color: #ac92ec; } Just make sure you add these new classes after the other CSS we're using to style our <dt> tags. This will ensure that the CSS cascades the way we intended. So we have our CSS code to change the arrows and show our questions are open, but how do we actually use that new class? We'll use jQuery to add the class to our question when it is opened and to remove the class when it's closed. jQuery provides some nice methods to work with CSS classes. The addClass method will add a class to a jQuery object and the removeClass method will remove a class. However, we want to toggle our class just like we're toggling the show and hide phenomenon of our questions. jQuery's got us covered for that too. We want the class to change when we click on the question, so we'll add a line of code inside our dynamicFAQ function that we're calling each time a <dt> tag is clicked as follows: $('dt').on('click', function(){ $(this).toggleClass('open'); $(this).next().slideToggle(); }); Now when you view the page, you'll see your open styles being applied to the <dt> tags when they're open and removed again when they're closed. To see this, have a look at the following screenshot: However, we can actually crunch our code to be a little bit smaller. Remember how we chain methods in jQuery? We can take advantage of chaining again. We have a bit of redundancy in our code because we're starting two different lines with $(this). We can remove this extra $(this) and just add our toggleClass method to the chain we've already started as follows: $(this).toggleClass('open').next().slideToggle(); This helps keep our code short and concise, and just look at what we're accomplishing in one line of code! What just happened? We created the CSS styles to style the open and closed states of our questions, and then we added a bit of code to our JavaScript to change the CSS class of the question to use our new styles. jQuery provides a few different methods to update CSS classes, which is o t en a quick and easy way to update the display of our document in response to input from the site visitor. In this case, since we wanted to add and remove a class, we used the toggleClass method. It saved us from having to figure out on our own whether we needed to add or remove the open class. We also took advantage of chaining to simply add this new functionality to our existing line of code, making the animated show and hide phenomenon of the answer and the change of CSS class of our question happen all in just one line of code. How's that for impressive power in a small amount of code? Summary You learned how to set up a basic FAQ page that hides the answers to the questions until the site visitor needs to see them. Because jQuery made this so simple, we had plenty of time left over to enhance our FAQ page even more, adding animations to our show and hide phenomenon for the answers, and taking advantage of CSS to style our questions with special open and closed classes to communicate to our site visitors how our page works. And we did all of that with just a few lines of code! Resources for Article: Further resources on this subject: Calendars in jQuery 1.3 with PHP using jQuery Week Calendar Plugin: Part 1 [article] Using jQuery and jQuery Animation: Tips and Tricks [article] Using jQuery and jQueryUI Widget Factory plugins with RequireJS [article]
Read more
  • 0
  • 0
  • 9102

article-image-designing-simple-robust-object-detector-and-classifier
Packt
13 Jun 2016
19 min read
Save for later

Designing a Simple, Robust Object Detector and Classifier

Packt
13 Jun 2016
19 min read
In this article by Joseph Howse, author of the book, iOS Application Development with OpenCV 3, illustrates a scale-invariant,rotation-invariant approach to object detection and classification, using OpenCV 3and just 250 lines of custom C++ code. The technique relies on blob detection, histogram analysis, and SURF (or ORB if SURF is unavailable).The classifier is sensitive to colors as well as keypoints, and itcan work with a small number of training images. For background information, sample images, and a complete tutorial on how to integrate this detector and classifier into an iOS application, refer toChapter 5, Classifying Coins and Commodities in the book,iOS Application Development with OpenCV 3 (Packt Publishing, 2016). You could also use this article's C++ code on other platforms besides iOS. (For more resources related to this topic, see here.) Defining blobs and a blob detector For our purposes, a blob simply has an image and a label. The image is cv::Mat and the label is an unsigned integer. The label's default value is 0, which shall signify that the blob has not yet been classified. Create a new header file, Blob.h, and fill it with the following declaration of a Blob class: #ifndef BLOB_H #define BLOB_H   #include <opencv2/core.hpp>   class Blob { public:   Blob(const cv::Mat &mat, uint32_t label = 0ul);     /**    * Construct an empty blob.    */   Blob();     /**    * Construct a blob by copying another blob.    */   Blob(const Blob &other);     bool isEmpty() const;     uint32_t getLabel() const;   void setLabel(uint32_t value);     const cv::Mat &getMat() const;   int getWidth() const;   int getHeight() const;   private:   uint32_t label;     cv::Mat mat; };   #endif // BLOB_H A Blob's image does not change after construction, but the label may change as a result of our classification process. Note that most of Blob's methods have the const modifier, but of course,setLabel does not because it changes the label. Now, let's declare a BlobDetector class in another new header file, BlobDetector.h. This class provides a detect public method to analyze a given image and populate vector<Blob> based on detected objects in the image. Another public method, getMask, returns a thresholded version of the most recent image that the detect method received. Internally, BlobDetector uses several more matrices and vectors to hold intermediate results, including the mask, detected edges, detected contours, and hierarchy that describes the contours' relationship to each other. Here is the detector's declaration: class BlobDetector { public:   void detect(cv::Mat &image, std::vector<Blob>&blob,     double resizeFactor = 1.0, bool draw = false);     const cv::Mat &getMask() const;   private:   void createMask(const cv::Mat &image);     cv::Mat resizedImage;   cv::Mat mask;   cv::Mat edges;   std::vector<std::vector<cv::Point>> contours;   std::vector<cv::Vec4i> hierarchy; };   #endif // !BLOB_DETECTOR_H Later, in the Detecting blobs against a plain background section, we will define the methods' bodies in new files called Blob.cpp and BlobDetector.cpp. Defining blob descriptors and a blob classifier If you are familiar with keypoint matching, you know that a keypoint has a descriptor or set of descriptive statistics. Similarly, we can define a custom descriptor for a blob. As our classifier relies on histogram comparison and keypoint matching, let's say that a blob's descriptor consists of a normalized histogram and matrix of keypoint descriptors. The descriptor object is also a convenient place to put the label. Create a new header file, BlobDescriptor.h, and put the following declaration of a BlobDescriptor class in it: #ifndef BLOB_DESCRIPTOR_H #define BLOB_DESCRIPTOR_H   #include <opencv2/core.hpp>   class BlobDescriptor { public:   BlobDescriptor(const cv::Mat &normalizedHistogram,     const cv::Mat &keypointDescriptors, uint32_t label);     const cv::Mat &getNormalizedHistogram() const;   const cv::Mat &getKeypointDescriptors() const;   uint32_t getLabel() const;   private:   cv::Mat normalizedHistogram;   cv::Mat keypointDescriptors;   uint32_t label; };   #endif // !BLOB_DESCRIPTOR_H Note that BlobDescriptor is designed as an immutable class. All its methods, except the constructor, have the const modifier. Now, let's declare a BlobClassifier class in another new header file, BlobClassifier.h. Publicly, this class receives Blob objects via an update method (for reference blobs) and a classify method (for blobs that the detector found in the scene). Privately, BlobClassifier creates, owns, and compares BlobDescriptor objects that pertain to the Blob objects. Thus, BlobClassifier is the only part of our program that needs to deal with BlobDescriptor. BlobClassifier also owns instances of OpenCV classes that are responsible for keypoint detection, description, and matching. Here is our classifier's declaration: #ifndef BLOB_CLASSIFIER_H #define BLOB_CLASSIFIER_H   #import "Blob.h" #import "BlobDescriptor.h"   #include <opencv2/features2d.hpp>   class BlobClassifier { public:   BlobClassifier();     /**    * Add a reference blob to the classification model.    */   void update(const Blob &referenceBlob);     /**    * Clear the classification model.    */   void clear();     /**    * Classify a blob that was detected in a scene.    */   void classify(Blob &detectedBlob) const;   private:   BlobDescriptor createBlobDescriptor(const Blob &blob) const;   float findDistance(const BlobDescriptor &detectedBlobDescriptor,     const BlobDescriptor &referenceBlobDescriptor) const;     /**    * A feature detector and descriptor extractor.    * It finds features in images.    * Then, it creates descriptors of the features.    */   cv::Ptr<cv::Feature2D> featureDetectorAndDescriptorExtractor;     /**    * A descriptor matcher.    * It matches features based on their descriptors.    */   cv::Ptr<cv::DescriptorMatcher> descriptorMatcher;     /**    * Descriptors of the reference blobs.    */   std::vector<BlobDescriptor> referenceBlobDescriptors; };   #endif // !BLOB_CLASSIFIER_H Later, in the Classifying blobs by color and keypoints section, we will write the methods' bodies in new files called BlobDescriptor.cpp and BlobClassifier.cpp. Detecting blobs against a plain background Let's assume that the background has a distinctive color range, such as "cream to snow white". Our blob detector will calculate the image's dominant color range and search for large regions whose colors differ from this range. These anomalous regions will constitute the detected blobs. For small objects such as a bean or coin, a user can easily find a plain background such as a blank sheet of paper, plain table-top, plain piece of clothing, or even the palm of a hand. As our blob detector dynamically estimates the background color range, it can cope with various backgrounds and lighting conditions; it is not limited to a lab environment. Create a new file, BlobDetector.cpp, for the implementation of our BlobDetector class. (To review the header, refer back to the Defining blobs and a blob detector section.) At the top of BlobDetector.cpp, we will define several constants that pertain to the breadth of the background color range, the size and smoothing of the blobs, and the color of the blobs' rectangles in the preview image. Here is the relevant code: #include <opencv2/imgproc.hpp>   #include "BlobDetector.h"   const double MASK_STD_DEVS_FROM_MEAN = 1.0; const double MASK_EROSION_KERNEL_RELATIVE_SIZE_IN_IMAGE = 0.005; const int MASK_NUM_EROSION_ITERATIONS = 8;   const double BLOB_RELATIVE_MIN_SIZE_IN_IMAGE = 0.05;   const cv::Scalar DRAW_RECT_COLOR(0, 255, 0); // Green Of course, the heart of BlobDetector is its detect method. Optionally, the method creates a downsized version of the image for faster processing. Then, we call a helper method, createMask, to perform thresholding and erosion on the (resized) image. We pass the resulting mask to the cv::Canny function to perform Canny edge detection. We pass the edge mask to the cv::findContours function, which populates a vector of contours, in the vector<vector<cv::Point>> format. That is to say, each contour is a vector of points. For each contour, we find the points' bounding rectangle. If we are working with a resized image, we restore the bounding rectangle to the original scale. We reject rectangles that are very small. Finally, for each accepted rectangle, we put a new Blob object in the output vector and optionally draw the rectangle in the original image. Here is the detect method's implementation: void BlobDetector::detect(cv::Mat &image,   std::vector<Blob>&blobs, double resizeFactor, bool draw) {   blobs.clear();     if (resizeFactor == 1.0) {     createMask(image);   } else {     cv::resize(image, resizedImage, cv::Size(), resizeFactor,       resizeFactor, cv::INTER_AREA);     createMask(resizedImage);   }     // Find the edges in the mask.   cv::Canny(mask, edges, 191, 255);     // Find the contours of the edges.   cv::findContours(edges, contours, hierarchy, cv::RETR_TREE,     cv::CHAIN_APPROX_SIMPLE);     std::vector<cv::Rect> rects;   int blobMinSize = (int)(MIN(image.rows, image.cols) *     BLOB_RELATIVE_MIN_SIZE_IN_IMAGE);   for (std::vector<cv::Point> contour : contours) {       // Find the contour's bounding rectangle.     cv::Rect rect = cv::boundingRect(contour);       // Restore the bounding rectangle to the original scale.     rect.x /= resizeFactor;     rect.y /= resizeFactor;     rect.width /= resizeFactor;     rect.height /= resizeFactor;       if (rect.width < blobMinSize || rect.height < blobMinSize) {       continue;     }       // Create the blob from the sub-image inside the bounding     // rectangle.     blobs.push_back(Blob(cv::Mat(image, rect)));       // Remember the bounding rectangle in order to draw it later.     rects.push_back(rect);   }     if (draw) {     // Draw the bounding rectangles.     for (const cv::Rect &rect : rects) {       cv::rectangle(image, rect.tl(), rect.br(), DRAW_RECT_COLOR);     }   } } The getMask method simply returns the mask that we previously computed in the detect method: const cv::Mat &BlobDetector::getMask() const {   return mask; } The createMask helper method begins by finding the image's mean color and standard deviation using the cv::meanStdDev function. We calculate a range of background colors based on a certain number of standard deviations from the mean, as defined by the MASK_STD_DEVS_FROM_MEAN constant near the top of BlobDetector.cpp. We deem values outside this range to be foreground colors. Using the cv::inRange function, we map the background colors (in the image) to white (in the mask) and the foreground colors (in the image) to black (in the mask). Then, we create a square kernel using the cv::getStructuringElement function. Finally, we use the kernel in the cv::erode function to apply the erosion morphological operation to the mask. This has the effect of smoothing the black (foreground) regions such that they swallow up little gaps that are probably just noise. Here is the relevant code: void BlobDetector::createMask(const cv::Mat &image) {     // Find the image's mean color.   // Presumably, this is the background color.   // Also find the standard deviation.   cv::Scalar meanColor;   cv::Scalar stdDevColor;   cv::meanStdDev(image, meanColor, stdDevColor);     // Create a mask based on a range around the mean color.   cv::Scalar halfRange = MASK_STD_DEVS_FROM_MEAN * stdDevColor;   cv::Scalar lowerBound = meanColor - halfRange;   cv::Scalar upperBound = meanColor + halfRange;   cv::inRange(image, lowerBound, upperBound, mask);     // Erode the mask to merge neighboring blobs.   int kernelWidth = (int)(MIN(image.cols, image.rows) *     MASK_EROSION_KERNEL_RELATIVE_SIZE_IN_IMAGE);   if (kernelWidth > 0) {     cv::Size kernelSize(kernelWidth, kernelWidth);     cv::Mat kernel = cv::getStructuringElement(cv::MORPH_RECT,       kernelSize);     cv::erode(mask, mask, kernel, cv::Point(-1, -1),       MASK_NUM_EROSION_ITERATIONS);   } } That is the end of the blob detector's code. As you can see, it uses a general-purpose and rather linear approach, without any special cases for different kinds of objects.Moreover, we are using a separate blob detector and blob classifier, and this separation of responsibilities enables us to keep each class's implementation relatively simple. For completeness, note that the Blob class's constructors have straightforward implementations that copy the arguments. For the blob's image, we make a deep copy because the original may change. (For example, the original may be a subimage in a frame of video, and after detection we may draw rectangles atop the frame of video.) Similarly, Blob's getter and setter methods are self-explanatory. Create a new file, Blop.cpp, and fill it with the following implementation: #import "Blob.h"   Blob::Blob(const cv::Mat &mat, uint32_t label) : label(label) {   mat.copyTo(this->mat); }   Blob::Blob() { }   Blob::Blob(const Blob &other) : label(other.label) {   other.mat.copyTo(mat); }   bool Blob::isEmpty() const {   return mat.empty(); } uint32_t Blob::getLabel() const {   return label; } void Blob::setLabel(uint32_t value) {   label = value; } const cv::Mat &Blob::getMat() const {   return mat; } int Blob::getWidth() const {   return mat.cols; } int Blob::getHeight() const {   return mat.rows; } Classifying blobs by color and keypoints Our classifier operates on the assumption that a blob contains distinctive colors, distinctive keypoints, or both. To conserve memory and precompute as much relevant information as possible, we do not store images of the reference blobs, but instead we store histograms and keypoint descriptors. Create a new file, BlobClassifier.cpp, for the implementation of our BlobClassifier class. (To review the header, refer back to the Defining blob descriptors and a blob classifier section.) At the top of BlobDetector.cpp, we will define several constants that pertain to the number of histogram bins, the histogram comparison method, and the relative importance of the histogram comparison versus the keypoint comparison. Here is the relevant code: #include <opencv2/imgproc.hpp>   #include "BlobClassifier.h"   #ifdef WITH_OPENCV_CONTRIB #include <opencv2/xfeatures2d.hpp> #endif   const int HISTOGRAM_NUM_BINS_PER_CHANNEL = 32; const int HISTOGRAM_COMPARISON_METHOD = cv::HISTCMP_CHISQR_ALT;   const float HISTOGRAM_DISTANCE_WEIGHT = 0.98f; const float KEYPOINT_MATCHING_DISTANCE_WEIGHT = 1.0f -   HISTOGRAM_DISTANCE_WEIGHT; Beware that the HISTOGRAM_NUM_BINS_PER_CHANNEL constant has a cubic relationship to memory usage. For each blob descriptor, we store a three-dimensional (BGR) histogram with HISTOGRAM_NUM_BINS_PER_CHANNEL^3 elements, and each element is a 32-bit floating point number. If the constant is 32, each histogram's size in megabytes is (32^3)*32/(10^6)=1.0. This is fine for a small set of reference descriptors. If the constant is 256 (the maximum number of bins for an 8-bit color channel), the histogram's size goes up to a whopping value of (256^3)*32/(10^6)=536.9 megabytes! For an iOS application, this is unacceptable, given the platform's memory constraints. At best, in a high-end iOS device, one gigabyte of RAM might be available to each application. Conservatively, you should worry if your app's memory usage approaches 100 megabytes. Remember that OpenCV's SURF implementation is in the xfeatures2d module, which is part of opencv_contrib. If opencv_contrib is available, let's define the WITH_OPENCV_CONTRIB preprocessor flag. Then, our code imports the <opencv/xfeatures2d.hpp> header, and we use SURF. Otherwise, we use ORB. This selection also affects the implementation of BlobClassifier's constructor. OpenCV provides factory methods for various feature detectors, descriptors, and matchers, so we simply have to use the right combination of factory methods for SURF with Flann matching or ORB with brute-force matching based on the Hamming distance. Here is the constructor's implementation: BlobClassifier::BlobClassifier() { #ifdef WITH_OPENCV_CONTRIB   featureDetectorAndDescriptorExtractor =     cv::xfeatures2d::SURF::create();   descriptorMatcher = cv::DescriptorMatcher::create("FlannBased"); #else   featureDetectorAndDescriptorExtractor = cv::ORB::create();   descriptorMatcher = cv::DescriptorMatcher::create( "BruteForce-HammingLUT"); #endif } The update method's implementation calls a helper method, createBlobDescriptor, and adds the resulting BlobDescriptor to a vector of reference descriptors: void BlobClassifier::update(const Blob &referenceBlob) {   referenceBlobDescriptors.push_back(     createBlobDescriptor(referenceBlob)); } The clear method's implementation discards all the reference descriptors such that the BlobClassifier reverts to its initial, untrained state: void BlobClassifier::clear() {   referenceBlobDescriptors.clear(); } The implementation of the classify method relies on another helper method, findDistance. For each reference descriptor, classify calls findDistance to obtain a measure of dissimilarity between the query blob's descriptor and reference descriptor. We find the reference descriptor with the least distance (best similarity) and return its label as the classification result. If there are no reference descriptors, classify returns 0, the "unknown" label. Here is classify's implementation: void BlobClassifier::classify(Blob &detectedBlob) const {   BlobDescriptor detectedBlobDescriptor =     createBlobDescriptor(detectedBlob);   float bestDistance = FLT_MAX;   uint32_t bestLabel = 0;   for (const BlobDescriptor &referenceBlobDescriptor :       referenceBlobDescriptors) {     float distance = findDistance(detectedBlobDescriptor,       referenceBlobDescriptor);     if (distance < bestDistance) {       bestDistance = distance;       bestLabel = referenceBlobDescriptor.getLabel();     }   }   detectedBlob.setLabel(bestLabel); } The createBlobDescriptor helper method is responsible for calculating a normalized histogram of Bloband keypoint descriptors and using them to build a new BlobDescriptor. To calculate the (non-normalized) histogram, we use the cv::calcHist function. Among its arguments, it requires three arrays to specify the channels we want to use, the number of bins per channel, and the range of each channel's values. To normalize the resulting histogram, we divide by the number of pixels in the blob's image. The following code, pertaining to the histogram, is the first half of implementation of createBlobDescriptor: BlobDescriptor BlobClassifier::createBlobDescriptor(   const Blob &blob) const {    const cv::Mat &mat = blob.getMat();   int numChannels = mat.channels();     // Calculate the histogram of the blob's image.   cv::Mat histogram;   int channels[] = { 0, 1, 2 };   int numBins[] = { HISTOGRAM_NUM_BINS_PER_CHANNEL,     HISTOGRAM_NUM_BINS_PER_CHANNEL,     HISTOGRAM_NUM_BINS_PER_CHANNEL };   float range[] = { 0.0f, 256.0f };   const float *ranges[] = { range, range, range };   cv::calcHist(&mat, 1, channels, cv::Mat(), histogram, 3,     numBins, ranges);     // Normalize the histogram.   histogram *= (1.0f / (mat.rows * mat.cols)); Now, we must convert the blob's image to grayscale and obtain keypoints and keypoint descriptors using the detect and compute methods of cv::Feature2D. With the normalized histogram and keypoint descriptors, we have everything that we need to construct and return a new BlobDescriptor. Here is the remainder of implementation of createBlobDescriptor: // Convert the blob's image to grayscale.   cv::Mat grayMat;   switch (numChannels) {     case 4:       cv::cvtColor(mat, grayMat, cv::COLOR_BGRA2GRAY);       break;     default:       cv::cvtColor(mat, grayMat, cv::COLOR_BGR2GRAY);       break;   }     // Detect features in the grayscale image.   std::vector<cv::KeyPoint> keypoints;   featureDetectorAndDescriptorExtractor->detect(grayMat,     keypoints);     // Extract descriptors of the features.   cv::Mat keypointDescriptors;   featureDetectorAndDescriptorExtractor->compute(grayMat,     keypoints, keypointDescriptors);     return BlobDescriptor(histogram, keypointDescriptors,     blob.getLabel()); } The findDistance helper method performs histogram comparison using the cv::compareHist function and keypoint matching using the match method of cv::DescriptorMatcher. Each of the resulting keypoint matches has a distance, and we sum these distances. Then, as an overall measure of distance between the two blob descriptors, we return a weighted average of the histogram distance and the total keypoint matching distance. Here is the relevant code: float BlobClassifier::findDistance(   const BlobDescriptor &detectedBlobDescriptor,   const BlobDescriptor &referenceBlobDescriptor) const {    // Calculate the histogram distance.   float histogramDistance = (float)cv::compareHist(     detectedBlobDescriptor.getNormalizedHistogram(),     referenceBlobDescriptor.getNormalizedHistogram(),     HISTOGRAM_COMPARISON_METHOD);     // Calculate the keypoint matching distance.   float keypointMatchingDistance = 0.0f;   std::vector<cv::DMatch> keypointMatches;   descriptorMatcher->match(     detectedBlobDescriptor.getKeypointDescriptors(),     referenceBlobDescriptor.getKeypointDescriptors(),     keypointMatches);   for (const cv::DMatch &keypointMatch : keypointMatches) {     keypointMatchingDistance += keypointMatch.distance;   }     return histogramDistance * HISTOGRAM_DISTANCE_WEIGHT +     keypointMatchingDistance * KEYPOINT_MATCHING_DISTANCE_WEIGHT; } That is the end of the blob classifier's code. Again, we see that a single class can provide useful, general-purpose computer vision functionality without a terribly complicated implementation. Perhaps this is a Zen moment; our previous work and studieshave been a path to (some kind of) simplicity! Of course, OpenCV hides a lot of complexity for us in its implementations of histogram-related functions and keypoint-related classes, and in this way, the library offers us a relatively gentle path. For completeness, note that the BlobDescriptor class has a straightforward implementation. Create a new file, BlobDescriptor.cpp, and fill it with the following bodies for a constructor and getters: #include "BlobDescriptor.h"   BlobDescriptor::BlobDescriptor(const cv::Mat &normalizedHistogram, const cv::Mat &keypointDescriptors, uint32_t label) : normalizedHistogram(normalizedHistogram) , keypointDescriptors(keypointDescriptors) , label(label) { }   const cv::Mat &BlobDescriptor::getNormalizedHistogram() const {   return normalizedHistogram; } const cv::Mat &BlobDescriptor::getKeypointDescriptors() const {   return keypointDescriptors; } uint32_t BlobDescriptor::getLabel() const {   return label; } Summary Now, we have finished all the code for the detector, descriptor, and classifier! Again, for more information, refer to Chapter 5, Classifying Coins and Commodities in the book,iOS Application Development with OpenCV 3. Resources for Article: Further resources on this subject: Making subtle color shifts with curves [article] New functionality in OpenCV 3.0 [article] Camera Calibration [article]
Read more
  • 0
  • 0
  • 9100
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-languages-and-language-settings-moodle
Packt
22 Oct 2009
7 min read
Save for later

Languages and Language Settings in Moodle

Packt
22 Oct 2009
7 min read
Language The default Moodle installation includes many Language packs. A language pack is a set of translations for the Moodle interface. Language packs translate the Moodle interface, and not the course content. Here's the Front Page of a site when the user selects Spanish from the language menu: Note that every aspect of the interface is being presented in Spanish: menu names, menu items, section names, buttons, and system messages. Now, let's take a look at the same Front Page when the user selects Romanian from the language menu: Note that much of the interface has not been translated. For example, the Site Administration menu and section name for Site news are still in English. When a part of Moodle's interface is not translated into the selected language, Moodle uses the English version. Language Files When you install an additional language, Moodle places the language pack in its data directory under the subdirectory /lang. It creates a subdirectory for each language files. The following screenshot shows the results of installing the International Spanish and Romanian languages: For example, the subdirectory, /lang/en_us, holds files for the U.S. English translation, and /lang/es_es holds the files for traditional Spanish (Espanol / Espana). The name of the subdirectory is the 'language code'. Knowing this code will come in handy later. In the previous example, es_utf8 tells us that the language code for International Spanish is es. Inside a language pack's directory, we see a list of files that contain the translations: For example, the /lang/es_utf8/forum.php file holds text used on the forum pages. Let us suppose that we are creating a course for students. This file would include the text that is displayed to the course creator while creating the forum, and the text that is displayed to the students when they use the forum. Here are the first few lines from the English version of that file: $string['addanewdiscussion'] = 'Add a new discussion topic';$string['addanewtopic'] = 'Add a new topic';$string['advancedsearch'] = 'Advanced search'; And here are the same first three lines from the Spanish version of that file: $string['addanewdiscussion'] = 'Colocar un nuevo tema de discusión aquí';$string['addanewtopic'] = 'Agregar un nuevo tema';$string['advancedsearch'] = 'Búsqueda avanzada'; The biggest task in localizing Moodle consists of translating these language files into the appropriate languages. Some translations are surprisingly complete. For example, most of the interface has been translated to Irish Gaelic, even though this language is used by only about 350,000 people everyday. The Romanian interface remains mostly untranslated although Romania has a population of over 23 million. This means that if a Moodle user chooses the Romanian language (ro), most of the interface will still default to English. Language Settings You access the Language settings page from the Site Administration menu. Default Language and Display Language Menu The Default language setting specifies the language that users will see when they first encounter your site. If you also select Display language menu, users can change the language. Selecting this displays a language menu on your Front Page. Languages on Language Menu and Cache Language Menu The setting Languages on language menu enables you to specify the languages that users can pick from the language menu. There are directions for you to enter the 'language codes'. These codes are the names of the directories which hold the Language packs. In the subsection on Language Files on the previous page, you saw that the directory es_utf8 holds the language files for International Spanish. If you wanted to enter that language in the list, it would look like this: Leaving this field blank will enable your students to pick from all available languages. Entering the names of languages in this field limits the list to only those entered. Sitewide Locale Enter a language code into this field, and the system displays dates in the format appropriate to that language. Excel Encoding Most of the reports that Moodle generates can be downloaded as Excel files. User logs and grades are two examples. This setting lets you choose the encoding for those Excel files. Your choices are Unicode and Latin. The default is Unicode, because this character set includes many more characters other than Latin. In many cases, Latin encoding doesn't offer enough characters to completely represent a non-English language. Offering Courses in Multiple Languages The settings on the Language settings page are also applicable for translating the Moodle interface. However, they are not applicable for translating course content. If you want to offer course content in multiple languages, you have several choices. First, you could put all the different languages into each course. That is, each document would appear in a course in several languages. For example, if you offered a botany course in English and Spanish, you might have a document defining the different types of plants in both English and Spanish, side by side in the same course–Types of Plants or Tipos de Plantaras. While taking the course, students would select the documents in their language. Course names would appear in only one language. Second, you could create separate courses for each language, and offer them on the same site. Course names would appear in each language. In this case, students would select the course in English or Spanish– Basic Botany or Botánica Básica. Third, you could create a separate Moodle site for each language, for example, http://moodle.williamrice.com/english and http://moodle.williamrice.com/spanish. At the Home Page of your site, students would select their language and would be directed to the correct Moodle installation. In this case, the entire Moodle site would appear in the students' language: the site name, the menus, the course names, and the course content. These are things you should consider before installing Moodle. Installing Additional Languages To install additional languages, you must be connected to the Internet. Then, from the Site Administration menu, select Language | Language packs. The page displays a list of all available Language packs: This list is taken from Moodle's /install/lang directory. In that directory, you will find a folder for each language pack that can be installed. The folder contains a file called install.php. That file retrieves the most recent version of the language pack from the Web and installs it. This is why Moodle must be connected to the Web to use this feature. If Moodle is not connected, you will need to download the language pack and copy it into the /lang directory yourself. If you don't see the language you want on the list of Available language packs, either it's not available in the official Moodle site, or your list of available languages is out of date. Click to update this list. If the language doesn't appear, it's not available from official sources. Summary In this article, we have seen how Moodle website supports different languages and how to configure these different languages. This feature can be particularly useful while designing courses for students who come from different ethnic backgrounds. Language support can not only make the website more friendlier but also makes it more easy to browse.
Read more
  • 0
  • 0
  • 9098

Packt
26 Dec 2014
11 min read
Save for later

Getting Started with XenServer®

Packt
26 Dec 2014
11 min read
This article is written by Martez Reed, the author of Mastering Citrix® XenServer®. One of the most important technologies in the information technology field today is virtualization. Virtualization is beginning to span every area of IT, including but not limited to servers, desktops, applications, network, and more. Our primary focus is server virtualization, specifically with Citrix XenServer 6.2. There are three major platforms in the server virtualization market: VMware's vSphere, Microsoft's Hyper-V, and Citrix's XenServer. In this article, we will cover the following topics: XenServer's overview XenServer's features What's new in Citrix XenServer 6.2 Planning and installing Citrix XenServer (For more resources related to this topic, see here.) Citrix® XenServer® Citrix XenServer is a type 1 or bare metal hypervisor. A bare metal hypervisor does not require an underlying host operating system. Type 1 hypervisors have direct access to the underlying hardware, which provides improved performance and guest compatibility. Citrix XenServer is based on the open source Xen hypervisor that is widely deployed in various industries and has a proven record of stability and performance. Citrix® XenCenter® Citrix XenCenter is a Windows-based application that provides a graphical user interface for managing the Citrix XenServer hosts from a single management interface. Features of Citrix® XenServer® The following section covers the features offered by Citrix XenServer: XenMotion/Live VM Migration: The XenMotion feature allows for running virtual machines to be migrated from one host to another without any downtime. XenMotion relocates the processor and memory instances of the virtual machine from one host to another, while the actual data and settings reside on the shared storage. This feature is pivotal in providing maximum uptime when performing maintenance or upgrades. This feature requires shared storage among the hosts. Storage XenMotion / Live Storage Migration: The Storage XenMotion feature provides functionality similar to that of XenMotion, but it is used to move a virtual machine's virtual disk from one storage repository to another without powering off the virtual machine. High Availability: High Availability automatically restarts the virtual machines on another host in the event of a host failure. This feature requires shared storage among the hosts. Resource pools: Resource pools are a collection of Citrix XenServer hosts grouped together to form a single pool of compute, memory, network, and storage resources that can be managed as a single entity. The resource pool allows the virtual machines to be started on any of the hosts and seamlessly moved between them. Active Directory integration: Citrix XenServer can be joined to a Windows Active Directory domain to provide centralized authentication for XenServer administrators. This eliminates the need for multiple independent administrator accounts on each XenServer host in a XenServer environment. Role-based access control (RBAC): RBAC is a feature that takes advantage of the Active Directory integration and allows administrators to define roles that have specific privileges associated with them. This allows administrative permissions to be segregated among different administrators. Open vSwitch: The default network backend for the Citrix XenServer 6.2 hypervisor is Open vSwitch. Open vSwitch is an open source multilayer virtual switch that brings advanced network functionality to the XenServer platform such as NetFlow, SPAN, OpenFlow, and enhanced Quality of Service (QoS). The Open vSwitch backend is also an integral component of the platform's support of software-defined networking (SDN). Dynamic Memory Control: Dynamic Memory Control allows XenServer to maximize the physical memory utilization by sharing unused physical memory among the guest virtual machines. If a virtual machine has been allocated 4 GB of memory and is only using 2 GB, the remaining memory can be shared with the other guest virtual machines. This feature provides a mechanism for memory oversubscription. IntelliCache: IntelliCache is a feature aimed at improving the performance of Citrix XenDesktop virtual desktops. IntelliCache creates a cache on a XenServer local storage repository, and as the virtual desktops perform read operations, the parent VM's virtual disk is copied to the cache. Write operations are also written to the local cache when nonpersistent or shared desktops are used. This mechanism reduces the load on the storage array by retrieving data from a local source for reads instead of the array. This is particularly beneficial when multiple desktops share the same parent image. This feature is only available with Citrix XenDesktop. Disaster Recovery: The XenServer Disaster Recovery feature provides a mechanism to recover the virtual machines and vApps in the event of the failure of an entire pool or site. Distributed Virtual Switch Controller (DVSC): DVSC provides centralized management and visibility of the networking in XenServer. Thin provisioning: Thin provisioning allows for a given amount of disk space to be allocated to virtual machines but only consume the amount that is actually being used by the guest operating system. This feature provides more efficient use of the underlying storage due to the on-demand consumption. What's new in Citrix® XenServer® 6.2 Citrix has added a number of new and exciting features in the latest version of XenServer: Open source New licensing model Improved guest support Open source Starting with Version 6.2, the Citrix XenServer hypervisor is now open sourced, but continues to be managed by Citrix Systems. The move to an open source model was the result of Citrix Systems' desire to further collaborate and integrate the XenServer product with its partners and the open source community. New licensing model The licensing model has been changed in Version 6.2, with the free version of the XenServer platform now providing full functionality, the previous advanced, enterprise, and platinum versions have been eliminated. Citrix will offer paid support for the free version of the XenServer hypervisor that will include the ability to install patches/updates using the XenCenter GUI, in addition to Citrix technical support. Improved guest support Version 6.2 has added official support for the following guest operating systems: Microsoft Windows 8 (full support) Microsoft Windows Server 2012 SUSE Linux Enterprise Server (SLES) 11 SP2 (32/64 bit) Red Hat Enterprise Linux (RHEL) (32/64 bit) 5.8, 5.9, 6.3, and 6.4 Oracle Enterprise Linux (OEL) (32/64 bit) 5.8, 5.9, 6.3, and 6.4 CentOS (32/64 bit) 5.8, 5.9, 6.3, and 6.4 Debian Wheezy (32/64 bit) VSS support for Windows Server 2008 R2 has been improved and reintroduced Citrix XenServer 6.2 Service Pack 1 adds support for the following operating systems: Microsoft Windows 8.1 Microsoft Windows Server 2012 R2 Retired features The following features have been removed from Version 6.2 of Citrix XenServer: Workload Balancing (WLB) SCOM integration Virtual Machine Protection Recovery (VMPR) Web Self Service XenConvert (this has been replaced by XenServer Conversion Manager) Deprecated features The following features will be removed from the future releases of Citrix XenServer. Citrix has reviewed the XenServer market and has determined that there are third-party products that are able to provide the product functionality more effectively: Microsoft System Center Virtual Machine Manager SCVMM support Integrated StorageLink Planning and Installing Citrix® XenServer® Installing Citrix XenServer is generally a simple and straightforward process that can be completed in 10 to 15 minutes. While the actual install is simple, there are several major decisions that need to be made prior to installing Citrix XenServer in order to ensure a successful deployment. Selecting the server hardware Typically, the first step is to select the server hardware that will be used. While the thought might be to just pick a server that fits our needs, we should also ensure that the hardware meets the documented system requirements. Checking the hardware against the Hardware Compatibility List (HCL) provided by Citrix Systems is advised to ensure that the system qualifies for Citrix support and that the system will properly run Citrix XenServer. The HCL provides a list of server models that have been verified to work with Citrix XenServer. The HCL can be found online at http://www.citrix.com/xenserver/hcl. Meeting the system requirements The following sections cover the minimal system requirements for Citrix XenServer 6.2. Processor requirements The following list covers the minimum requirements for the processor(s) to install Citrix XenServer 6.2: One or more 64-bit x86 CPU(s), 1.5 GHz minimum, 2 GHz or faster multicore CPU To support VMs running on Windows, an Intel VT or AMD-V 64-bit x86-based system with one or more CPU(s) is required Virtualization technology needs to be enabled in the BIOS Virtualization technology is disabled by default on many server platforms and needs to be manually enabled. Memory requirements The minimum memory requirement for installing Citrix XenServer 6.2 is 2 GB with a recommendation of 4 GB or more for production workloads. In addition to the memory usage of the guest virtual machines, the Xen hypervisor on the Control Domain (dom0) consumes the memory resources. The amount of resources consumed by the Control Domain (dom0) is based on the amount of physical memory in the host. Hard disk requirements The following are the minimum requirements for the hard disk(s) to install Citrix XenServer 6.2: 16 GB of free disk space minimum and 60 GB of free disk space is recommended Direct attached storage in the form of SATA, SAS, SCSI, or PATA interfaces are supported XenServer can be installed on a LUN presented from a storage area network (SAN) via a host bus adapter (HBA) in the XenServer host A physical HBA is required to boot XenServer from a SAN. Network card requirements 100 Mbps or a faster NIC is required for installing Citrix XenServer. One or more gigabit NICs is recommended for faster P2V, export/import data transfers, and VM live migrations. Installing Citrix® XenServer® 6.2 The following sections cover the installation of Citrix XenServer 6.2. Installation methods The Citrix XenServer 6.2 installer can be launched via two methods as listed: CD/DVD PXE or network boot Installation source There are several options where the Citrix XenServer installation files can be stored, and depending on the scenario, one would be preferred over another. Typically, the HTTP, FTP, or NFS option would be used when the installer is booted over the network via PXE or when a scripted installation is being performed. The installation sources are as follows: Local media (CD/DVD) HTTP or FTP NFS Supplemental packs Supplemental packs provide additional functionality to the XenServer platform through features such as enhanced hardware monitoring and third-party management software integration. The supplemental packs are typically downloaded from the vendor's website and are installed when prompted during the XenServer installation. XenServer® installation The following steps cover installing Citrix XenServer 6.2 from a CD: Boot the server from the Citrix XenServer 6.2 installation media and press Enter when prompted to start the Citrix XenServer 6.2 installer. Select the desired key mapping and select Ok to proceed. Press F9 if additional drivers need to be installed or select Ok to continue. Accept the EULA. Select the hard drive for the Citrix XenServer installation and choose Ok to proceed. Select the hard drive(s) to be used for storing the guest virtual machines and choose Ok to continue. You need to select the Enable thin provisioning (Optimized storage for XenDesktop) option to make use of the IntelliCache feature. Select the installation media source and select Ok to continue. Install supplemental packs if necessary and choose No to proceed. Select Verify installation source and select Ok to begin the verification. The installation media should be verified at least once to ensure that none of the installation files are corrupt. Choose Ok to continue after the verification has successfully completed. Provide and confirm a password for the root account and select Ok to proceed. Select the network interface to be used as the primary management interface and choose Ok to continue. Select the Static configuration option and provide the requested information. Choose Ok to continue. Enter the desired hostname and DNS server information. Select Ok to proceed. Select the appropriate geographical area to configure the time zone and select Ok to continue. Select the appropriate city or area to configure the time zone and select Ok to proceed. Select Using NTP or Manual time entry for the server to determine the local time and choose Ok to continue. Using NTP to synchronize the time of XenServer hosts in a pool is recommended to ensure that the time on all the hosts in the pool is synchronized. Enter the IP address or hostname of the desired NTP server(s) and select Ok to proceed. Select Install XenServer to start the installation. Click on Ok to restart the server after the installation has completed. The following screen should be presented after the reboot: Summary In this article, we covered an overview of Citrix XenServer along with the features that were available. We also looked at the new features that were added in XenServer 6.2 and then examined installing XenServer. Resources for Article: Further resources on this subject: Understanding Citrix®Provisioning Services 7.0 [article] Designing a XenDesktop® Site [article] Installation and Deployment of Citrix Systems®' CPSM [article]
Read more
  • 0
  • 0
  • 9093

article-image-parallax-scrolling
Packt
13 Mar 2014
16 min read
Save for later

Parallax scrolling

Packt
13 Mar 2014
16 min read
(For more resources related to this topic, see here.) As a developer, we have been asked numerous times about how to implement parallax scrolling in a 2D game. Cerulean Games, my game studio, has even had the elements of parallax scrolling as the "do or die" requirement to close a project deal with a client. In reality, this is incredibly easy to accomplish, and there are a number of ways to do this. In Power Rangers Samurai SMASH! (developed by Cerulean Games for Curious Brain; you can find it in the iOS App Store), we implemented a simple check that would see what the linear velocity of the player is and then move the background objects in the opposite direction. The sprites were layered on the Z plane, and each was given a speed multiplier based on its distance from the camera. So, as the player moved to the right, all parallax scrolling objects would move to the left based on their multiplier value. That technique worked, and it fared us and our client quite well for the production of the game. This is also a common and quick way to manage parallax scrolling, and it's also pretty much how we're going to manage it in this game as well. OK, enough talk! Have at you! Well, have at the code: Create a new script called ParallaxController and make it look like the following code: -using UnityEngine; using System.Collections; public class ParallaxController : MonoBehaviour {     public GameObject[] clouds;     public GameObject[] nearHills;     public GameObject[] farHills;     public float cloudLayerSpeedModifier;     public float nearHillLayerSpeedModifier;     public float farHillLayerSpeedModifier;     public Camera myCamera;     private Vector3 lastCamPos;     void Start()     {         lastCamPos = myCamera.transform.position;     }     void Update()     {         Vector3 currCamPos = myCamera.transform.position;         float xPosDiff = lastCamPos.x - currCamPos.x;         adjustParallaxPositionsForArray(clouds,           cloudLayerSpeedModifier, xPosDiff);         adjustParallaxPositionsForArray(nearHills,           nearHillLayerSpeedModifier, xPosDiff);         adjustParallaxPositionsForArray(farHills,           farHillLayerSpeedModifier, xPosDiff);         lastCamPos = myCamera.transform.position;     }     void adjustParallaxPositionsForArray(GameObject[]       layerArray, float layerSpeedModifier, float xPosDiff)     {         for(int i = 0; i < layerArray.Length; i++)         {             Vector3 objPos =               layerArray[i].transform.position;             objPos.x += xPosDiff * layerSpeedModifier;             layerArray[i].transform.position = objPos;         }     } } Create a new GameObject in your scene and call it _ParallaxLayers. This will act as the container for all the parallax layers. Create three more GameObjects and call them _CloudLayer, _NearHillsLayer, and _FarHillsLayer, respectively. Place these three objects inside the _ParallaxLayers object, and place the Parallax Controller component onto the _ParallaxLayers object. Done? Good. Now we can move some sprites. Import the sprites from SpritesParallaxScenery. Start placing sprites in the three layer containers you created earlier. For the hills you want to be closer, place the sprites in the _NearHillsLayer container; for the hills you want to be further away, place the sprites in _FarHillsLayer; and place the clouds in the _CloudLayer container. The following screenshot shows an example of what the layers will now look like in the scene: Pro tip Is this the absolute, most efficient way of doing parallax? Somewhat; however, it's a bit hardcoded to only really fit the needs of this game. Challenge yourself to extend it to be flexible and work for any scenario! Parallax layer ordering Wait, you say that the objects are layered in the wrong order? Your hills are all mixed up with your platforms and your platforms are all mixed up with your hills? OK, don't panic, we've got this. What you need to do here is change the Order in Layer option for each of the parallax sprites. You can find this property in the Sprite Renderer component. Click on one of the sprites in your scene, such as one of the clouds, and you can see it in the Inspector panel. Here's a screenshot to show you where to look: Rather than changing each sprite individually, we can easily adjust the sprites in bulk by performing the following steps: Select all of your cloud layer sprites, and under their Sprite Renderer components, set their Order in Layer to 0. Set the Order in Layer property of the _NearHillsLayer sprites to 1 and that of the _FarHillsLayer sprites to 0. Select the Prefab named Platform and set its Order in Layer to 2; you should see all of your Platform sprites instantly update in the scene. Set the Order in Layer values of the Prefabs for Enemy and Player Bullet to 2. Set the sprite on the Player object in the scene to 2 as well. Finally, set the Wall objects to 3 and you're good to go. With the layers all set up, let's finish setting up the parallax layers. First, finish placing any additional parallax sprites; I'll wait. Brilliant! Now, go to the _ParallaxLayers object and let's play around with that Parallax Controller component. We're going to want to add all of those sprites to Parallax Controller. To make this easy, look at the top-right corner of the Inspector panel. See the little lock icon? Click on it. Now, regardless of what you do, the Parallax Controller component will not be deselected. Since it can't be deselected, you can now easily drag-and-drop all of the Cloud sprites into the Clouds array in the ParallaxController component, and all of the _FarHillsLayer child objects into the Far Hills array—you see where this is going. Set the My Camera field to use the Main Camera object. Finally, let's set some values in those Layer Speed Modifier fields. The higher the number, the faster the object will move as the camera moves. As an example, we set the Cloud layer to 0.05, the Near layer to 0.2, and the Far layer to 0.1. Feel free though to play with the values and see what you like! Go ahead and play the game. Click on the play button and watch those layers move! But, what's this? The particles that burst when an enemy is defeated render behind the sprites in the background—actually, they render behind all the sprites! To fix this, we need to tell Unity to render the particles on a layer in front of the sprites. By default, the sprites render after the particles. Let's change that. First, we need to create a new sorting layer. These are special types of layers that tell Unity the order to render things in. Go to the Tags & Layers window and look out for the drop-down menu called Sorting Layers. Add a new layer called ParticleLayer on Layer 1, as shown in the following screenshot: With this in place, it means anything with the Sorting Layers menu of ParticleLayer will render after the Default layer. Now, we need a way to assign this Sorting Layer to the particle system used when enemies are defeated. Create a new script called ParticleLayering and make it look like the following code: using UnityEngine; using System.Collections; public class ParticleLayering : MonoBehaviour {     public string sortLayerString = "";     void Start ()     {         particleSystem.renderer.sortingLayerName = sortLayerString;     } } Add this script to the EnemyDeathFX Prefab and set the Sort Layer String field to ParticleLayer. Go ahead and play the game again now to watch those particles fly in front of the other objects. Finally, if you want a solid color background to your scene, you don't need to worry about adding a colored plane or anything. Simply select the Main Camera object, and in the Inspector panel, look for the Background field in the Camera component. Adjust the color there as per your need. For the example game, we made this color a nice sky blue with the following values: R: 128, G: 197, B: 232, and A: 0. The one thing you may notice we're missing is something at the bottom of the scene. Here's a nice little challenge for you. We've given you a Lava sprite. Now, add in a lava layer of parallax sprites in the foreground using all the info you've read in this article. You can do this! Let's score! One of the most important elements to a game is being able to track progress. A quick and simple way to do this is to implement a score system. In our case, we will have a score that increases whenever you defeat an enemy. Now, Unity does have a built-in GUI system. However, it has some drawbacks. With this in mind, we won't be relying on Unity's built-in system. Instead, we are going to create objects and attach them to the camera, which in turn will allow us to have a 3D GUI. Pro tip If you want to use what this author believes is the best UI system for Unity, purchase a license for NGUI from the Unity Asset Store. I'm not the only one to think it's the best; Unity hired the NGUI developer to build the new official UI system for Unity itself. Let's build out some GUI elements: Create a 3D Text object by navigating to the menu item GameObject | Create Other; name it Score. Make it a child of the Main Camera GameObject and align it such that it sits in the top-left corner of the screen. Set its position to X: -6.91, Y: 4.99, Z: 10 to get this effect. Make the text color solid black and adjust the scaling so that it looks the way you want it to. Set the Anchor field to Upper Left and Alignment to Left. Adjust the scene to your taste, but it should look a little something like the following screenshot: Pro tip Unity's default 3D text font looks rather low quality in most situations. Try importing your own font and then set the font size to something much higher than you would usually need; often around 25 to 40. Then, when you place it in the world, it will look crisp and clean. Let's make it so that the Score visual element can actually track the player's score. Create a new script called ScoreWatcher and write the following code in it: using UnityEngine; using System.Collections; public class ScoreWatcher : MonoBehaviour {     public int currScore = 0;     private TextMesh scoreMesh = null;         void Start()     {         scoreMesh = gameObject.GetComponent<TextMesh>(); scoreMesh.text = "0";     }         void OnEnable()     {         EnemyControllerScript.enemyDied += addScore;     }         void OnDisable()     {         EnemyControllerScript.enemyDied -= addScore;     }         void addScore(int scoreToAdd)     {         currScore += scoreToAdd;         scoreMesh.text = currScore.ToString();     } } You may notice that in the preceding script, we are listening to the enemyDied event on the EnemyControllerScript. What we did here was we allowed other objects to easily create scoring events that the Score object can optionally listen to. There is lots of power to this! Let's add that event and delegate to the enemy. Open up EnemyControllerScript, and in the beginning, add the following code:     // States to allow objects to know when an enemy dies     public delegate void enemyEventHandler(int scoreMod);         public static event enemyEventHandler enemyDied; Then, down in the hitByPlayerBullet function, add the following code just above Destroy(gameObject,0.1f);, right around line 95: // Call the EnemyDied event and give it a score of 25. if(enemyDied != null)     enemyDied(25); Add the ScoreWatcher component to the Score object. Now, when you play the game and defeat the enemies, you can watch the score increase by 25 points each time! Yeeee-haw! Sorry 'bout that... shootin' things for points always makes me feel a bit Texan. Enemies – forever! So, you defeated all your enemies and now find yourself without enemies to defeat. This gets boring fast; so, let's find a way to get more enemies to whack. To do this, we are going to create enemy spawn points in the form of nifty rotating vortexes and have them spit out enemies whenever we kill other enemies. It shall be glorious, and we'll never be without friends to give gifts—and by gifts, we mean bullets. First things first. We need to make a cool-looking vortex. This vortex will be a stacked, animated visual FX object that is built for a 2D world. Don't worry, we've got you covered on textures, so please go through the following steps: Import the ones in the assets folder under SpritesVortex. Create a new GameObject called Vortex and add all three of the Vortex sprites in it, with each of their Position values set to X:0, Y:0, and Z:0. Adjust their Order in Layer values so that the Vortex_Back child is set to 10, Vortex_Center is set to 11, and Vortex_Front is set to 12. You should now have an object that looks something like the following screenshot: Go ahead and give it a nice spinning animation by rotating the Z axis from 0 to 356. Once you're happy with it, create a new script called EnemyRespawner and code it up as shown in the following code snippet: using UnityEngine; using System.Collections; public class EnemyRespawner : MonoBehaviour {     public GameObject spawnEnemy = null;     float respawnTime = 0.0f;         void OnEnable()     {         EnemyControllerScript.enemyDied += scheduleRespawn;     }         void OnDisable()     {         EnemyControllerScript.enemyDied -= scheduleRespawn;     }         // Note: Even though we don't need the enemyScore, we still need to accept it because the event passes it     void scheduleRespawn(int enemyScore)     {         // Randomly decide if we will respawn or not         if(Random.Range(0,10) < 5)             return;                 respawnTime = Time.time + 4.0f;     }         void Update()     {         if(respawnTime > 0.0f)         {             if(respawnTime < Time.time)             {                 respawnTime = 0.0f;                 GameObject newEnemy = Instantiate(spawnEnemy) as GameObject;                 newEnemy.transform.position = transform.position;             }         }     } } Now attach the preceding script to your Vortex object, populate the Spawn Enemy field with the Enemy Prefab, and save the Vortex object as a Prefab. Scatter a bunch of Vortex Prefabs around the level and you can get the hydra effect, where killing one enemy will create two more enemies or even more than two! Also, if you haven't already done so, you may want to go to the Physics Manager option and adjust the settings so that enemies won't collide with other enemies. One more thing—those enemies sort of glide out of their portals very awkwardly. Let's boost the gravity so they fall faster. Click on the main Enemy Prefab and change the Gravity Scale value of the RigidBody 2D component to 30. Now, they'll fall properly! Pro tip There are so many things you can do with enemy spawners that go far, far outside the context of this article. Take a shot at adding some features yourself! Here are a few ideas: Make the spawn vortexes play a special visual effect when an enemy is spawned Give vortexes a range so that they only spawn an enemy if another enemy was killed in their range Make vortexes move around the level Make vortexes have multiple purposes so that enemies can walk into one and come out another Have a special gold enemy worth bonus points spawn after every 100 kills Make an enemy that, when defeated, spawns other enemies or even collectable objects that earn the player bonus points! Summary So, what have we learned here today aside from the fact that shooting enemies with bullets earns you points? Well, check this out. You now know how to use parallax scrolling, 2D layers, and generate objects; and how to use a scoring system. Enemies dying, enemies spawning, freakin' vortexes? I know, you're sitting there going, "Dude, OK, I'm ready to get started on my first 2D game... the next side scrolling MMO Halo meets Candy Crush with bits of Mass Effect and a little Super Mario Bros!" Resources for Article: Further resources on this subject: Unity Game Development: Interactions (Part 1) [Article] Unity Game Development: Interactions (Part 2) [Article] Unity Game Development: Welcome to the 3D world [Article]
Read more
  • 0
  • 0
  • 9088

article-image-using-model-serializers-eliminate-duplicate-code
Packt
23 Sep 2016
12 min read
Save for later

Using model serializers to eliminate duplicate code

Packt
23 Sep 2016
12 min read
In this article by Gastón C. Hillar, author of, Building RESTful Python Web Services, we will cover the use of model serializers to eliminate duplicate code and use of default parsing and rendering options. (For more resources related to this topic, see here.) Using model serializers to eliminate duplicate code The GameSerializer class declares many attributes with the same names that we used in the Game model and repeats information such as the types and the max_length values. The GameSerializer class is a subclass of the rest_framework.serializers.Serializer, it declares attributes that we manually mapped to the appropriate types, and overrides the create and update methods. Now, we will create a new version of the GameSerializer class that will inherit from the rest_framework.serializers.ModelSerializer class. The ModelSerializer class automatically populates both a set of default fields and a set of default validators. In addition, the class provides default implementations for the create and update methods. In case you have any experience with Django Web Framework, you will notice that the Serializer and ModelSerializer classes are similar to the Form and ModelForm classes. Now, go to the gamesapi/games folder folder and open the serializers.py file. Replace the code in this file with the following code that declares the new version of the GameSerializer class. The code file for the sample is included in the restful_python_chapter_02_01 folder. from rest_framework import serializers from games.models import Game class GameSerializer(serializers.ModelSerializer): class Meta: model = Game fields = ('id', 'name', 'release_date', 'game_category', 'played') The new GameSerializer class declares a Meta inner class that declares two attributes: model and fields. The model attribute specifies the model related to the serializer, that is, the Game class. The fields attribute specifies a tuple of string whose values indicate the field names that we want to include in the serialization from the related model. There is no need to override either the create or update methods because the generic behavior will be enough in this case. The ModelSerializer superclass provides implementations for both methods. We have reduced boilerplate code that we didn’t require in the GameSerializer class. We just needed to specify the desired set of fields in a tuple. Now, the types related to the game fields is included only in the Game class. Press Ctrl + C to quit Django’s development server and execute the following command to start it again. python manage.py runserver Using the default parsing and rendering options and move beyond JSON The APIView class specifies default settings for each view that we can override by specifying appropriate values in the gamesapi/settings.py file or by overriding the class attributes in subclasses. As previously explained the usage of the APIView class under the hoods makes the decorator apply these default settings. Thus, whenever we use the decorator, the default parser classes and the default renderer classes will be associated with the function views. By default, the value for the DEFAULT_PARSER_CLASSES is the following tuple of classes: ( 'rest_framework.parsers.JSONParser', 'rest_framework.parsers.FormParser', 'rest_framework.parsers.MultiPartParser' ) When we use the decorator, the API will be able to handle any of the following content types through the appropriate parsers when accessing the request.data attribute. application/json application/x-www-form-urlencoded multipart/form-data When we access the request.data attribute in the functions, Django REST Framework examines the value for the Content-Type header in the incoming request and determines the appropriate parser to parse the request content. If we use the previously explained default values, Django REST Framework will be able to parse the previously listed content types. However, it is extremely important that the request specifies the appropriate value in the Content-Type header. We have to remove the usage of the rest_framework.parsers.JSONParser class in the functions to make it possible to be able to work with all the configured parsers and stop working with a parser that only works with JSON. The game_list function executes the following two lines when request.method is equal to 'POST': game_data = JSONParser().parse(request) game_serializer = GameSerializer(data=game_data) We will remove the first line that uses the JSONParser and we will pass request.data as the data argument for the GameSerializer. The following line will replace the previous lines: game_serializer = GameSerializer(data=request.data) The game_detail function executes the following two lines when request.method is equal to 'PUT': game_data = JSONParser().parse(request) game_serializer = GameSerializer(game, data=game_data) We will make the same edits done for the code in the game_list function. We will remove the first line that uses the JSONParser and we will pass request.data as the data argument for the GameSerializer. The following line will replace the previous lines: game_serializer = GameSerializer(game, data=request.data) By default, the value for the DEFAULT_RENDERER_CLASSES is the following tuple of classes: ( 'rest_framework.renderers.JSONRenderer', 'rest_framework.renderers.BrowsableAPIRenderer', ) When we use the decorator, the API will be able to render any of the following content types in the response through the appropriate renderers when working with the rest_framework.response.Response object. application/json text/html By default, the value for the DEFAULT_CONTENT_NEGOTIATION_CLASS is the rest_framework.negotiation.DefaultContentNegotiation class. When we use the decorator, the API will use this content negotiation class to select the appropriate renderer for the response based on the incoming request. This way, when a request specifies that it will accept text/html, the content negotiation class selects the rest_framework.renderers.BrowsableAPIRenderer to render the response and generate text/html instead of application/json. We have to replace the usages of both the JSONResponse and HttpResponse classes in the functions with the rest_framework.response.Response class. The Response class uses the previously explained content negotiation features, renders the received data into the appropriate content type and returns it to the client. Now, go to the gamesapi/games folder folder and open the views.py file. Replace the code in this file with the following code that removes the JSONResponse class, uses the @api_view decorator for the functions and the rest_framework.response.Response class. The modified lines are highlighted. The code file for the sample is included in the restful_python_chapter_02_02 folder. from rest_framework.parsers import JSONParser from rest_framework import status from rest_framework.decorators import api_view from rest_framework.response import Response from games.models import Game from games.serializers import GameSerializer @api_view(['GET', 'POST']) def game_list(request): if request.method == 'GET': games = Game.objects.all() games_serializer = GameSerializer(games, many=True) return Response(games_serializer.data) elif request.method == 'POST': game_serializer = GameSerializer(data=request.data) if game_serializer.is_valid(): game_serializer.save() return Response(game_serializer.data, status=status.HTTP_201_CREATED) return Response(game_serializer.errors, status=status.HTTP_400_BAD_REQUEST) @api_view(['GET', 'PUT', 'POST']) def game_detail(request, pk): try: game = Game.objects.get(pk=pk) except Game.DoesNotExist: return Response(status=status.HTTP_404_NOT_FOUND) if request.method == 'GET': game_serializer = GameSerializer(game) return Response(game_serializer.data) elif request.method == 'PUT': game_serializer = GameSerializer(game, data=request.data) if game_serializer.is_valid(): game_serializer.save() return Response(game_serializer.data) return Response(game_serializer.errors, status=status.HTTP_400_BAD_REQUEST) elif request.method == 'DELETE': game.delete() return Response(status=status.HTTP_204_NO_CONTENT) After you save the previous changes, run the following command: http OPTIONS :8000/games/ The following is the equivalent curl command: curl -iX OPTIONS :8000/games/ The previous command will compose and send the following HTTP request: OPTIONS http://localhost:8000/games/. The request will match and run the views.game_list function, that is, the game_list function declared within the games/views.py file. We added the @api_view decorator to this function, and therefore, it is capable of determining the supported HTTP verbs, parsing and rendering capabilities. The following lines show the output: HTTP/1.0 200 OK Allow: GET, POST, OPTIONS, PUT Content-Type: application/json Date: Thu, 09 Jun 2016 21:35:58 GMT Server: WSGIServer/0.2 CPython/3.5.1 Vary: Accept, Cookie X-Frame-Options: SAMEORIGIN { "description": "", "name": "Game Detail", "parses": [ "application/json", "application/x-www-form-urlencoded", "multipart/form-data" ], "renders": [ "application/json", "text/html" ] } The response header includes an Allow key with a comma-separated list of HTTP verbs supported by the resource collection as its value: GET, POST, OPTIONS. As our request didn’t specify the allowed content type, the function rendered the response with the default application/json content type. The response body specifies the Content-type that the resource collection parses and the Content-type that it renders. Run the following command to compose and send and HTTP request with the OPTIONS verb for a game resource. Don’t forget to replace 3 with a primary key value of an existing game in your configuration: http OPTIONS :8000/games/3/ The following is the equivalent curl command: curl -iX OPTIONS :8000/games/3/ The previous command will compose and send the following HTTP request: OPTIONS http://localhost:8000/games/3/. The request will match and run the views.game_detail function, that is, the game_detail function declared within the games/views.py file. We also added the @api_view decorator to this function, and therefore, it is capable of determining the supported HTTP verbs, parsing and rendering capabilities. The following lines show the output: HTTP/1.0 200 OK Allow: GET, POST, OPTIONS Content-Type: application/json Date: Thu, 09 Jun 2016 20:24:31 GMT Server: WSGIServer/0.2 CPython/3.5.1 Vary: Accept, Cookie X-Frame-Options: SAMEORIGIN { "description": "", "name": "Game List", "parses": [ "application/json", "application/x-www-form-urlencoded", "multipart/form-data" ], "renders": [ "application/json", "text/html" ] } The response header includes an Allow key with comma-separated list of HTTP verbs supported by the resource as its value: GET, POST, OPTIONS, PUT. The response body specifies the content-type that the resource parses and the content-type that it renders, with the same contents received in the previous OPTIONS request applied to a resource collection, that is, to a games collection. When we composed and sent POST and PUT commands, we had to use the use the -H "Content-Type: application/json" option to indicate curl to send the data specified after the -d option as application/json instead of the default application/x-www-form-urlencoded. Now, in addition to application/json, our API is capable of parsing application/x-www-form-urlencoded and multipart/form-data data specified in the POST and PUT requests. Thus, we can compose and send a POST command that sends the data as application/x-www-form-urlencoded with the changes made to our API. We will compose and send an HTTP request to create a new game. In this case, we will use the -f option for HTTPie that serializes data items from the command line as form fields and sets the Content-Type header key to the application/x-www-form-urlencoded value. http -f POST :8000/games/ name='Toy Story 4' game_category='3D RPG' played=false release_date='2016-05-18T03:02:00.776594Z' The following is the equivalent curl command. Notice that we don’t use the -H option and curl will send the data in the default application/x-www-form-urlencoded: curl -iX POST -d '{"name":"Toy Story 4", "game_category":"3D RPG", "played": "false", "release_date": "2016-05-18T03:02:00.776594Z"}' :8000/games/ The previous commands will compose and send the following HTTP request: POST http://localhost:8000/games/ with the Content-Type header key set to the application/x-www-form-urlencoded value and the following data: name=Toy+Story+4&game_category=3D+RPG&played=false&release_date=2016-05-18T03%3A02%3A00.776594Z The request specifies /games/, and therefore, it will match '^games/$' and run the views.game_list function, that is, the updated game_detail function declared within the games/views.py file. As the HTTP verb for the request is POST, the request.method property is equal to 'POST', and therefore, the function will execute the code that creates a GameSerializer instance and passes request.data as the data argument for its creation. The rest_framework.parsers.FormParser class will parse the data received in the request, the code creates a new Game and, if the data is valid, it saves the new Game. If the new Game was successfully persisted in the database, the function returns an HTTP 201 Created status code and the recently persisted Game serialized to JSON in the response body. The following lines show an example response for the HTTP request, with the new Game object in the JSON response: HTTP/1.0 201 Created Allow: OPTIONS, POST, GET Content-Type: application/json Date: Fri, 10 Jun 2016 20:38:40 GMT Server: WSGIServer/0.2 CPython/3.5.1 Vary: Accept, Cookie X-Frame-Options: SAMEORIGIN { "game_category": "3D RPG", "id": 20, "name": "Toy Story 4", "played": false, "release_date": "2016-05-18T03:02:00.776594Z" } After the changes we made in the code, we can run the following command to see what happens when we compose and send an HTTP request with an HTTP verb that is not supported: http PUT :8000/games/ The following is the equivalent curl command: curl -iX PUT :8000/games/ The previous command will compose and send the following HTTP request: PUT http://localhost:8000/games/. The request will match and try to run the views.game_list function, that is, the game_list function declared within the games/views.py file. The @api_view decorator we added to this function doesn’t include 'PUT' in the string list with the allowed HTTP verbs, and therefore, the default behavior returns a 405 Method Not Allowed status code. The following lines show the output with the response from the previous request. A JSON content provides a detail key with a string value that indicates the PUT method is not allowed. HTTP/1.0 405 Method Not Allowed Allow: GET, OPTIONS, POST Content-Type: application/json Date: Sat, 11 Jun 2016 00:49:30 GMT Server: WSGIServer/0.2 CPython/3.5.1 Vary: Accept, Cookie X-Frame-Options: SAMEORIGIN { "detail": "Method "PUT" not allowed." } Summary This article covers the use of model serializers and how it is effective in removing duplicate code. Resources for Article: Further resources on this subject: Making History with Event Sourcing [article] Implementing a WCF Service in the Real World [article] WCF – Windows Communication Foundation [article]
Read more
  • 0
  • 0
  • 9087
article-image-how-large-language-models-reshape-trading-stats
Anshul Saxena
21 Sep 2023
15 min read
Save for later

How Large Language Models Reshape Trading Stats

Anshul Saxena
21 Sep 2023
15 min read
Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!IntroductionStock analysis is not just about numbers; it's a sophisticated dance of interpretation and prediction. Advanced techniques, such as the ones discussed here, offer deeper insights into the world of stocks. The journey begins with Volatility Analysis, utilizing Rolling Standard Deviation to grasp the extent of stock price movements, offering a window into the stock's inherent risk. Predictive Modeling then takes the stage, harnessing past data to provide a lens into potential future stock prices. Yet, for any analysis to stand on solid ground, the data itself must be pristine. This is where Data Cleaning comes into play, meticulously weeding out inconsistencies and errors, and laying a trustworthy foundation. To make sense of this vast sea of data, Visualizations transform intricate datasets into digestible charts and graphs, bridging the gap between numbers and narratives. Now, with these advanced steps in view, it's time to Dive In. This guide is designed for those keen to delve into a more profound layer of stock analysis, be it beginners eager to learn or veterans aiming to hone their skills.Step 1. Volatility Analysis gauges stock price fluctuations using Rolling Standard Deviation.Step 2. Data Cleaning: Ensure the data is free of inconsistencies and errors before diving into analysis.Step 3. Visualizations: Create intuitive charts and graphs to represent data, making it easily comprehensible at a glance.Step 4. Heat Map: Utilize heatmaps to visualize monthly average closing prices, helping traders spot recurring patterns, seasonality, and stock price shifts for informed investment decisions.Step 5. Predictive Modeling: Leverage historical data to attempt predictions of future stock prices.You can access the data here (Link)Step 1: Volatility Analysis - Rolling Standard DeviationVolatility Analysis employs the Rolling Standard Deviation to measure the consistency of stock price fluctuations over time.Prompt 1: As a diligent stock analyst, recognizing and quantifying a stock's volatility is paramount in understanding its risk and potential reward. To delve into this aspect, let's employ the rolling standard deviation methodology on the daily returns. Using a 21-day window - a conventional choice for monthly volatility assessments - will yield a dynamic depiction of the stock's volatility. The subsequent plot showcasing the 21-day rolling volatility essentially captures the standard deviation of the stock's daily returns over this time frame. Such a visualization can aid investors in discerning periods of heightened uncertainty or stability, assisting in refined decision-making.Response: Absolutely, the rolling standard deviation of daily returns is a commonly used metric to assess the volatility of a stock. A higher standard deviation signifies greater volatility, while a lower value indicates stability.To assess the stock's volatility:1. Compute the rolling standard deviation of the daily returns using a 21-day window.2. Plot the resulting 21-day rolling volatility.Let's proceed with these steps.The graph showcases the 21-day rolling volatility of the stock, which is essentially the standard deviation of the stock's daily returns over a 21-day window.# Compute the 21-day rolling standard deviation of the daily returns data['21-day Volatility'] = data['Daily Returns'].rolling(window=21).std() # Plotting the 21-day rolling volatility plt.figure(figsize=(15, 7)) plt.plot(data.index, data['21-day Volatility'], label='21-day Volatility', color='purple') plt.title("21-day Rolling Volatility") plt.xlabel("Date") plt.ylabel("Volatility (Standard Deviation of Daily Returns)") plt.legend() plt.grid(True) plt.tight_layout() plt.show() Interpretation: The chart shows the stock's ups and downs over time using a purple line. When this line peaks, it means the stock had a lot of movement, possibly because of big company news or changes in the wider market. On the other hand, when the line is low, the stock was more stable. While some investors might see these peaks as chances to make quick gains, others might prefer the steadier, low points. Simply put, this chart helps investors decide when to buy or sell based on how much the stock is expected to move.After analyzing the volatility, let’s clean the data  for further analysisStep 2: Data Cleaning and TransformationData Cleaning and Transformation involves addressing missing values, normalizing or standardizing data for modeling purposes, and ensuring date uniformity for a robust time series analysis by setting it as the index.Prompt 2: As a meticulous stock analyst, it's crucial to ensure the integrity and completeness of our dataset before deep-diving into further analysis. Start by scanning the dataset for any missing values, and addressing them appropriately, either through imputation or deletion based on the dataset's nature and the missing data's extent. Moreover, consistency is key; hence, confirm that the "Date" column adheres to a uniform format. This foundational data preparation step ensures the reliability of our subsequent analyses and insights.Response: Absolutely, ensuring the data's integrity and completeness is a foundational step in any analysis. To maintain the reliability of our subsequent analyses, we'll:1. Scan the dataset for missing values.2. Address these missing values, deciding on either imputation or deletion based on the nature and extent of the missing data.3. Confirm that the "Date" column is in a consistent format.Let's start by checking for missing values in our dataset.# Check for missing values in the dataset missing_values = data.isnull().sum() missing_valuesThe dataset has missing values in the following columns:1. Daily Returns: 1 missing value - This is expected since the first entry won't have a previous day to compute the return from.2. 50-day MA: 49 missing values - The 50-day moving average will have missing values for the initial 49 days since there aren't enough data points to compute the average.3. 200-day MA: 199 missing values - Similarly, the 200-day moving average will lack values for the initial 199 days.4. 21-day Volatility: 21 missing values - The 21-day rolling volatility will be absent for the initial 21 days.Given the nature of these missing values (they're a result of our calculations and are expected due to the rolling window computations), we don't need to impute them. However, for some analyses, we might want to focus on dates after these initial missing values to ensure continuity.Next, let's ensure the "Date" column (which is now our index) is consistent in format.# Check the data type and consistency of the "Date" column (which is now the index) date_format_consistency = data.index date_format_consistency The "Date" column, which is now our index, is of type `datetime64[ns]`. This confirms that the dates are in a consistent datetime format, ensuring that our subsequent analyses are accurate and reliable.Step 3: Visualizations - Distribution PlotsVisualizations encompass the use of candlestick plots to intricately depict stock price movements, distribution plots to showcase price and volume variances, and heatmaps to represent monthly or yearly average stock prices.Prompt 3: In your role as a stock analyst, visualizing the distribution of data can offer profound insights into its behavior and characteristics. Now, direct your attention to the distribution plots of both the stock prices (specifically the Close/Last prices) and the trading volume. These plots will elucidate the spread, skewness, peaks, and central tendencies of the data.1. Distribution of Close/Last Prices: By examining this plot, you can discern the most frequent closing prices, identify any outliers, and understand the distribution's symmetry or skewness. Does it lean towards higher or lower prices? Are there any significant peaks?2. Distribution of Trading Volume: This visualization will shed light on the typical trading volume levels. Are there days of extraordinarily high or low trading volumes? Does the distribution hint at a particular trend in trading activity?Through these plots, you're better positioned to understand the stock's price movements and the market's interest in it, guiding future analysis or investment recommendations.Response: To visualize the distribution of data, we'll employ distribution plots (or histograms) for both the stock's closing prices and the trading volume.1. Distribution of Close/Last Prices: This will help us understand the frequency distribution of the closing prices, offering insights into the most common price levels, the data's symmetry or skewness, and potential outliers.2. Distribution of Trading Volume: This will shed light on the frequency distribution of trading volumes, allowing us to discern common volume levels and any anomalies in trading activity.Let's begin by visualizing the distribution of the stock's Close/Last prices.# Importing necessary libraries for visualization import seaborn as sns # Setting up the figure and axes fig, ax = plt.subplots(nrows=2, ncols=1, figsize=(15, 10)) # Distribution of Close/Last Prices sns.histplot(data['Close/Last'], bins=50, kde=True, ax=ax[0], color='blue') ax[0].set_title("Distribution of Close/Last Prices") ax[0].set_xlabel("Close/Last Price ($)") ax[0].set_ylabel("Frequency") # Distribution of Trading Volume sns.histplot(data['Volume'], bins=50, kde=True, ax=ax[1], color='gray') ax[1].set_title("Distribution of Trading Volume") ax[1].set_xlabel("Trading Volume") ax[1].set_ylabel("Frequency") plt.tight_layout() plt.show() Interpretation: The plots show how often the stock closes at different prices and how its trading volume varies. For closing prices, most often they land in a certain range, but sometimes they go a bit higher. There don't seem to be any unusual values, but a different chart could confirm this. The trading volume, or how much the stock is traded, usually stays within a specific range. However, there are rare days when it's traded a lot, possibly due to big news about the company. In short, these charts give a snapshot of the stock's usual behavior and any deviations from it, helping guide investment choices.Step 4: Visualizations – HeatmapHeat Map visualization showcases monthly average closing prices, empowering traders to identify recurring patterns, discern seasonality, and detect stock price shifts, thereby guiding informed investment choices.Prompt 9: As an astute stock analyst, appreciating the power of visualizations to detect patterns is vital. To enhance our understanding of the stock's behavior over time, let's craft a heatmap that displays the monthly average closing prices. Heatmaps, with their gradient color schemes, provide an intuitive way to quickly spot variations and trends. By focusing on monthly averages, this visualization will surface any recurring monthly patterns, seasonality, or discernible shifts in the stock's closing prices. Such insights can be instrumental for traders and investors seeking to capitalize on consistent temporal trends in the stock market.Response: The heatmap showcases the monthly average closing prices for the stock:# Define the price columns price_columns = ['Close/Last', 'Open', 'High', 'Low'] # Convert the price columns to float values by removing the '$' sign and converting to float for col in price_columns:    data[col] = data[col].str.replace('$', '', regex=False).astype(float) # Set the 'Date' column as the index data.set_index('Date', inplace=True) # Compute the daily returns data['Daily Returns'] = data['Close/Last'].pct_change() # Resample the data for monthly average and transform for heatmap monthly_avg = data['Close/Last'].resample('M').mean() heatmap_data = monthly_avg.groupby([monthly_avg.index.year, monthly_avg.index.month]).mean().unstack() # Plotting the heatmap plt.figure(figsize=(12, 6)) sns.heatmap(heatmap_data, cmap="YlGnBu", annot=True, fmt=".2f", cbar_kws={'label': 'Average Closing Price ($)'}) plt.title("Monthly Average Closing Prices") plt.xlabel("Month") plt.ylabel("Year") plt.show()  Interpretation:Color Gradient: The color gradient, ranging from light to dark shades of blue, represents the average closing price. Darker shades indicate higher average closing prices for that particular month, while lighter shades suggest lower averages.Annotations: Each cell in the heatmap is annotated with the specific average closing price for that month and year.By examining this heatmap:Patterns & Seasonality: You can spot any recurring monthly patterns, seasonality, or discernible shifts in the stock's closing prices over the years.Variations: The color variations can quickly help identify months with particularly high or low average closing prices.This visualization provides a comprehensive and intuitive view of the stock's price movements on a monthly basis, aiding traders and investors in spotting consistent temporal trends and guiding future analyses or investment strategies. Now let’s look into the future trend.Step 5: Predictive Modeling (Time Series Forecasting)In the realm of stock analysis, forecasting is crucial to anticipate market movements. Starting with chronological time series data, one should ensure its stationarity, utilize ACF and PACF for ARIMA parameter selection, then train, forecast, visualize the results, and continuously validate and refine, ensuring well-informed investment decisions.Prompt 5: As a dedicated stock analyst, I understand the nuances and intricacies of time series forecasting. When working with stock price data, the ARIMA (Autoregressive Integrated Moving Average) model can be an invaluable tool. Begin with your time series data, ensuring chronological ordering and handling any missing values. It's crucial to ensure the data's stationarity, as ARIMA thrives on stationary datasets. This can be validated visually or with tests like the Augmented Dickey-Fuller. Diving deeper, ARIMA's efficacy lies in its parameters: \(p\) for the lagged values, \(d\) for differencing, and \(q\) for the moving average term. Helpful tools like ACF and PACF plots can guide the selection of these parameters. Once defined, train your ARIMA model on the dataset. With this trained model, predict future stock prices for your desired time frame. Visual representation can be achieved by plotting the original time series alongside the ARIMA predictions. For a more holistic view, overlay the forecast's confidence intervals. Remember, in the world of stock forecasting, model validation is paramount. Compare your model's predictions against a holdout set if available, employing metrics such as MAE or RMSE. Continual refinement might be necessary, so be prepared to iterate and adjust as needed for the most accurate forecasting.Response: We can fit the ARIMA model to our data. However, it's worth noting that multiple combinations of \(p\), \(d\), and \(q\) might be suitable. Model selection criteria, like AIC or BIC, can help refine our choices.Let's proceed by fitting the ARIMA model using the initial parameters and then forecast future stock prices.from statsmodels.tsa.arima.model import ARIMA from datetime import timedelta # Initial parameters based on ACF and PACF plots p, d, q = 1, 1, 1 # Fit the ARIMA model model = ARIMA(data['Close/Last'], order=(p, d, q)) model_fit = model.fit() # Forecast the next 30 days forecast_steps = 30 forecast = model_fit.get_forecast(steps=forecast_steps) forecast_index = [data.index[-1] + timedelta(days=i) for i in range(1, forecast_steps + 1)] # Extract the forecast mean and confidence intervals forecast_mean = forecast.predicted_mean confidence_intervals = forecast.conf_int() # Plot the original series and the forecast plt.figure(figsize=(15, 7)) data['Close/Last'].plot(label='Observed', color='blue') plt.plot(forecast_index, forecast_mean, color='red', label='Forecast') plt.fill_between(forecast_index, confidence_intervals.iloc[:, 0], confidence_intervals.iloc[:, 1], color='pink', alpha=0.3) plt.title('Stock Closing Price Forecast using ARIMA') plt.xlabel('Date') plt.ylabel('Close Price ($)') plt.legend() plt.grid(True) plt.tight_layout() plt.show() Interpretation: The chart shows past stock prices (in blue) and what they might be in the next 30 days (in red), based on a prediction model called ARIMA. Around the red line is a pink area that suggests where the stock prices might realistically land. However, it's vital to remember that this is just an estimate. Predicting stock prices is tricky because so many different things can affect them. As time goes on, the pink area gets broader, meaning the predictions are less certain. While this model offers a glimpse into potential future prices, always be cautious when basing decisions on predictions, as the stock market is full of surprises.ConclusionStock analysis, often seen as a realm of pure numbers, is actually a delicate blend of art and science, interpretation paired with prediction. As we've journeyed through, advanced techniques like Volatility Analysis have provided clarity on the unpredictable nature of stocks, while Data Cleaning ensures that our foundation is rock-solid. Visual tools, especially intuitive heatmaps, act as a compass, highlighting subtle patterns and variations in monthly stock prices. At the heart of it all, Predictive Modeling stands as a beacon, illuminating potential future paths using the wisdom of past data. Whether one is just stepping into this vast ocean or is a seasoned navigator, the tools and techniques discussed here not only simplify the journey but also enhance the depth of understanding. In stock analysis, as in many fields, knowledge is power, and with these methods in hand, both newcomers and experts are well-equipped to make informed, strategic decisions in the dynamic world of stocks.Author BioDr. Anshul Saxena is an author, corporate consultant, inventor, and educator who assists clients in finding financial solutions using quantum computing and generative AI. He has filed over three Indian patents and has been granted an Australian Innovation Patent. Anshul is the author of two best-selling books in the realm of HR Analytics and Quantum Computing (Packt Publications). He has been instrumental in setting up new-age specializations like decision sciences and business analytics in multiple business schools across India. Currently, he is working as Assistant Professor and Coordinator – Center for Emerging Business Technologies at CHRIST (Deemed to be University), Pune Lavasa Campus. Dr. Anshul has also worked with reputed companies like IBM as a curriculum designer and trainer and has been instrumental in training 1000+ academicians and working professionals from universities and corporate houses like UPES, CRMIT, and NITTE Mangalore, Vishwakarma University, Pune & Kaziranga University, and KPMG, IBM, Altran, TCS, Metro CASH & Carry, HPCL & IOC. With a work experience of 5 years in the domain of financial risk analytics with TCS and Northern Trust, Dr. Anshul has guided master's students in creating projects on emerging business technologies, which have resulted in 8+ Scopus-indexed papers. Dr. Anshul holds a PhD in Applied AI (Management), an MBA in Finance, and a BSc in Chemistry. He possesses multiple certificates in the field of Generative AI and Quantum Computing from organizations like SAS, IBM, IISC, Harvard, and BIMTECH.Author of the book: Financial Modeling Using Quantum Computing
Read more
  • 0
  • 0
  • 9082

article-image-exporting-data-ms-access-2003-mysql
Packt
07 Oct 2009
4 min read
Save for later

Exporting data from MS Access 2003 to MySQL

Packt
07 Oct 2009
4 min read
Introduction It is assumed that you have a working copy of MySQL which you can use to work with this article. The MySQL version used in this article came with the XAMPP download. XAMPP is an easy to install (and use) Apache distribution containing MySQL, PHP, and Perl. The distribution used in this article is XAMPP for Windows. You can find documentation here. Here is a screen shot of the XAMPP control panel where you can turn the services on and off and carry out other administrative tasks. You need to follow the steps indicated here: Create a database in MySQL to which you will export a table from Microsoft Access 2003 Create a ODBC DSN that helps you connecting Microsoft Access to MySQL Export the table or tables Verify the exported items Creating a database in MySQL You can create a database in MySQL by using the command 'Create Database' in MySQL or using a suitable graphic user interface such as MySQL workbench. You will have to refer to documentation that works with your version of MySQL. Herein the following version was used. The next listing shows how a database named TestMove was created in MySQL starting from the bin folder of the MySQL program folder. Follow the commands and the response from the computer. The Listing 1 and the folders are appropriate for my computer and you may find it in your installation directory. The databases you will be seeing will be different from what you see here except for those created by the installation. Listing 1: Login and create a database Microsoft Windows XP [Version 5.1.2600](C) Copyright 1985-2001 Microsoft Corp.C:Documents and SettingsJayaram Krishnaswamy>cdC:>cd xamppmysqlbinC:xamppmysqlbin>mysql -h localhost -u root -pEnter password: *********Welcome to the MySQL monitor. Commands end with ; or g.Your MySQL connection id is 2Server version: 5.1.30-community MySQL Community Server (GPL)Type 'help;' or 'h' for help. Type 'c' to clear the buffer.mysql> show databases;+--------------------+| Database |+--------------------+| information_schema || cdcol || expacc || mengerie || mydb || mysql || phpmyadmin || test || testdemo || webauth |+--------------------+10 rows in set (0.23 sec)mysql> create database TestMove;Query OK, 1 row affected (0.00 sec)mysql> show databases;+--------------------+| Database |+--------------------+| information_schema || cdcol || expacc || mengerie || mydb || mysql || phpmyadmin || test || testdemo || testmove || webauth |+--------------------+11 rows in set (0.00 sec)mysql> The login detail that works error free is shown. The preference for host name is localhost v/s either the Machine Name (in this case Hodentek2) or the IP address. The first 'Show Databases' command does not display the TestMove we created which you can see in response to the 2nd Show Databases command. In windows the commands are not case sensitive. Creating an ODBC DSN to connect to MySQL When you install from XAMPP you will also be installing an ODBC driver for MySQL for the version of MySQL included in the bundle. In the MySQL version used for this article the version is MySQL ODBC5.1 and the file name is MyODBC5.dll. Click Start | Control Panel | Administrative Tools | Data Sources (ODBC) and open the ODBC Data Source Administrator window as shown. The default tab is User DSN. Change to System DSN as shown here. Click the Add... button to open the Create New Data Source window as shown. Scroll down and choose MySQL ODBC 5.1 Driver as the driver and click Finish. The MySQL Connector/ODBC Data Source Configuration window shows up. You will have to provide a Data Source Name (DSN) and a description. The server is the localhost. You must have your User Name/Password information to proceed further. The database is the name of the database you created earlier (TestMove) and this should show up in the drop-down list if the rest of the information is correct. Accept the default port. If all information is correct the Test button gets enabled. Click and test the connection using the Test button. You should get a response as shown. Click the OK button on the Test Result window. Click OK on the MySQL Connector/ODBC Data Source Configuration window. There are a number of other flags that you can set up using the 'Details' button. The defaults are acceptable for this article. You have successfully created a System DSN 'AccMySQL' as shown in the next window. Click OK. Verify the contents of TestMove The TestMove is a new database created in MySQL and as such it is empty as you verify in the following listing. Listing 2: Database TestMove is empty mysql> use testmove;Database changedmysql> show tables;Empty set (0.00 sec)mysql>
Read more
  • 0
  • 0
  • 9070

article-image-solving-least-privilege-problems-application-compatibility-toolkit
Packt
28 Jul 2010
5 min read
Save for later

Solving Least Privilege Problems with the Application Compatibility Toolkit

Packt
28 Jul 2010
5 min read
(For more resources on Microsoft products, see here.) Quick compatibility fixes using the Program Compatibility Wizard In small enterprise environments or on home computers, compatibility fixes can be applied without using the Application Compatibility Toolkit through the user interface. If you want to know how the Windows Application Compatibility Infrastructure works in detail or if you want to deploy fixes to multiple devices, skip straight to the second part of this article: Achieving application compatibility in enterprise environments. Applying compatibility modes to legacy applications Compatibility fixes can be grouped together to form compatibility modes. Windows XP, Vista, and Windows 7 all come with a default set of compatibility modes out of the box. System administrators can also use Compatibility Administrator to create their own compatibility modes. Compatibility modes can be applied directly from the user interface either by using the Program Compatibility Wizard in Windows XP, or by right-clicking on an executable file, or a shortcut to an executable file, and selecting Properties from the menu. Windows XP contains the following compatibility modes: Windows 95 Windows 98 / Windows ME Windows NT 4.0 (Service Pack 5) Windows 2000 As shown in the following screenshot, the Compatibility tab in Vista and Windows 7(left) looks a little different from Windows XP (right). Privilege level in Vista and Windows 7Note that the Run this program as an administrator option will be grayed out if an application manifest is included with the program. Program Compatibility Wizard Included in Windows Vista and later, the Program Compatibility Wizard allows users or system administrators to test legacy applications running against different compatibility modes. To launch the Program Compatibility Wizard in Windows XP, select Start | Accessories | Program Compatibility Wizard. To launch the Program Compatibility Wizard in Windows Vista, type the following command into the Search box on the Start menu:%systemroot%System32mshta.exe res://acprgwiz.dll/compatmode.htaTo launch the Program Compatibility Wizard in Windows 7, search for program compatibility troubleshooter in Start | Help and Support. Click Next on the welcome screen in Help and Support Center. Leave the default I want to choose from a list of programs option selected and click on Next. Select the application you want to test, in this case Maxthon, from the Select a program list and click on Next. Choose a compatibility mode from the list, such as Microsoft Windows 95 or Windows 2000, and click on Next. Compatibility modes set using the Program Compatibility Wizard only apply to the currently logged in user. Compatibility settings in Vista and Windows 7 can be applied to the logged in user or to all users.Administrative privileges are required to apply compatibility settings for a given application to all users of a system. If required, select display settings (256 colors or 640 x 480 screen resolution) to be used with the program and then click on Next. Click Next to test the compatibility mode against your application. The application will launch and you can test it to see if the selected compatibility mode resolves the identified problems. Once you've finished testing, close the application. Back in the Help and Support Center, choose to either apply the compatibility settings or try different settings. Browsing compatibility settings for each userIf compatibility modes are set on programs for specific users, you can use Compatibility Administrator to browse the settings for each user under the Per User Compatibility Settings node. This node only appears if per user settings are found on the system. Program Compatibility Assistant Introduced in Windows Vista, the Program Compatibility Assistant helps to automate the process of applying compatibility fixes to legacy applications by monitoring for known problems when programs are running. This feature runs as a service, and prompts users to apply suggested fixes for applications, either during the setup phase or when running an installed application. The Program Compatibility Assistant can detect the following problems automatically: Errors when launching setup programs Failures in install routines Failures caused by User Account Control An install needing to run as an administrator A control panel applet requiring administrative privileges Errors caused because a component is not present in the current version of Windows Notifying users about unsigned drivers on 64-bit versions of Windows Matching applications against a list of programs with known problems and notifying the user at program startup The Program Compatibility Assistant intercepts an installation routine, prompting the user to try again using recommended settings. Disabling the Program Compatibility Assistant The Program Compatibility Assistant is intended to help home users resolve problems with legacy applications. In an enterprise environment, to avoid potential problems with messages generated by the Program Compatibility Assistant, you should consider disabling the service in Group Policy. The Turn off Program Compatibility Assistant setting can be found in the Group Policy Management Editor under Computer or User Configuration | Policies | Administrative Templates | Windows Components | Application Compatibility. Excluding executables from the Program Compatibility Assistant The Program Compatibility Assistant doesn't monitor programs that have an application manifest that marks them as compatible with Vista or Windows 7. However, if you want to exclude a program but don't want to disable the Program Compatibility Assistant completely, you can create a REG_MULTI_SZ registry value named ExecutablesToExclude under the following key:HKLM SoftwareMicrosoftWindows NTCurrentVersionCompatibility Assistant
Read more
  • 0
  • 0
  • 9069
article-image-sql-server-powershell
Packt
19 Oct 2015
8 min read
Save for later

SQL Server with PowerShell

Packt
19 Oct 2015
8 min read
In this article by Donabel Santos, author of the book, SQL Server 2014 with Powershell v5 Cookbook explains scripts and snippets of code that accomplish basic SQL Server tasks using PowerShell. She discusses simple tasks such as Listing SQL Server Instances and Discovering SQL Server Services to make you comfortable working with SQL Server programmatically. However, even if ever you explore how to create some common database objects using PowerShell, keep in mind that PowerShell will not always be the best tool for the task. There will be tasks that are best completed using T-SQL. It is still good to know what is possible in PowerShell and how to do them, so you know that you have alternatives depending on your requirements or situation. For the recipes, we are going to use PowerShell ISE quite a lot. If you prefer running the script from the PowerShell console rather run running the commands from the ISE, you can save the scripts in a .ps1 file and run it from the PowerShell console. (For more resources related to this topic, see here.) Listing SQL Server Instances In this recipe, we will list all SQL Server Instances in the local network. Getting ready Log in to the server that has your SQL Server development instance as an administrator. How to do it... Let's look at the steps to list your SQL Server instances: Open PowerShell ISE as administrator. Let's use the Start-Service cmdlet to start the SQL Browser service: Import-Module SQLPS -DisableNameChecking #out of the box, the SQLBrowser is disabled. To enable: Set-Service SQLBrowser -StartupType Automatic #sql browser must be installed and running for us #to discover SQL Server instances Start-Service "SQLBrowser" Next, you need to create a ManagedComputer object to get access to instances. Type the following script and run: $instanceName = "localhost" $managedComputer = New-Object Microsoft.SqlServer.Management.Smo.Wmi.ManagedComputer $instanceName #list server instances $managedComputer.ServerInstances Your result should look similar to the one shown in the following screenshot: Notice that $managedComputer.ServerInstances gives you not only instance names, but also additional properties such as ServerProtocols, Urn, State, and so on. Confirm that these are the same instances you see from SQL Server Management Studio. Open SQL Server Management Studio. Go to Connect | Database Engine. In the Server Name dropdown, click on Browse for More. Select the Network Servers tab and check the instances listed. Your screen should look similar to this: How it works... All services in a Windows operating system are exposed and accessible using Windows Management Instrumentation (WMI). WMI is Microsoft's framework for listing, setting, and configuring any Microsoft-related resource. This framework follows Web-based Enterprise Management (WBEM). The DISTRIBUTED MANAGEMENT TASK FORCE, INC. (http://www.dmtf.org/standards/wbem) defines WBEM as follows: A set of management and Internet standard technologies developed to unify the management of distributed computing environments. WBEM provides the ability for the industry to deliver a well-integrated set of standard-based management tools, facilitating the exchange of data across otherwise disparate technologies and platforms. In order to access SQL Server WMI-related objects, you can create a WMI ManagedComputer instance: $managedComputer = New-Object Microsoft.SqlServer.Management.Smo.Wmi.ManagedComputer $instanceName The ManagedComputer object has access to a ServerInstance property, which in turn lists all available instances in the local network. These instances however are only identifiable if the SQL Server Browser service is running. The SQL Server Browser is a Windows Service that can provide information on installed instances in a box. You need to start this service if you want to list the SQL Server-related services. There's more... The Services instance of the ManagedComputer object can also provide similar information, but you will have to filter for the server type SqlServer: #list server instances $managedComputer.Services | Where-Object Type –eq "SqlServer" | Select-Object Name, State, Type, StartMode, ProcessId Your result should look like this: Instead of creating a WMI instance by using the New-Object method, you can also use the Get-WmiObject cmdlet when creating your variable. Get-WmiObject, however, will not expose exactly the same properties exposed by the Microsoft.SqlServer.Management.Smo.Wmi.ManagedComputer object. To list instances using Get-WmiObject, you will need to discover what namespace is available in your environment: $hostName = "localhost" $namespace = Get-WMIObject -ComputerName $hostName -Namespace rootMicrosoftSQLServer -Class "__NAMESPACE" | Where-Object Name -like "ComputerManagement*" #see matching namespace objects $namespace #see namespace names $namespace | Select-Object -ExpandProperty "__NAMESPACE" $namespace | Select-Object -ExpandProperty "Name" If you are using PowerShell v2, you will have to change the Where-Object cmdlet usage to use the curly braces {} and the $_ variable: Where-Object {$_.Name -like "ComputerManagement*" } For SQL Server 2014, the namespace value is: ROOTMicrosoftSQLServerComputerManagement12 This value can be derived from $namespace.__NAMESPACE and $namespace.Name. Once you have the namespace, you can use this with Get-WmiObject to retrieve the instances. We can use the SqlServiceType property to filter. According to MSDN (http://msdn.microsoft.com/en-us/library/ms179591.aspx), these are the values of SqlServiceType: SqlServiceType Description 1 SQL Server Service 2 SQL Server Agent Service 3 Full-Text Search Engine Service 4 Integration Services Service 5 Analysis Services Service 6 Reporting Services Service 7 SQL Browser Service Thus, to retrieve the SQL Server instances, we need to provide the full namespace ROOTMicrosoftSQLServerComputerManagement12. We also need to filter for SQL Server Service type, or SQLServiceType = 1. The code is as follows: Get-WmiObject -ComputerName $hostName -Namespace "$($namespace.__NAMESPACE)$($namespace.Name)" -Class SqlService | Where-Object SQLServiceType -eq 1 | Select-Object ServiceName, DisplayName, SQLServiceType | Format-Table –AutoSize Your result should look similar to the following screenshot: Yet another way to list all the SQL Server instances in the local network is by using the System.Data.Sql.SQLSourceEnumerator class, instead of ManagedComputer. This class has a static method called Instance.GetDataSources that will list all SQL Server instances: [System.Data.Sql.SqlDataSourceEnumerator]: :Instance.GetDataSources() | Format-Table -AutoSize When you execute, your result should look similar to the following: If you have multiple SQL Server versions, you can use the following code to display your instances: #list services using WMI foreach ($path in $namespace) { Write-Verbose "SQL Services in:$($path.__NAMESPACE)$($path.Name)" Get-WmiObject -ComputerName $hostName ` -Namespace "$($path.__NAMESPACE)$($path.Name)" ` -Class SqlService | Where-Object SQLServiceType -eq 1 | Select-Object ServiceName, DisplayName, SQLServiceType | Format-Table –AutoSize } Discovering SQL Server Services In this recipe, we will enumerate all SQL Server Services and list their statuses. Getting ready Check which SQL Server services are installed in your instance. Go to Start | Run and type services.msc. You should see a screen similar to this: How to do it... Let's assume you are running this script on the server box: Open PowerShell ISE as administrator. Add the following code and execute: Import-Module SQLPS -DisableNameChecking #you can replace localhost with your instance name $instanceName = "localhost" $managedComputer = New-Object Microsoft.SqlServer.Management.Smo.Wmi.ManagedComputer $instanceName #list services $managedComputer.Services | Select-Object Name, Type, ServiceState, DisplayName | Format-Table -AutoSize Your result will look similar to the one shown in the following screenshot: Items listed in your screen will vary depending on the features installed and running in your instance Confirm that these are the services that exist in your server. Check your services window. How it works... Services that are installed on a system can be queried using WMI. Specific services for SQL Server are exposed through SMO's WMI ManagedComputer object. Some of the exposed properties are as follows: ClientProtocols ConnectionSettings ServerAliases ServerInstances Services There's more... An alternative way to get SQL Server-related services is by using Get-WMIObject. We will need to pass in the host name as well as the SQL Server WMI Provider for the ComputerManagement namespace. For SQL Server 2014, this value is ROOTMicrosoftSQLServerComputerManagement12. The script to retrieve the services is provided here. Note that we are dynamically composing the WMI namespace. The code is as follows: $hostName = "localhost" $namespace = Get-WMIObject -ComputerName $hostName -NameSpace rootMicrosoftSQLServer -Class "__NAMESPACE" | Where-Object Name -like "ComputerManagement*" Get-WmiObject -ComputerName $hostname -Namespace "$($namespace.__NAMESPACE)$($namespace.Name)" -Class SqlService | Select-Object ServiceName If you have multiple SQL Server versions installed and want to see just the most recent version's services, you can limit to the latest namespace by adding Select-Object –Last 1: $namespace = Get-WMIObject -ComputerName $hostName -NameSpace rootMicrosoftSQLServer -Class "__NAMESPACE" | Where-Object Name -like "ComputerManagement*" | Select-Object –Last 1 Yet another alternative but less accurate way of listing possible SQL Server related services is the following snippet of code: #alterative - but less accurate Get-Service *SQL* This uses the Get-Service cmdlet and filters base on the service name. This is less accurate because this grabs all processes that have SQL in the name, but may not necessarily be related to SQL Server. For example, if you have MySQL installed, it will get picked up as a process. Conversely, this will not pick up SQL Server-related services that do not have SQL in the name, such as ReportServer. Summary You will find that many of the scripts can be accomplished using PowerShell and SQL Management Objects (SMO). SMO is a library that exposes SQL Server classes that allow programmatic manipulation and automation of many database tasks. For some , we will also explore alternative ways of accomplishing the same tasks using different native PowerShell cmdlets. Now that we have a gist of SQL Server 2014 with PowerShell, lets build a full-fledged e-commerce project with SQL Server 2014 with Powershell v5 Cookbook. Resources for Article: Further resources on this subject: Exploring Windows PowerShell 5.0 [article] Working with PowerShell [article] Installing/upgrading PowerShell [article]
Read more
  • 0
  • 0
  • 9069

article-image-installing-zenoss
Packt
07 Oct 2009
8 min read
Save for later

Installing Zenoss

Packt
07 Oct 2009
8 min read
Installing Zenoss Our first step is to choose one of the three installation methods: virtual appliance, binary installer, or source. The virtual appliance makes a good choice, if we want to evaluate or demonstrate Zenoss. The virtual appliance runs a functional Zenoss system using VMware Player or VMware Server out-of-the-box and needs no Linux knowledge. When run from VMware, the Zenoss virtual appliance may be used to monitor networks with relatively few devices. The binary installer makes a good choice if we want to avoid building Zenoss from source, and we run a supported distribution. The Supported Operating Systems section in this tutorial includes a list of distributions that have binary installation support. We can build from source on a variety of Unix-based environments, like Ubuntu and Mac OS X. A source installation gives us the ability to install Zenoss in the environment of our choice but requires more work. Of the three installation methods, a source install requires the most familiarity with your operating system and presents more points of failure. As we move beyond installing Zenoss to set up, we focus on firewall policies and Simple Network Management Protocol (SNMP) for Linux and Windows systems. Even though Zenoss can use other methods to monitor devices, SNMP is the default monitoring protocol. We are free to change how we monitor and collect information at any time. During the installation and the set up, we work from the command line because it's fast and it's consistent from one distribution to the next. If an error does occur, we can see the error immediately printed to the terminal window. When working from the command line, we assume knowledge of two basic tasks: opening the terminal window and navigating the file structure. For all other tasks, the book provides the exact command to type. After installation and set up, we spend most of our time working with Zenoss through the web interface. Let's get this installation out of the way so we can discover Zenoss. Server Specifications Actual server specifications may vary depending on the amount and frequency of the data you collect. Zenoss Inc. recommends the following hardware specifications as a starting point based on feedback from the community: Network with up to 250 devices: 4 GB RAM Core 2 Duo E6300 1.86/1066 RTL 75 GB disk storage Network with more than 250 devices : 8 GB RAM XEON 5120 DC 1.86/1066/4MB Four 75 GB drives in two RAID-1 pairs Supported Operating Systems Zenoss requires a Unix-based platform and installs on systems capable of running a GNU build environment. However, Zenoss supports only a few distributions with binary installers. The following table shows the available installation options. Installation Type   Platform   Virtual Appliance   Windows Linux   Binary Installer Red Hat Enterprise Linux 5 Fedora Core 6 SUSE   Source Ubuntu FreeBSD Solaris 10 Mac 0S X Other Linux environments   As more binary installers become available, Zenoss posts them to http://www.zenoss.com/download Zenoss Dependencies Virtual appliance users do not need to install any dependencies because they are included in the image. For all other installations, you need to install the following software packages prior to installing Zenoss: MySQL 5.0.22 or higher MySQL development environment Python 2.3.5 or 2.4 Python development environment If you plan to build a Zenoss installation from source code, you need to install the following: SWIG Autoconf GNU build environment Dependent software packages are available via your distribution's normal software package manager. However, the package names and installation commands vary based on distribution. Consult your distribution's documentation for more information. Quick Start with Virtual Appliance If we know how to download and install software in our host environment, we can get a working Zenoss system with the virtual appliance. The Zenoss virtual appliance packages a working Zenoss Core installation inside a Linux guest that can be booted from a host system, including Windows, using VMware's Player, Server, or Workstation programs. The virtual appliance is great for: Users with little or no Linux knowledge Demonstrations and Evaluations Monitoring small networks with a few devices Install Virtual Appliance We will finish the installation as fast as we can download files and install the VMware Player. Let's begin: Download the VMware Player from http://www.vmware.com/player/.Registration is required to complete the download. Install VMware Player according to VMware's installation instructions for your operating system. Download the Zenoss virtual appliance from http://www.zenoss.com/download/ Unzip the Zenoss virtual appliance download file to a working directory in your system. Open VMware Player: On Windows, select Start > Programs > VMware Player. On Linux, select VMplayer from the application menu, or type the command: vmware VMwarePlayer prompts us to load the virtual machine configuration file we previously unzipped, as shown in the following screenshot: Open the Zenoss virtual appliance we unzipped in step 4.   The Zenoss virtual appliance takes a few minutes to load depending on the performance of your system. When the appliance boots, a welcome window opens and displays the IP address of the Zenoss management console and the standard Linux login prompt, as shown in the following screenshot:                 When we connect to Zenoss through our web browser, we use the IP address of the Zenoss management console that displays on the welcome screen (e.g. http://192.168.1.125.8080/). We cannot access our virtualized Zenoss installation by navigating to localhost, which is the host name of the Zenoss virtual appliance. If the IP address of the Zenoss console does not display, we can obtain the IP address using the ifconfig command, as described in the next section: Working with The Virtual Appliance. Zenoss is ready to monitor. Our next step is to set up the servers on our network to be monitored. If you can't wait to see Zenoss in action, feel free to skip the server setup section for now and check out Tutorial 4 for an introduction to web interface. You can come back and set up your servers later.      If this is the first time you are working with VMware or Linux, take a few minutes to get acquainted with the environment.      Working with The Virtual Appliance The Zenoss virtual appliance is a streamlined but functional Linux system, which means we can log in and have access to the underlying Linux environment. Let's cover a few basic tasks.      In order to type inside the virtual appliance window, use the keyboard shortcut: Ctrl + G      To return the cursor to the host desktop, use the keyboard shortcut: Ctrl + Alt      By default, the root login does not have a password assigned. To log in to the virtual appliance, enter the following user name at the login prompt: root   To set a password for the root user, enter the command:      passwd The passwd command prompts us to enter a new password. Assigning a password to the root user makes the system more secure and allows us to connect to the virtual appliance as root via SSH.      The IP address of the Zenoss virtual appliance is displayed at the top of the terminal window when the appliance loads. The most confusing part about using the Zenoss virtual appliance may be picking the correct IP address and port number. We connect to Zenoss on port 8080. So if our virtual appliance has an IP address of 192.168.1.103, then we use http://192.168.1.103:8080/o open the Zenoss login screen. If we use port 8003, we access the rPath management console, which is the underlying system used to build the Zenoss virtual appliance. After login, we can find additional IP configuration as shown in the following screenshot, with the command: ifconfig     To shut down the virtual appliance, select Player > Exit from the VMware Player. We may also use the the command: shutdown -h now If we shut down the virtual appliance, Zenoss no longer monitors the network and the web interface is not accessible.We may now jump ahead to the Server Setup section of this tutorial for help in configuring the servers we wish to monitor. Binary Installation Zenoss provides a binary installer in RPM format for Red Hat Enterprise Linux, which covers CentOS and Fedora Core. Binaries for additional distributions are added by Zenoss as the market demands and as time allows. To install Zenoss and its dependencies on Red Hat: Download the latest RPM for Red Hat Enterprise Linux from http://www.zenoss.com/download/ Open a terminal window and become the root user: su - If you have not yet installed the Zenoss dependencies, run: yum -y install mysql mysql-server net-snmp net-snmp-utils / python python-dev Install the Zenoss RPM by running the following command from the download directory where x.x-x equals the latest version number: rpm -ivh zenoss-2.x.x-x.el5.i386.rpm Start SNMP: service snmp start Start MySQL: service mysqld start Start Zenoss: /etc/init.d/zenoss start Let's test our installation. Open a browser and enter the URL of the Zenoss server, which listens on port 8080 (for example http://192.168.115:8080/ ). A screen appears as shown in the following screenshot.  
Read more
  • 0
  • 0
  • 9064
Modal Close icon
Modal Close icon