Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-customizing-ipython
Packt
03 Feb 2016
9 min read
Save for later

Customizing IPython

Packt
03 Feb 2016
9 min read
In this article written by Cyrille Rossant, author of Learning IPython for Interactive Computing and Data Visualization - Second edition, we look at how the Jupyter Notebook is a highly-customizable platform. You can configure many aspects of the software and can extend the backend (kernels) and the frontend (HTML-based Notebook). This allows you to create highly-personalized user experiences based on the Notebook. In this article, we will cover the following topics: Creating a custom magic command in an IPython extension Writing a new Jupyter kernel Customizing the Notebook interface with JavaScript Creating a custom magic command in an IPython extension IPython comes with a rich set of magic commands. You can get the complete list with the %lsmagic command. IPython also allows you to create your own magic commands. In this section, we will create a new cell magic that compiles and executes C++ code in the Notebook. We first import the register_cell_magic function: In [1]: from IPython.core.magic import register_cell_magic To create a new cell magic, we create a function that takes a line (containing possible options) and a cell's contents as its arguments, and we decorate it with @register_cell_magic, as shown here: In [2]: @register_cell_magic def cpp(line, cell): """Compile, execute C++ code, and return the standard output.""" # We first retrieve the current IPython interpreter # instance. ip = get_ipython() # We define the source and executable filenames. source_filename = '_temp.cpp' program_filename = '_temp' # We write the code to the C++ file. with open(source_filename, 'w') as f: f.write(cell) # We compile the C++ code into an executable. compile = ip.getoutput("g++ {0:s} -o {1:s}".format( source_filename, program_filename)) # We execute the executable and return the output. output = ip.getoutput('./{0:s}'.format(program_filename)) print('n'.join(output)) C++ compiler This recipe requires the gcc C++ compiler. On Ubuntu, type sudo apt-get install build-essential in a terminal. On OS X, install Xcode. On Windows, install MinGW (http://www.mingw.org) and make sure that g++ is in your system path. This magic command uses the getoutput() method of the IPython InteractiveShell instance. This object represents the current interactive session. It defines many methods for interacting with the session. You will find the comprehensive list at http://ipython.org/ipython-doc/dev/api/generated/IPython.core.interactiveshell.html#IPython.core.interactiveshell.InteractiveShell. Let's now try this new cell magic. In [3]: %%cpp #include<iostream> int main() { std::cout << "Hello world!"; } Out[3]: Hello world! This cell magic is currently only available in your interactive session. To distribute it, you need to create an IPython extension. This is a regular Python module or package that extends IPython. To create an IPython extension, copy the definition of the cpp() function (without the decorator) to a Python module, named cpp_ext.py for example. Then, add the following at the end of the file: def load_ipython_extension(ipython): """This function is called when the extension is loaded. It accepts an IPython InteractiveShell instance. We can register the magic with the `register_magic_function` method of the shell instance.""" ipython.register_magic_function(cpp, 'cell') Then, you can load the extension with %load_ext cpp_ext. The cpp_ext.py file needs to be in the PYTHONPATH, for example in the current directory. Writing a new Jupyter kernel Jupyter supports a wide variety of kernels written in many languages, including the most-frequently used IPython. The Notebook interface lets you choose the kernel for every notebook. This information is stored within each notebook file. The jupyter kernelspec command allows you to get information about the kernels. For example, jupyter kernelspec list lists the installed kernels. Type jupyter kernelspec --help for more information. At the end of this section, you will find references with instructions to install various kernels including IR, IJulia, and IHaskell. Here, we will detail how to create a custom kernel. There are two methods to create a new kernel: Writing a kernel from scratch for a new language by re-implementing the whole Jupyter messaging protocol. Writing a wrapper kernel for a language that can be accessed from Python. We will use the second, easier method in this section. Specifically, we will reuse the example from the last section to write a C++ wrapper kernel. We need to slightly refactor the last section's code because we won't have access to the InteractiveShell instance. Since we're creating a kernel, we need to put the code in a Python script in a new folder named cpp: In [1]: %mkdir cpp The %%writefile cell magic lets us create a cpp_kernel.py Python script from the Notebook: In [2]: %%writefile cpp/cpp_kernel.py import os import os.path as op import tempfile # We import the `getoutput()` function provided by IPython. # It allows us to do system calls from Python. from IPython.utils.process import getoutput def exec_cpp(code): """Compile, execute C++ code, and return the standard output.""" # We create a temporary directory. This directory will # be deleted at the end of the 'with' context. # All created files will be in this directory. with tempfile.TemporaryDirectory() as tmpdir: # We define the source and executable filenames. source_path = op.join(tmpdir, 'temp.cpp') program_path = op.join(tmpdir, 'temp') # We write the code to the C++ file. with open(source_path, 'w') as f: f.write(code) # We compile the C++ code into an executable. os.system("g++ {0:s} -o {1:s}".format( source_path, program_path)) # We execute the program and return the output. return getoutput(program_path) Out[2]: Writing cpp/cpp_kernel.py Now we create our wrapper kernel by appending some code to the cpp_kernel.py file created above (that's what the -a option in the %%writefile cell magic is for): In [3]: %%writefile -a cpp/cpp_kernel.py """C++ wrapper kernel.""" from ipykernel.kernelbase import Kernel class CppKernel(Kernel): # Kernel information. implementation = 'C++' implementation_version = '1.0' language = 'c++' language_version = '1.0' language_info = {'name': 'c++', 'mimetype': 'text/plain'} banner = "C++ kernel" def do_execute(self, code, silent, store_history=True, user_expressions=None, allow_stdin=False): """This function is called when a code cell is executed.""" if not silent: # We run the C++ code and get the output. output = exec_cpp(code) # We send back the result to the frontend. stream_content = {'name': 'stdout', 'text': output} self.send_response(self.iopub_socket, 'stream', stream_content) return {'status': 'ok', # The base class increments the execution # count 'execution_count': self.execution_count, 'payload': [], 'user_expressions': {}, } if __name__ == '__main__': from ipykernel.kernelapp import IPKernelApp IPKernelApp.launch_instance(kernel_class=CppKernel) Out[3]: Appending to cpp/cpp_kernel.py In production code, it would be best to test the compilation and execution, and to fail gracefully by showing an error. See the references at the end of this section for more information. Our wrapper kernel is now implemented in cpp/cpp_kernel.py. The next step is to create a cpp/kernel.json file describing our kernel: In [4]: %%writefile cpp/kernel.json { "argv": ["python", "cpp/cpp_kernel.py", "-f", "{connection_file}" ], "display_name": "C++" } Out[4]: Writing cpp/kernel.json The argv field describes the command that is used to launch a C++ kernel. More information can be found in the references below. Finally, let's install this kernel with the following command: In [5]: !jupyter kernelspec install --replace --user cpp Out[5]: [InstallKernelSpec] Installed kernelspec cpp in /Users/cyrille/Library/Jupyter/kernels/cpp The --replace option forces the installation even if the kernel already exists. The --user option serves to install the kernel in the user directory. We can test the installation of the kernel with the following command: In [6]: !jupyter kernelspec list Out[6]: Available kernels: cpp python3 Now, C++ notebooks can be created in the Notebook, as shown in the following screenshot: C++ kernel in the Notebook Finally, wrapper kernels can also be used in the IPython terminal or the Qt console, using the --kernel option, for example ipython console --kernel cpp. Here are a few references: Kernel documentation at http://jupyter-client.readthedocs.org/en/latest/kernels.html Wrapper kernels at http://jupyter-client.readthedocs.org/en/latest/wrapperkernels.html List of kernels at https://github.com/ipython/ipython/wiki/IPython%20kernels%20for%20other%20languages bash kernel at https://github.com/takluyver/bash_kernel R kernel at https://github.com/takluyver/IRkernel Julia kernel at https://github.com/JuliaLang/IJulia.jl Haskell kernel at https://github.com/gibiansky/IHaskell Customizing the Notebook interface with JavaScript The Notebook application exposes a JavaScript API that allows for a high level of customization. In this section, we will create a new button in the Notebook toolbar to renumber the cells. The JavaScript API is not stable and not well-documented. Although the example in this section has been tested with IPython 4.0, nothing guarantees that it will work in future versions without changes. The commented JavaScript code below adds a new Renumber button. In [1]: %%javascript // This function allows us to add buttons // to the Notebook toolbar. IPython.toolbar.add_buttons_group([ { // The button's label. 'label': 'Renumber all code cells', // The button's icon. // See a list of Font-Awesome icons here: // http://fortawesome.github.io/Font-Awesome/icons/ 'icon': 'fa-list-ol', // The callback function called when the button is // pressed. 'callback': function () { // We retrieve the lists of all cells. var cells = IPython.notebook.get_cells(); // We only keep the code cells. cells = cells.filter(function(c) { return c instanceof IPython.CodeCell; }) // We set the input prompt of all code cells. for (var i = 0; i < cells.length; i++) { cells[i].set_input_prompt(i + 1); } } }]); Executing this cell displays a new button in the Notebook toolbar, as shown in the following screenshot: Adding a new button in the Notebook toolbar You can use the jupyter nbextension command to install notebook extensions (use the --help option to see the list of possible commands). Here are a few repositories with custom JavaScript extensions contributed by the community: https://github.com/minrk/ipython_extensions https://github.com/ipython-contrib/IPython-notebook-extensions So, we have covered several customization options of IPython and the Jupyter Notebook, but there’s so much more that can be done. Take a look at the IPython Interactive Computing and Visualization Cookbook to learn how to create your own custom widgets in the Notebook.
Read more
  • 0
  • 0
  • 4994

article-image-ubuntu-user-interface-tweaks
Packt
01 Oct 2009
10 min read
Save for later

Ubuntu User Interface Tweaks

Packt
01 Oct 2009
10 min read
I have spent time on all of the major Desktop operating systems and Linux is by far the most customizable. The GNOME Desktop environment, the default environment of Ubuntu and many other Linux distributions, is a very simple yet very customizable interface. I have spent a lot of time around a lot of Linux users and rarely do I find two Desktops that look the same. Whether it is simple desktop background customizations or much more complex UI alterations, GNOME allows you to make your desktop your own. Just like any other environment that you're going to find yourself in for an extended period of time, you're going to want to make it your own. The GNOME Desktop offers this ability in a number of ways. First of all I'll cover perhaps the more obvious methods, and then I'll move to the more complex. As mentioned in the introduction, by the end of this article you'll know how to automate (script) the customization of your desktop down to the very last detail. This is perfect for those that find themselves reinstalling their machines on a regular basis. Appearance GNOME offers a number of basic customizations within the Applications menu. To use the "Appearance Preferences" tool simply navigate to: System > Preferences > Appearance You'll find that the main screen allows you to change your basic theme. The theme includes the environment color scheme, icon set and window bordering. This is often one of the very first things that users will change on a new installation. Of the default theme selections I generally prefer "Clearlooks" over the default Ubuntu brown color. The next tab allows you to set your background. This is the graphic, color or gradient that you want to appear on your desktop. This is also a very common customization. More often than not users will find third-party graphics specific in this section. A great place to find user-generated desktop content is the http://gnome-look.org website. It is dedicated to user-generated Artwork for the GNOME and Ubuntu desktop. On the third tab you'll find Fonts. I have found that fonts do play a very important role in the look of your desktop. For the longest time I didn't bother with customizing my fonts, but after being introduced to a few that I like, it is a must-have in my desktop customization list. My personal preference is to use the "Droid Sans" font, at 10pt for all settings. I think this is a very clean, crisp font design that really changes the desktop look and feel.  If you'd like to try out this font set you'll have to install it. This can be done using: sudo aptitude install ttf-droid Another noticeable customization to your desktop on the Fonts tab is the Rendering option. For laptops you'll definitely want to select the bottom-right option, "Subpixel Smoothing (LCDs)". You should notice a change right away when you check the box. Finally, the Details button on the fonts tab can make great improvements to the over all look. This is where you can set your font resolution. I highly suggest setting this value to 96 dots per inch (dpi). Recent versions of Ubuntu try to dynamically detect the preferred dpi. Unfortunately I haven't had the best of luck with this new feature, so I've continued to set it manually. I think you'll notice a change if your system is on something other than 96. Setting the font to "Droid Sans" 10pt and the resolution to 96 dpi is one of the biggest visual changes that I make to my system! The final tab in the Appearances tool is the Interface. This tab allows you to customize simple things like whether or not your Applications menu should display icons or not. Personally, I have found that I like the default settings, but I would suggest trying a few customizations and finding out what you like. If you've followed the suggestions so far I'm sure your desktop likely looks a lot different than it did out of the box. By changing the theme, desktop background, font and dpi you may have already made drastic changes. I'd like to also share with you some of the additional changes that I make, which will help demonstrate some of the more advanced, little known features of the GNOME desktop. gconf-editor A default Ubuntu system comes with a behind-the-scenes tool called the "gconf-editor". This is basically a graphical editor for your entire GNOME configuration settings. At first use it can be a bit confusing, but once you figure out where and how to find your preferred settings it becomes much easier. To launch the gconf-editor press the key combination ALT-F2 and type: gconf-editor I have heard people compare this tool to Microsoft's registry tool, but I assure you that it is far less complicated! It simply stores GNOME configuration and application settings. It even includes the changes that you made above! Anytime you make a change to the graphical interface it gets stored, and this tool is a graphical way to view those changes. Let's change something else, this time using the gconf-editor. Another of my favorite interface customizations includes the panels. By default you have two panels, one at the top and one at the bottom of your screen. I prefer to have both panels at the top of my screen, and I like them to be a bit smaller than they are out of the box. Here is how we would make that change using the gconf-editor. Navigate to Edit > Search and search for bottom_panel or top_panel. I will start with bottom_panel. You should come up with a few results, the first one being /apps/panel/toplevels/bottom_panel_screen0. You can now customize the color, size, auto-hide feature and much more of your panel. If you find orientation, double-click the entry, and change the value to "top" you'll find that your panel instantly moves to the top of the screen. You may want to alter the size entry while you're in there as well. Make a note of the Key name that you see for each item. These will come in handy a little bit later. A few other settings that you might find interesting are Nautilus desktop settings such as: computer_icon_visible home_icon_visible network_icon_visible trash_icon_visible volumes_visible These are simple check-box settings, activating or deactivating an option upon click. Basically these allow you to toggle the computer, home, network or trash icons on your desktop. I prefer to make sure each of these is turned off. The only one that I do like to keep on is volumes_visible. Try this out yourself and see what you prefer. Automation Earlier I mentioned that you'll want to make note of the Key Name for the settings that you're playing with. It is these names that allow us to automate, or script, the customization of our desktop environment. After putting a little bit of time into finding the key names for each of the customizations that I like I am now able to completely customize every aspect of my desktop by running a simple script! Let me give you a few examples. Above we found that the key name for the bottom panel was: /apps/panel/toplevels/bottom_panel_screen0 The key name specifically for the orientation was: /apps/panel/toplevels/bottom_panel_screen0/orientation The value we changed was top or bottom. We can now make this change from the command line using by typing: gconftool-2 -s --type string /apps/panel/toplevels/bottom_panel_screen0/orientation top Let us see a few more examples, these will change the font settings for each entry that we saw in the Appearances menu: gconftool -s --type string /apps/nautilus/preferences/desktop_font "Droid Sans 10"gconftool -s --type string /apps/metacity/general/titlebar_font "Droid Sans 10"gconftool -s --type string /desktop/gnome/interface/monospace_font_name "Droid Sans 10"gconftool -s --type string /desktop/gnome/interface/document_font_name "Droid Sans 10"gconftool -s --type string /desktop/gnome/interface/font_name "Droid Sans 10" You may or may not have made these changes manually, but just think about the time you could save on your next Ubuntu installation by pasting in these five commands instead! I will warn you though, once you start making a list of gconftool commands it's hard to stop. Considering how simple it is to make environment changes using simple commands, why not list everything! I'd like to share the script that I use to make my preferred changes. You'll likely want to edit the values to match your preferences. #!/bin/bash## customize GNOME interface# (christer@rootcertified.com)#gconftool-2 -s --type string /apps/nautilus/preferences/desktop_font "Droid Sans 10"gconftool-2 -s --type string /apps/metacity/general/titlebar_font "Droid Sans 10"gconftool-2 -s --type string /desktop/gnome/interface/monospace_font_name "Droid Sans 10"gconftool-2 -s --type string /desktop/gnome/interface/document_font_name "Droid Sans 10"gconftool-2 -s --type string /desktop/gnome/interface/font_name "Droid Sans 10"gconftool-2 -s --type string /desktop/gnome/interface/icon_theme "gnome-brave"gconftool-2 -s --type bool /apps/nautilus/preferences/always_use_browser truegconftool-2 -s --type bool /apps/nautilus/desktop/computer_icon_visible falsegconftool-2 -s --type bool /apps/nautilus/desktop/home_icon_visible falsegconftool-2 -s --type bool /apps/nautilus/desktop/network_icon_visible falsegconftool-2 -s --type bool /apps/nautilus/desktop/trash_icon_visible falsegconftool-2 -s --type bool /apps/nautilus/desktop/volumes_visible truegconftool-2 -s --type bool /apps/nautilus-open-terminal/desktop_opens_home_dir truegconftool-2 -s --type bool /apps/gnome-do/preferences/Do/Platform/Linux/TrayIconPreferences/StatusIconVisible truegconftool-2 -s --type bool /apps/gnome-do/preferences/Do/CorePreferences/QuietStart truegconftool-2 -s --type bool /apps/gnome-terminal/profiles/Default/default_show_menubar falsegconftool-2 -s --type string /apps/gnome-terminal/profiles/Default/font "Droid Sans Mono 10"gconftool-2 -s --type string /apps/gnome-terminal/profiles/Default/scrollbar_position "hidden"gconftool-2 -s --type string /apps/gnome/interface/gtk_theme "Shiki-Brave"gconftool-2 -s --type string /apps/gnome/interface/icon_theme "gnome-brave"gconftool-2 -s --type integer /apps/panel/toplevels/bottom_panel_screen0/size 23gconftool-2 -s --type integer /apps/panel/toplevels/top_panel_screen0/size 23 Summary By saving the above script into a file called "gnome-setup" and running it after a fresh installation I'm able to update my theme, fonts, visible and non-visible icons, gnome-do preferences, gnome-terminal preferences and much more within seconds. My desktop actually feels like my desktop again! I find that maintaining a simple file like this greatly eases the customization of my desktop environment and lets me focus on getting things done. I no longer spend an hour tweaking each little setting to make my machine my home again. I install, run my script, and get to work! If you have read this article you may be interested to view : Compiling and Running Handbrake in Ubuntu Control of File Types in Ubuntu Ubuntu 9.10: How To Upgrade Install GNOME-Shell on Ubuntu 9.10 "Karmic Koala" Five Years of Ubuntu Control of File Types in Ubuntu What's New In Ubuntu 9.10 "Karmic Koala" Create a Local Ubuntu Repository using Apt-Mirror and Apt-Cacher
Read more
  • 0
  • 0
  • 4993

article-image-tips-and-tricks-ibm-filenet-p8-content-manager
Packt
24 Feb 2011
11 min read
Save for later

Tips and Tricks on IBM FileNet P8 Content Manager

Packt
24 Feb 2011
11 min read
Getting Started with IBM FileNet P8 Content Manager Install, customize, and administer the powerful FileNet Enterprise Content Management platform Quickly get up to speed on all significant features and the major components of IBM FileNet P8 Content Manager Provides technical details that are valuable both for beginners and experienced Content Management professionals alike, without repeating product reference documentation Gives a big picture description of Enterprise Content Management and related IT areas to set the context for Content Manager Written by an IBM employee, Bill Carpenter, who has extensive experience in Content Manager product development, this book gives practical tips and notes with a step-by-step approach to design real Enterprise Content Management solutions to solve your business needs   Installation care If you are using a virtual server image with snapshot capability, it’s a good idea to use snapshots. In fact, we recommend that after each of the major installation steps. If something goes wrong in a later step, you can recover back to the snapshot point to save yourself the trouble of starting over. WAS Bootstrap Hostname In a development environment, the domain name might not resolve in your DNS. In that case, enter the IP address for that server instead. Populating Tivoli Directory Server (TDS) We could use the TDS Realms interface to construct our users and groups. If you use TDS in your enterprise, that’s a good way to go. It offers several user interface niceties for directory administration, and it also offers partial referential integrity for the entries. Directory concepts and notation Directory concepts and notation can seem pretty odd. Most people don’t encounter them every day. There is a lot of material available on the web to explain both the concepts and the notation. Here is one example that is clearly written and oriented toward directory novices: http://www.skills-1st.co.uk/papers/ldapschema-design-feb-2005/index.html. Close up all of the nodes before you exit FEM FEM remembers the state of the tree view from session to session. When you start FEM the next time, it will try to open the nodes you had open when you exited. That will often mean something of a delay as it reads extensive data for each open Object Store node. You might find it a useful habit to close up all of the nodes before you exit FEM. Using topology levels A set of configuration data, if used, is used as the complete configuration. That is, the configuration objects at different topology levels are not blended to create an "effective configuration". Trace logging Although similar technologies are used to provide trace logging in the CE server and the client APIs, the configuration mechanisms are completely separate. The panels in FEM control only tracing within the CE server and do not apply to any client tracing. If you find that performance still drags or that the trace log file continues to grow even after you have disabled trace logging in the Domain configuration, it could be that trace logging is still configured at a more specific level. That's very easy to overlook, especially in more complex deployments or where CM administration duties are shared. Collaborative checkout Even with a collaborative checkout, the subsequent checkin is still subject to access checks, so you can still have fine-grained control over that. In fact, because you can use fine-grained security to limit who can do a checkin, you might as well make the Object Store default be Collaborative unless you have some specific use case that demands Exclusive. Cancel the creation of the class Although the final step in the Create a Class Wizard will still let you cancel the creation of the class, any property templates and choice lists you created along the way will already have been created in the Object Store. If you wish to completely undo your work, you will have to delete them manually. FEM query interface A historical quirk of the FEM query interface is that the SELECT list must begin with the This property. That is not a general requirement of CE SQL. Running the CSE installer If you are running the CSE installer and eventually the CSE itself on the same machine as the CE, you might be tempted to use localhost as the CSE server host. From the CE point of view, that would be technically correct. However, exploiting little tricks like that is a bad habit to get into. It certainly won't work in any environment where you install the CSE separately from the CE or have multiple CSE servers installed. We suggest you use a proper host name. Be sure to get the server name correct since the installer and the Verity software will sprinkle it liberally throughout several configuration files. If it is not correct by default, which is one of the hazards of using dynamic IP addresses, correct it now. CBR Locale field uni stands for Unicode and is generally the best choice for mixed languages support. If you think you don't need mixed-languages support, there's a pretty good chance you are mistaken, even if all of your users have the same locale settings in their environments. In any case, if you are tempted to use a different CBR locale, you should first read the K2 locale customization guide, since it's a reasonably complicated topic. Process Service does not start If the Process Service does not start, check to make sure that the Windows service named Process Engine Services Manager is started. If not, start it manually and make sure it is marked it for automatic startup. Configure two WAS profiles When trying to duplicate configuration aspects of one WAS profile into another WAS profile, we could theoretically have the WAS consoles open simultaneously in separate browser windows, which would facilitate side-by-side comparisons. In practice, this is likely to confuse the browser cookies holding the session information and drive you slightly crazy. If you have two different browsers installed, for example Firefox and Internet Explorer, you can open one WAS console in each. Disk space used by XT Disk space used by XT may exceed your expectations. We recommend having at least 2 gigabytes of disk space available when doing an XT installation. A lot of that can be recovered after XT is deployed into the application server. Object deletion Deleted objects are really, permanently deleted by the CE server. There is no undo or recycle bin or similar mechanism unless an application implements one. Notional locking & Cooperative locking Don't confuse the notional locking that comes via checkout with the unrelated feature of cooperative locking. Cooperative locking is an explicit mechanism for applications to mark a Document, Folder, or Custom Object as being locked. As the name implies, this only matters for applications which check for and honor cooperative locks. The CE will not prevent any update operation—other than locking operations themselves—just because there is a cooperative lock on the object. Synchronous or asynchronous subscription As a terminology convenience, events or event handlers are sometimes referred to as being synchronous or asynchronous. This is not technically correct because the designation is always made on the subscription object. An event can have either kind of subscription, and an event handler can be invoked both ways. Synchronous subscription event handlers The CE does not always throw an exception if the event handler for a synchronous subscription updates the triggering object. This has allowed many developers to ignore the rule that such updates are not allowed, assuming it is merely a best practice. Nonetheless, it has always been the rule that synchronous subscription event handlers are not allowed to do that. Even if it works in a particular instance, it may fail at random times that escape detection in testing. Don't fall into this trap! AddOn in the P8 Domain If you don't happen to be a perfect person, you might have to iterate a few times during the creation and testing of your AddOn until you get things exactly the way you want them. For the sake of mere mechanical efficiency, we usually do this kind of work using a virtual machine image that includes a snapshot capability. We make a snapshot just before creating the AddOn in the P8 Domain. Then we do the testing. If we need to iterate, it's pretty fast to roll back to the snapshot point. "anonymous access" complaints from the CE When an application server sees a Subject that it doesn't trust, since there is no trust relationship with the sending application server, it will often simply discard the Subject or strip vital information out of it. Hence the complaints from the CE that you are trying to do "anonymous access" often mean that there is something wrong with your trust relationship setup. Unknown ACE An "unknown" Access Control Entry (ACE) in an Access Control List (ACL) comes about because ACEs sometimes get orphaned. The user or group mentioned in the ACE gets deleted from the directory, but the ACE still exists in the CE repository. These ACEs will never match any calling user and so will never figure into any access control calculation. Application developers have to be aware of this kind of ACE when programmatically displaying or modifying the ACL. The unknown ACEs should be silently filtered out and not displayed to end users. (FEM displays unknown ACEs, but it is an administrator tool.) If updates are made to the ACL, the unknown ACEs definitely must be filtered out. Otherwise, the CE will throw an exception because it cannot resolve the user or group in the directory. Virtualization Several years ago, CM product documentation said that virtual machine technology was supported, but that you might have to reproduce any problems directly on physical hardware if you needed support. That's no longer the case, and virtualization is supported as a first-class citizen. For your own purposes, you will probably want to evaluate whether there are any significant performance costs to the virtualization technology you have chosen. The safest way to evaluate that is under similar configuration and load as that of your intended production environment. File Storage Area Folders used internally within a File Storage Area for content have no relationship to the folders used for filing objects within an Object Store. On reflection, this should be obvious, since you can store content for unfiled documents. Whereas the folders in an Object Store are an organizing technique for objects, the folders in a File Storage Area are used to avoid overwhelming the native filesystem with too many files in a single directory (which can impact performance). Sticky sessions All API interactions with the CE are stateless. In other words, except for load balancing, it doesn't matter which CE server is used for any particular API request. Requests are treated independently, and the CE does not maintain any session state on behalf of the application. On the other hand, some CM web applications do need to be configured for sticky sessions. A sticky session means that incoming requests (usually from a web browser) must return to the same copy of the application for subsequent requests. Disaster Recovery (DR) There is technology available for near real time replication for DR. It can be tempting to think of your DR site as your data backup, or at least eliminating the need for traditional backups. It seems too good to be true since all of your updates are almost instantaneously copied to another datacenter. The trap is that the replication can't tell desirable updates from mistakes. If you have to recover some of your data because of an operational mistake (for example, if you drop the tables in an Object Store database), the DR copy will reflect the same mistake. You should still do traditional backups even if you have a replicated DR site. Further resources on this subject: IBM FileNet P8 Content Manager: Administrative Tools and Tasks [Article] IBM FileNet P8 Content Manager: Exploring Object Store-level Items [Article] IBM FileNet P8 Content Manager: End User Tools and Tasks [Article]
Read more
  • 0
  • 0
  • 4991

article-image-drupal-faqs
Packt
30 Dec 2010
5 min read
Save for later

Drupal FAQs

Packt
30 Dec 2010
5 min read
  Drupal 7 First Look Learn the new features of Drupal 7, how they work and how they will impact you Get to grips with all of the new features in Drupal 7 Upgrade your Drupal 6 site, themes, and modules to Drupal 7 Explore the new Drupal 7 administration interface and map your Drupal 6 administration interface to the new Drupal 7 structure Complete coverage of the DBTNG database layer with usage examples and all API changes for both Themes and Modules Also includes an Appendix that introduces the beta release for Drupal 7. It is not a part of the book (print or e-book) but only available for free download Appendix         Read more about this book       Q: What is Drupal?A: Drupal is an Open Source Content Management System used for building dynamic websites.   Q: Why should I use Drupal and not any other CMS?A: By building on relevant standards and open source technologies, Drupal supports and enhances the potential of the Internet as a medium where diverse and geographically separated individuals and groups can collectively produce, discuss, and share information and ideas. With a central interest in and focus on communities and collaboration, Drupal's flexibility allows the collaborative production of online information systems and communities.   Q: What are the minimum requirements for Drupal 7?A: Drupal 7 requires PHP 5.2.0 or later to run the Drupal code. You will also need one of the following databases to run Drupal 7: MySQL version 5.0 or later PostgreSQL 8.3 or later SQLite 3.4.2 or later   Q: Where can one download Drupal 7 from?A: Head on over to http://drupal.org/project/drupal and click on the Drupal version number you wish to download—in this case it is Drupal 7. Click on Download and then save it to your C: drive or your My Documents folder (or wherever you want).   Q: What's new in Drupal 7?A: There are several key functionalities that made it to Drupal 7. Some of them are as follows: New administration toolbar and overlay administration: After installing Drupal 7 you will notice the new administration toolbar (shown in the following screenshot) that appears on all pages if you have the permission to administer the site: (Move the mouse over the image to enlarge it.) The toolbar groups commonly used tasks together making it easier for new administrators to learn how to configure Drupal and making it quicker for experienced administrators to get to commonly-used functionality. New Field API: The Field API allows site administrators to add additional attributes to a node type. It also supports translatable fields to allow for multi-lingual sites. Added context information to messages during translation: Drupal 7 adds an optional context for the translation to allow developers and themers to make translatable strings less ambiguous. Built-in automated cron functionality: Drupal 7 includes a new cron system that does not rely on running cron from the Unix cron system. The mechanism used is similar to the one used by poormanscron except that it runs from an AJAX request rather than delaying the response time of the page triggering cron. Added a new plugin manager: The plugin manager allows automatic updates of your Drupal installation. Seven theme for administration: A common complaint of Drupal administrators in previous versions was the look of the administration interface and that it could be difficult to tell when you were in the administration interface, since it used the same theme as regular content by default. To fix this, Drupal 7 has added a new administration theme called the Seven theme that is enabled by default. jQuery UI to core: jQuery UI (http://jqueryui.com) is a powerful JavaScript library that includes common controls like calendars, progress bars, tabs, sliders, and more. It also includes functionality to allow drag and drop, resizing, sorting, selection, and more. New Stark theme: The new Stark theme that is designed to make it easier to learn how to build a custom theme. Rewritten database layer (DBTNG): The biggest change in Drupal 7, at least for developers, is the new database layer, also called DBTNG (short for Database Layer: The Next Generation). DBTNG is a big change for developers since it changes how modules interact with the database. Queue API for long-running tasks: Drupal 7 adds a Queue API to manage long-running tasks. In general, any task that takes more than 30 seconds to a minute would be an excellent candidate for the Queue API. New test framework: Drupal 7 adds a comprehensive test framework called testing that allows developers and site administrators to run tests against an existing Drupal installation to ensure that it is behaving properly.   Q: How has the installation process improved in Drupal 7?A: Drupal 7 has a new installation routine. It is designed to make it easier for new Drupal users to set up Drupal. The new installation offers two types of install—the regular installation and a minimal installation.   The Minimal installation is similar to previous versions. The new Standard installation automatically enables commonly-used functionality during the installation to save time after setup.   Q: How has the interface for creating content and new content types improved in Drupal 7?A: Improved interface for creating content: A big, but welcome, change for editors is the redesigned and updated interface to create and edit content. A sample of the interface is shown in the following screenshot:   The redesigned screen makes it easier to quickly navigate to specific sections within the content. Improved interface for creating new content types: The interface for creating content types has been redesigned to keep all of the options in a smaller space so navigation is easier and all information can be quickly accessed.
Read more
  • 0
  • 0
  • 4990

article-image-restful-web-service-mocking-and-testing-soapui-raml-and-json-slurper-script-assertion
Packt
26 Feb 2015
15 min read
Save for later

RESTful Web Service Mocking and Testing with SoapUI, RAML, and a JSON Slurper Script Assertion

Packt
26 Feb 2015
15 min read
In this article written by Rupert Anderson, the author of SoapUI Cookbook, we will cover the following topics: Installing the SoapUI RAML plugin Generating a SoapUI REST project and mock service using the RAML plugin Testing response values using JSON Slurper As you might already know, despite being called SoapUI, the product actually has an excellent RESTful web service mock and testing functionality. Also, SoapUI is very open and extensible with a great plugin framework. This makes it relatively easy to use and develop plugins to support APIs defined by other technologies, for example RAML (http://raml.org/) and Swagger (http://swagger.io/). If you haven't seen it before, RESTful API Modeling Language or RAML is a modern way to describe RESTful web services that use YAML and JSON standards. As a brief demonstration, this article uses the excellent SoapUI RAML plugin to: Generate a SoapUI REST service definition automatically from the RAML definition Generate a SoapUI REST mock with an example response automatically from the RAML definition Create a SoapUI TestSuite, TestCase, and TestStep to call the mock Assert that the response contains the values we expect using a Script Assertion and JSON Slurper to parse and inspect the JSON content This article assumes that you have used SoapUI before, but not RAML or the RAML plugin. If you haven't used SoapUI before, then you can still give it a shot, but it might help to first take a look at Chapters 3 and 4 of the SoapUI Cookbook or the Getting Started, REST, and REST Mocking sections at http://www.soapui.org/. (For more resources related to this topic, see here.) Installing the SoapUI RAML plugin This recipe shows how to get the SoapUI RAML plugin installed and checked. Getting ready We'll use SoapUI open source version 5.0.0 here. You can download it from http://www.soapui.org/ if you need it. We'll also need the RAML plugin. You can download the latest version (0.4) from http://sourceforge.net/projects/soapui-plugins/files/soapui-raml-plugin/. How to do it... First, we'll download the plugin, install it, restart SoapUI, and check whether it's available: Download the RAML plugin zip file from sourceforge using the preceding link. It should be called soapui-raml-plugin-0.4-plugin-dist.zip. To install the plugin, if you're happy to, you can simply unzip the plugin zip file to <SoapUI Home>/java/app/bin; this will have the same effect as manually performing the following steps:     Create a plugins folder if one doesn't already exist in <SoapUI Home>/java/app/bin/plugins.     Copy the soapui-raml-plugin-0.4-plugin.jar file from the plugins folder of the expanded zip file into the plugins folder.     Copy the raml-parser-0.9-SNAPSHOT.jar and snakeyaml-1.13.jar files from the ext folder of the expanded zip file into the <SoapUI Home>/java/app/bin/ext folder.     The resulting folder structure under <SoapUI Home>/java/app/bin should now be something like: If SoapUI is running, we need to restart it so that the plugin and dependencies are added to its classpath and can be used, and check whether it's available. When SoapUI has started/restarted, we can confirm whether the plugin and dependencies have loaded successfully by checking the SoapUI tab log: INFO:Adding [/Applications/SoapUI-5.0.0.app/Contents/Resources/app/bin/ext/raml-parser-0.9-SNAPSHOT.jar] to extensions classpath INFO:Adding [/Applications/SoapUI-5.0.0.app/Contents/Resources/app/bin/ext/snakeyaml-1.13.jar] to extensions classpath INFO:Adding plugin from [/Applications/SoapUI-5.0.0.app/Contents/java/app/bin/plugins/soapui-raml-plugin-0.4-plugin.jar] To check whether the new RAML Action is available in SoapUI, we'll need a workspace and a project:     Create a new SoapUI Workspace if you don't already have one, and call it RESTWithRAML.     In the new Workspace, create New Generic Project; just enter Project Name of Invoice and click on OK. Finally, if you right-click on the created Invoice project, you should see a menu option of Import RAML Definition as shown in the following screenshot: How it works... In order to concentrate on using RAML with SoapUI, we won't go into how the plugin actually works. In very simple terms, the SoapUI plugin framework uses the plugin jar file (soapui-raml-plugin-0.4-plugin.jar) to load the standard Java RAML Parser (raml-parser-0.9-SNAPSHOT.jar) and the Snake YAML Parser (snakeyaml-1.13.jar) onto SoapUI's classpath and run them from a custom menu Action. If you have Java skills and understand the basics of SoapUI extensions, then many plugins are quite straightforward to understand and develop. If you would like to understand how SoapUI plugins work and how to develop them, then please refer to Chapters 10 and 11 of the SoapUI Cookbook. You can also take a look at Ole Lensmar's (plugin creator and co-founder of SoapUI) RAML Plugin blog and plugin source code from the following links. There's more... If you read more on SoapUI plugins, one thing to be aware of is that open source plugins are now termed "old-style" plugins in the SoapUI online documentation. This is because the commercial SoapUI Pro and newer SoapUI NG versions of SoapUI feature an enhanced plugin framework with Plugin Manger to install plugins from a plugin repository (see http://www.soapui.org/extension-plugins/plugin-manager.html). The new Plugin Manager plugins are not compatible with open source SoapUI, and open source or "old-style" plugins will not load using the Plugin Manager. See also More information on using and understanding various SoapUI plugins can be found in Chapter 10, Using Plugins of the SoapUI Cookbook More information on how to develop SoapUI extensions and plugins can be found in Chapter 11, Taking SoapUI Further of the SoapUI Cookbook The other open source SoapUI plugins can also be found in Ole Lensmar's blog, http://olensmar.blogspot.se/p/soapui-plugins.html Generating a SoapUI REST service definition and mock service using the RAML plugin In this section, we'll use the SoapUI RAML plugin to set up a SoapUI REST service definition, mock service, and example response using an RAML definition file. This recipe assumes you've followed the previous one to install the RAML plugin and create the SoapUI Workspace and Project. Getting ready For this recipe, we'll use the following simple invoice RAML definition: #%RAML 0.8title: Invoice APIbaseUri: http://localhost:8080/{version}version: v1.0/invoice:   /{id}:     get:       description: Retrieves an invoice by id.       responses:         200:           body:             application/json:             example: |                 {                   "invoice": {                     "id": "12345",                     "companyName": "Test Company",                     "amount": 100.0                   }                 } This RAML definition describes the following RESTful service endpoint and resource:http://localhost:8080/v1.0/invoice/{id}. The definition also provides example JSON invoice content (shown highlighted in the preceding code). How to do it... First, we'll use the SoapUI RAML plugin to generate the REST service, resource, and mock for the Invoice project created in the previous recipe. Then, we'll get the mock running and fire a test REST request to make sure it returns the example JSON invoice content provided by the preceding RAML definition: To generate the REST service, resource, and mock using the preceding RAML definition:     Right-click on the Invoice project created in the previous recipe and select Import RAML Definition.     Create a file that contains the preceding RAML definition, for example, invoicev1.raml.     Set RAML Definition to the location of invoicev1.raml.     Check the Generate MockService checkbox.     The Add RAML Definition window should look something like the following screenshot; when satisfied, click on OK to generate. If everything goes well, you should see the following log messages in the SoapUI tab: Importing RAML from [file:/work/RAML/invoicev1.raml] CWD:/Applications/SoapUI-5.0.0.app/Contents/java/app/bin Also, the Invoice Project should now contain the following:     An Invoice API service definition.     An invoice resource.     A sample GET request (Request 1).     An Invoice API MockService with a sample response (Response 200) that contains the JSON invoice content from the RAML definition. See the following screenshot: Before using the mock, we need to tweak Resource Path to remove the {id}, which is not a placeholder and will cause the mock to only respond to requests to /v1.0/invoice/{id} rather than /v1.0/invoice/12345 and so on. To do this:     Double-click on Invoice API MockService.     Double-click on the GET /v1.0/invoice/{id} action and edit the Resource Path as shown in the following screenshot: Start the mock. It should now publish the endpoint at http://localhost:8080/v1.0/invoice/and respond to the HTTP GET requests. Now, let's set up Request 1 and fire it at the mock to give it a quick test:     Double-click on Request 1 to edit it.     Under the Request tab, set Value of the id parameter to 12345.     Click on the green arrow to dispatch the request to the mock.     All being well, you should see a successful response, and under the JSON or RAW tab, you should see the JSON invoice content from the RAML definition, as shown in the following screenshot: How it works... Assuming that you're familiar with the basics of SoapUI projects, mocks, and test requests, the interesting part will probably be the RAML Plugin itself. To understand exactly what's going on in the RAML plugin, we really need to take a look at the source code on GitHub. A very simplified explanation of the plugin functionality is: The plugin contains a custom SoapUI Action ImportRamlAction.java. SoapUI Actions define menu-driven functionality so that when Import RAML Definition is clicked, the custom Action invokes the main RAML import functionality in RamlImporter.groovy. The RamlImporter.groovy class: Loads the RAML definition and uses the Java RAML Parser (see previous recipe) to build a Java representation of the definition. This Java representation is then traversed and used to create and configure the SoapUI framework objects (or Model items), that is, service definition, resource, mock, and its sample response. Once the plugin has done its work, everything else is standard SoapUI functionality! There's more... As you might have noticed under the REST service's menu or in the plugin source code, there are two other RAML menu options or Actions: Export RAML: This generates an RAML file from an existing SoapUI REST service; for example, you can design your RESTful API manually in a SoapUI REST project and then export an RAML definition for it. Update from RAML definition: Similar to the standard SOAP service's Update Definition, you could use this to update SoapUI project artifacts automatically using a newer version of the service's RAML definition. To understand more about developing custom SoapUI Actions, Model items, and plugins, Chapters 10 and 11 of the SoapUI Cookbook should hopefully help, as also the SoapUI online docs: Custom Actions: http://www.soapui.org/extension-plugins/old-style-extensions/developing-old-style-extensions.html Model Items: http://www.soapui.org/scripting---properties/the-soapui-object-model.html See also The blog article of the creator of the RAML plugin and the cofounder of SoapUI can be found at http://olensmar.blogspot.se/2013/12/a-raml-apihub-plugin-for-soapui.html The RAML plugin source code can be found at https://github.com/olensmar/soapui-raml-plugin There are many excellent RAML tools available for download at http://raml.org/projects.html For more information on SoapUI Mocks, see Chapter 3, Developing and Deploying Dynamic REST and SOAP Mocks of the SoapUI Cookbook Testing response values using JsonSlurper Now that we have a JSON invoice response, let's look at some options of testing it: XPath: Because of the way SoapUI stores the response, it is possible to use XPath Assertions, for example, to check whether the company is "Test Company": We could certainly use this approach, but to me, it doesn't seem completely appropriate to test JSON values! JSONPath Assertions (SoapUI Pro/NG only): The commercial versions of SoapUI have several types of JSONPath Assertion. This is a nice option if you've got it, but we'll only use open source SoapUI for this article. Script Assertion: When you want to assert something in a way that isn't available, there's always the (Groovy) Script Assertion option! In this recipe, we'll use option 3 and create a Script Assertion using JsonSlurper (http://groovy.codehaus.org/gapi/groovy/json/JsonSlurper.html) to parse and assert the invoice values that we expect to see in the response. How to do it.. First, we'll need to add a SoapUI TestSuite, TestCase, and REST Test Request TestStep to our invoice project. Then, we'll add a new Script Assertion to TestStep to parse and assert that the invoice response values are according to what we expect. Right-click on the Invoice API service and select New TestSuite and enter a name, for example, TestSuite. Right-click on the TestSuite and select New TestCase and enter a name, for example, TestCase. Right-click on TestCase:     Select Add TestStep | REST Test Request.     Enter a name, for example, Get Invoice.     When prompted to select REST method to invoke for request, choose Invoice API -> /{id} -> get -> Request 1.     This should create a preconfigured REST Test Request TestStep, like the one in the following screenshot. Open the Assertions tab:     Right-click on the Assertions tab.     Select Add Assertion | Script Assertion.     Paste the following Groovy script: import groovy.json.JsonSlurper def responseJson = messageExchange.response.contentAsString def slurper = new JsonSlurper().parseText(responseJson) assert slurper.invoice.id=='12345' assert slurper.invoice.companyName=='Test Company' assert slurper.invoice.amount==100.0 Start Invoice API MockService if it isn't running. Now, if you run REST Test Request TestStep, the Script Assertion should pass, and you should see something like this: Optionally, to make sure the Script Assertion is checking the values, do the following:     Edit Response 200 in Invoice API MockService.     Change the id value from 12345 to 54321.     Rerun the Get Invoice TestStep, and you should now see an Assertion failure like this: com.eviware.soapui.impl.wsdl.teststeps.assertions.basic.GroovyScriptAssertion@6fe0c5f3assert slurper.invoice.id=='12345'       |       |       | |       |       |       | false       |       |       54321       |       [amount:100.0, id:54321, companyName:Test Company]       [invoice:[amount:100.0, id:54321, companyName:Test Company]] How it works JsonSlurper is packaged as a part of the standard groovy-all-x.x.x,jar that SoapUI uses, so there is no need to add any additional libraries to use it in Groovy scripts. However, we do need an import statement for the JsonSlurper class to use it in our Script Assertion, as it is not part of the Groovy language itself. Similar to Groovy TestSteps, Script Assertions have access to special variables that SoapUI populates and passes in when Assertion is executed. In this case, we use the messageExchange variable to obtain the response content as a JSON String. Then, JsonSlurper is used to parse the JSON String into a convenient object model for us to query and use in standard Groovy assert statements. There's more Another very similar option would have been to create a Script Assertion to use JsonPath (https://code.google.com/p/json-path/). However, since JsonPath is not a standard Groovy library, we would have needed to add its JAR files to the <SoapUI home>/java/app/bin/ext folder, like the RAML Parser in the previous recipe. See also You may also want to check the response for JSON schema compliance. For an example of how to do this, see the Testing response compliance using JSON schemas recipe in Chapter 4, Web Service Test Scenarios of the SoapUI Cookbook. To understand more about SoapUI Groovy scripting, the SoapUI Cookbook has numerous examples explained throughout its chapters, and explores many common use-cases when scripting the SoapUI framework. Some interesting SoapUI Groovy scripting examples can also be found at http://www.soapui.org/scripting---properties/tips---tricks.html. Summary In this article, you learned about installing the SoapUI RAML plugin, generating a SoapUI REST project and mock service using the RAML plugin, and testing response values using JSON Slurper. Resources for Article: Further resources on this subject: SOAP and PHP 5 [article] Web Services Testing and soapUI [article] ADempiere 3.6: Building and Configuring Web Services [article]
Read more
  • 0
  • 0
  • 4988

article-image-defining-data-model-spatial-data-storage
Packt
04 Oct 2013
26 min read
Save for later

Defining a Data Model for Spatial Data Storage

Packt
04 Oct 2013
26 min read
(For more resources related to this topic, see here.) The spatial component of a real-world feature is the geometric representation of its shape in some coordinate space (either in 2D or 3D), and in vector space, this is referred to as its geometry. Oracle Spatial is designed to make spatial data management easier and more natural to users of location-enabled business applications and geographic information system (GIS) applications. Oracle allows the storage of spatial data in a table using the SDO_GEOMETRY data type that is just like any other data type in the database. Once the spatial data is stored in the Oracle database, it can be easily manipulated, retrieved, and related to all other data stored in the database. A spatial database should be designed just like any other database with a fully specified model. A fully specified model that is application independent should control the spatial data storage. A good data model supports and enhances application access without compromising the quality. In addition to these features, database features can be used to support applications that have limited functionality when it comes to table and column design. For example, some applications mandate a single spatial column per table or only a single homogeneous geometry type per spatial column. These limitations can be accommodated quite easily using database features such as views and triggers. In addition, there are a number of issues that arise when designing a data model that directly affects the data quality, performance, and access. The goal of this article is to give readers an understanding of how to model spatial data as SDO_GEOMETRY columns within tables, how to support spatial constraints for improved data quality, how to use synchronous and asynchronous triggers for implementing topological constraint checking, and to present methods for coping with multiple representations for faster web service access. All these issues, with solutions, are covered in this article: Defining a sample schema Using spatial metadata Using Oracle metadata views Using OGC metadata views Using different types of geometric representations Implementing tables with homogeneous and heterogeneous columns Implementing multiple representations for a single object Implementing multiple instances of a single column, for example, pre-thinned data for different scales and reprojection for faster web service access Restricting data access via views Using views to expose a single geometry type when multiple geometry types are present in the table Using views to expose tables with single geometry columns when multiple geometry columns are present in the table Implementing spatial constraints at the database level Restricting geometry types Spatial topological constraints Implementation of synchronous triggers Implementation of asynchronous triggers Defining a sample schema We will first define a sample schema that will be used for all the examples in this article. The schema is intended to model typical spatial assets maintained in a city-level GIS. Oracle Spatial provides all the functionality needed to model or describe the spatial properties of an asset (in modeling, it is often called an entity). This spatial description of an asset should not be treated differently from any other descriptive attribute. In addition, a data model should describe all assets/entities within it independently of any application. This should include, to the best of the ability of SQL, all business rules that define or control these assets/entities within the database, and these rules should be implemented using standard database practices. Defining the data model We use a schema with 12 tables to represent a spatial database for a city. This schema has tables to represent administrative areas managed at the city level, such as land parcels and neighborhoods, along with tables to manage natural features such as water boundaries. The LAND_PARCELS table has information about land at the lowest administrative level of the city. Buildings have to be fully contained in these land parcels. A table called BUILDING_FOOTPRINTS has information about all the buildings in the city. This table has the footprint of each building along with other information, such as name, height, and other attributes. Sets of neighborhoods are defined as a collection of land parcels to create more granular administrative areas. These neighborhoods are stored in the PLANNING_NEIGHBORHOODS table. There is a master table, BASE_ADDRESSES, to store information about all the valid street addresses in the city. Every record in the BUILDING_FOOTPRINTS table must have one parent record in this master address table. Note that the master address table does not list all the addresses of the apartments in a building. Rather, it stores one record for each street level address. So, each record in the BUILDING_FOOTPRINTS table has only one corresponding record in the master address table. There is also a master table, ROADS, that is used to store information about all the roads in the city. ROADS stores one record for each named road in the city so that all common information for the road can be stored together in one table. This is the only table in the schema without any geometry information. Each road in turn maps to a set of road segments that are stored in the ROAD_CLINES table. This table is used to store the geometric representation of center lines of road segments. This table also stores information about address ranges on these road segments. Road segments typically have different address ranges on the left side of the road and on the right side of the road. Each road segment also has a parent ROAD_ID associated with it from the ROADS table. A city usually manages sidewalks and other assets, such as street lamps, trashcans, and benches that are placed on these sidewalks. The SIDEWALKS table stores the information for all the sidewalks managed by the city. The CITY_FURNITURE table stores all the data corresponding to the assets, such as benches, streetlights, and trashcans. The ORTHO_PHOTOS table stores the collected information using aerial photography. The raster information stored in this table can be used to look for changes over time for the built-in features of the city. The water features of the city are stored in two different tables: the WATER_LINE table is used to store the water line features, such as creeks, rivers, and canals. The WATER_AREA table is used to store area features, such as lakes, rivers, and bays. The following figure shows the entity-relationship (E-R) diagram for this data model: The following figure shows the further E-R diagram for same data model: Creating tables in the schema Create a user called BOOK and assign it a password. Load the script <schema_load.sql> and it will create the tables required for running the examples described in this article. It will also create the Oracle Spatial metadata required for these tables. The following privileges are granted to the BOOK user: grant connect,resource to Book identified by <password>; grant connect, resource to book; grant create table to book; grant create view to book; grant create sequence to book; grant create synonym to book; grant create any directory to book; grant query rewrite to book; grant unlimited tablespace to book; Understanding spatial metadata Oracle Spatial requires certain metadata before the spatial data can be meaningfully used by applications. The database views that contain this metadata also act as a catalog for all the spatial data in the database. There are two basic views defined to store this metadata information: USER_SDO_GEOM_METADATA and ALL_SDO_GEOM_METADATA. The USER_ view is used to create a metadata entry for a single SDO_GEOMETRY column within a database table or view. An entry must be created for each SDO_GEOMETRY column within a table; entries for SDO_GEOMETRY columns in views are optional. If a table has more than one column of type SDO_GEOMETRY, then there is one metadata entry for each column of spatial data in that table. The ALL_ view shows all of the spatial layers that can be accessed by the current user. If a user has the Select grant on another user’s table with SDO_GEOMETRY columns, the first user can see the metadata entries for those tables in the ALL_ view. The views are set up so that owners of the spatial tables or views can create the metadata for them. And, the owner of a layer can grant read access to a layer to other users in the system. Granting a Select privilege on the table or view to other users will let them see the metadata for these tables and views. The ALL_ view displays all the spatial tables owned by the user along with other spatial tables for which the current user has read access. Spatial Reference System Each SDO_GEOMETRY object has a Spatial Reference System (SRS) associated with it, and all the SDO_GEOMETRY objects in a column should have the same SRS. In Oracle Spatial, a Spatial Reference ID (SRID) is used to associate an SRS with SDO_GEOMETRY objects. There are cases (for example, engineering drawings) where there is no SRS associated with an SDO_GEOMETRY object. In such cases, a NULL SRID is used to denote that the spatial data has no spatial reference information. An SRS can be geographic or non-geographic. A geographic SRS is used when the spatial data is used to represent features on the surface of the Earth. These types of SRS usually have a reference system that can relate the coordinates of the spatial data to locations on Earth. A unit of measurement is also associated with an SRS so that measurements can be done using a well-defined system. A non-geographic SRS is used when the spatial data is not directly related to locations on Earth. But these systems usually have a unit of measurement associated with them. Building floor plans is a good example of spatial data that is often not directly related to locations on Earth. A geographic system can be either geodetic or projected. Coordinates in a geodetic system are often described using longitude and latitude. In Oracle Spatial, the convention is to use longitude as the first coordinate and latitude as the second coordinate. A projected system is a Cartesian system that is defined as a planar projection based on the datum and projection parameters. Before an entry is created for a layer of data, the SRID associated with the data should be identified along with the tolerance to be used for the spatial layer. All the spatial data has an inherent accuracy associated with it. Hence, the tolerance value used for a spatial layer is very important and should be determined based on the accuracy of the data. Once these two values are identified, you are ready to create the metadata for the spatial layer. More on Spatial Reference Systems Oracle Spatial supports hundreds of SRSs, and it is very important to choose the right SRS for any given data set. The definition of an SRS can be easily obtained by looking at the well-known text (WKT) for that SRS. The WKTs for all the SRSs supplied as part of Oracle Spatial are available from the MDSYS.CS_SRS view. In addition to this view, there are several other metadata tables under MDSYS that contain more details on how these SRSs are defined. Oracle Spatial also supports the EPSG standard-based SRSs. SRS in Oracle Spatial is flexible and allows users to define new reference systems if they are not present in the supplied SRSs. Creating spatial metadata The tables used in our sample schema contain data that is geographically referenced. Spatial metadata can be created using Insert statements into the USER_SDO_GEOM_METADATA view. This view is defined as a public-writable view on top of MDSYS.SDO_GEOM_METADATA_TABLE that is used to store metadata for all the spatial columns in the database. Let us look at the metadata creation process for some of the spatial tables. Most of the tables used in the current schema have spatial data in the California State Plane Zone 3 Coordinate System. In Oracle Spatial, the corresponding SRID for this SRS is 2872. This coordinate system has foot as the unit of measurement and we will use 0.05 as our tolerance (that is, five-hundredths of a foot). The metadata is created using the Insert statement as shown in the following code: Insert Into USER_SDO_GEOM_METDATA Values (‘LAND_PARCELS’, ‘GEOM’, SDO_DIM_ARRAY(SDO_DIM_ARRAY(SDO_DIM_ELEMENT(‘X’, 5900000, 6100000, 0.05), SDO_DIM_ELEMENT(‘Y’,2000000, 2200000, 0.05)), 2872); The SDO_DIM_ELEMENT object is used to specify the lower and upper bounds for each dimension of the coordinate system along with the tolerance value. The metadata allows one entry for each dimension, even though it is very common to use the same tolerance value for the X and Y dimensions. When storing 3D data, it is very common to use a different tolerance value for the Z dimension. The BASE_ADDRESSES table has geometries stored in two columns: GEOMETRY and GEOD_GEOMETRY. The GEOMETRY column has data in the 2872 SRID, while the GEOD_GEOMETRY column has data in longitude and latitude. As this is a geodetic system, the tolerance for such systems is required to be in meters. So, a tolerance of 0.05 means a tolerance of 5cm. For geodetic data, it is recommended that the tolerance should not be less than 5cm for all of the topology and distance-based operations. Insert Into USER_SDO_GEOM_METDATA Values(‘BASE_ADDRESSES’, ‘GEOD_GEOM’, SDO_DIM_ARRAY((SDO_DIM_ELEMENT(‘Longitude’, -122.51436, -122.36638, .05), SDO_DIM_ELEMENT(‘Latitude’, 37.7081463, 37.8309382, .05)), 8307); As this is a geodetic system, the longitude range goes from -180 to 180 and the latitude range goes from -90 to 90. Even though it is normal practice to use these ranges for the metadata entry, many developers use the actual ranges spanned by the SDO_GEOMETRY object. Mapping tools and applications typically use this extent from the metadata to compute the initial extent of the data in each column of the spatial data. Any application looking for all the spatial columns in the database should select the data from the ALL_SDO_GEOM_METADATA view. This will return one row for each column of spatial data in the database that is visible to the current user. OGC-defined metadata views Open Geospatial Consortium (OGC) defines a different set of standardized metadata views. OGC standard metadata can be defined using a new set of tables or views in Oracle Spatial. For a simple solution for the OGC metadata schema, we will show a view-based implementation using the Oracle Spatial metadata table. All Oracle supplied packages, functions, and types of Oracle Spatial are in the MDSYS schema. It is generally not recommended to create any user objects under this schema as it might cause problems during database upgrades. Oracle also supplies another predefined schema called MDDATA that can be used for Oracle Spatial-related user objects that are general purpose in nature. We use this MDDATA schema to create the OGC metadata views. This user comes locked and it is recommended that you do not unlock this user. But, it does require a few privileges to make the following code work, so grant those privileges as required. Connect to the database as a user with SYSDBA privileges and execute all the following steps as the MDDATA user by changing the current schema to MDDATA. We need to grant an explicit Select privilege on SDO_GEOM_METADATA_TABLE to MDDATA. Alter session set current_schema=MDDATA; GRANT Select on MDSYS.SDO_GEOM_METADATA_TABLE to MDDATA; The OGC standard requires the geometry type as part of the metadata view. But, this is not part of the MDSYS owned metadata view and has to be computed based on the geometry table information stored in the MDSYS table. So, first define a function that can compute the geometry type based on the rows in the spatial tables. Note that this function just looks at the first non-NULL geometry and returns the type of that geometry. Users can modify this to make it look at the whole table to decide on the geometry type, but it can be a very expensive operation. Create Or Replace Function MDDATA.GET_GEOMETRY_TYPE (tsname varchar2, tname varchar2, cname varchar2) Return Number IS gtype number; Begin Begin execute immediate ‘ Select a.’|| SYS.DBMS_ASSERT.ENQUOTE_NAME(cname, false)|| ‘.sdo_gtype From ‘|| SYS.DBMS_ASSERT.ENQUOTE_NAME(tsname, false)||’.’|| SYS.DBMS_ASSERT.ENQUOTE_NAME(tname, false)|| ‘ a Where a.’|| SYS.DBMS_ASSERT.ENQUOTE_NAME(cname, false)|| ‘ is not null and rownum < 2’ Into gtype; Return gtype MOD 100; EXCEPTION When OTHERS Then Return 4; End; End; Notice all the uses of the ENQUOTE_NAME function from the SYS.DBMS_ASSERT package. This is used to avoid any possible SQL injection issues typically associated with functions that create SQL statements using the user supplied SQL. As we are creating a general purpose function that can be invoked by any user directly or indirectly, it is a good idea to protect the function from any possible SQL injection. Next, we define an OGC metadata view to see all the rows from the MDSYS owned metadata table. Create Or Replace View MDDATA.OGC_GEOMETRY_COLUMNS As Select GM.SDO_OWNER As F_TABLE_SCHEMA, GM.SDO_TABLE_NAME As F_TABLE_NAME, GM.SDO_COLUMN_NAME As F_GEOMETRY_COLUMN, Get_Geometry_Type(GM.sdo_owner, GM.sdo_table_name, GM.sdo_column_name) As GEOMETRY_TYPE, (Select count(*) From Table(GM.SDO_DIMINFO) ) As COORD_DIMENSION, GM.SDO_SRID As SRID From MDSYS.SDO_GEOM_METADATA_TABLE GM; And finally, we define a user view that will show all the geometry columns that are visible to the current user. Create Or Replace View GEOMETRY_COLUMNS As Select b.F_TABLE_SCHEMA , b.F_TABLE_NAME , b.F_GEOMETRY_COLUMN, b.COORD_DIMENSION, b.SRID, b.GEOMETRY_TYPE From MDDATA.OGC_GEOMETRY_COLUMNS b, ALL_OBJECTS a Where b.F_TABLE_NAME = a.OBJECT_NAME And b.F_TABLE_SCHEMA = a.OWNER And a.OBJECT_TYPE in (‘TABLE’, ‘SYNONYM’, ‘VIEW’); Grant Select On MDDATA.GEOMETRY_COLUMNS to public; Create PUBLIC SYNONYM GEOMETRY_COLUMNS FOR MDDATA.GEOMETRY_COLUMNS; Tolerance in Oracle Spatial Tolerance is used in Oracle Spatial to associate a level of precision with the data and to check the validity of geometries among other things. Tolerance should be derived based on the resolution and accuracy of the data. If the devices or methods used to collect the spatial data are correct up to a five-meter resolution, the tolerance for that layer should be set to 5 meters. The actual tolerance value, inserted into the metadata view depends on the real-world tolerance value and the unit of measurement used in the coordinate system is used for the column of spatial data. For example, let the tolerance for the spatial data be 5 centimeters and the unit of measurement of the coordinate system used for the spatial column is feet. Then, the five-centimeter value should first be converted to feet (1 centimeter is 0.032 feet)—this comes out to be 0.164 feet. So, you use a value of 0.164 for tolerance in the metadata. In practice, Oracle Spatial uses the following rules based on tolerance to determine if the geometry is valid or not. These are in addition to other topological consistency rules (as described by the OGC Simple Feature Specification) used to check the validity of geometries: If the distance between two consecutive vertices in the geometry is less than the tolerance value, the geometry is invalid. This rule applies to line-string and polygon type geometries. If the distance between a vertex and the nearest edge to that vertex in a polygon is less than the tolerance value, the geometry is invalid. This rule only applies to the polygon type geometries. Managing homogeneous and heterogeneous data If a spatial column in a table contains data of one single geometry type (for example, polygon or line-string, but not both), we can say that spatial data in that column is homogeneous. In other situations, a column may contain data from one or more geometry types (heterogeneous representation). For example, the spatial description of a rare and endangered flora object may normally be a single plant (a plant via a point), but in other situations, it may be an area (a patch via a polygon). Consider the CITY_FURNITURE table that is used for storing city assets like benches, trashcans, and streetlights. The geometry for the benches is represented using line-strings, while streetlights and trashcans are represented with points. It is perfectly correct, semantically, to store different types of observations within a single geometry column. However, while some mapping software systems can cope with multiple geometry types per SDO_GEOMETRY column, others, such as some traditional GIS packages, require homogeneity. We will describe how to achieve this next, when a column in the table has heterogeneous data. We define three views on top of the CITY_FURNITURE table corresponding to each of the types of data stored in the table. This table has three classes of objects: benches, trashcans, and streetlights. After the views are defined, we also need to create metadata entries for these views in USER_SDO_GEOM_METADATA so that any GIS tool can discover these views as if they are tables. Create the database views corresponding to each of the three types of data stored in the CITY_FURNITURE table. -- DDL for View CITY_FURN_BENCHES Create Or Replace FORCE VIEW CITY_FURN_BENCHES (FID, FEATTYPE, GEOM) As Select FID, FEATTYPE, GEOM From CITY_FURNITURE Where FEATTYPE=’BENCH’; -- DDL for View CITY_FURN_LIGHTS Create Or Replace FORCE VIEW CITY_FURN_LIGHTS (FID, FEATTYPE, GEOM) As Select FID, FEATTYPE, GEOM From CITY_FURNITURE Where FEATTYPE=’LIGHT’; -- DDL for View CITY_FURN_TRASHCANS Create Or Replace FORCE VIEW CITY_FURN_TRASHCANS (FID, FEATTYPE, GEOM) As Select FID, FEATTYPE, GEOM From CITY_FURNITURE Where FEATTYPE=’TRASHCAN’; The preceding examples show how to use other relational attributes to create the required views. Another way to do this is to constrain based on the SDO_GTYPE attribute of the SDO_GEOMETRY column. The following example shows how to do this for one of the preceding views, as the rest can be done with similar SQL: -- DDL for View CITY__FURN_BENCHES Create Or Replace FORCE VIEW CITY_FURN_BENCHES (FID, FEATTYPE, GEOM) AS Select FID, FEATTYPE, GEOM From CITY_FURNITURE A Where A.GEOM.SDO_GTYPE = 2002; Now create the metadata for each of these views so that any GIS can access this as if it is stored in a separate table. Note that these additional metadata entries are not required for the correct usage of Oracle Spatial. They are created only to facilitate the GIS tools that don’t support heterogeneous data in spatial columns. Insert Into USER_SDO_GEOM_METADATA Values ( ‘CITY_FURN_BENCHES’, ‘GEOM’, SDO_DIM_ARRAY(SDO_DIM_ELEMENT(‘X’, 5900000, 6100000, .05), SDO_DIM_ELEMENT(‘Y’, 2000000, 2200000, .05)), 2872); Insert Into USER_SDO_GEOM_METADATA Values( ‘CITY_FURN_LIGHTS’, ‘GEOM’, SDO_DIM_ARRAY(SDO_DIM_ELEMENT(‘X’, 5900000, 6100000, .05), SDO_DIM_ELEMENT(‘Y’, 2000000, 2200000, .05)), 2872); Insert Into USER_SDO_GEOM_METADATA Values( ‘CITY_FURN_TRASHCANS’, ‘GEOM’, SDO_DIM_ARRAY(SDO_DIM_ELEMENT(‘X’, 5900000, 6100000, .05), SDO_DIM_ELEMENT(‘Y’, 2000000, 2200000, .05)), 2872); How metadata is used Applications typically look at the ALL_SDO_GEOM_METADATA view to see the spatial tables available in the database for a given user. If you select the data from this view, now you will see 11 rows returned: 8 rows corresponding to tables and 3 rows corresponding to the views defined in the CITY_FURNITURE table. From an application point of view, it does not make any difference whether this data is stored in a view or a table. It will all look the same to the application. Sometimes it is useful to constrain the type of spatial data stored in the table to be homogeneous. For example, the ROAD_CLINES table should contain only linear geometries, as the roads are usually geometries of line type. This can be done by constraints that can be imposed by the spatial index defined in the ROAD_CLINES table. While creating the spatial index, provide the LAYER_GTYPE keyword and specify the type of data that will be stored in this table. -- DDL for Index ROAD_CLINES_SIDX Create Index ROAD_CLINES_SIDX ON ROAD_CLINES (GEOM) INDEXTYPE IS MDSYS.SPATIAL_INDEX PARAMETERS (‘LAYER_GTYPE=LINE’); Now, if you try to insert a row with a geometry that has a different SDO_GTYPE attribute than 2002, it will raise an error. Insert Into ROAD_CLINES Values ( 198999, 402, 0, 0, 2300, 2498, 190201, 23564000, 23555000, 94107, 10, ‘Y’, ‘DPW’, ‘Potrero Hill’, ‘190201’, ‘03RD ST’, ‘3’, ‘3RD’, SDO_GEOMETRY(2001, 2872, NULL, SDO_ELEM_INFO_ARRAY(1, 1, 1), SDO_ORDINATE_ARRAY( 6015763.86, 2104882.29))); * ERROR at line 1: ORA-29875: failed in the execution of the ODCIINDEXInsert routine ORA-13375: the layer is of type [2002] while geometry inserted has type [2001] ORA-06512: at “MDSYS.SDO_INDEX_METHOD_10I”, line 720 ORA-06512: at “MDSYS.SDO_INDEX_METHOD_10I”, line 225 The error message clearly indicates that the row that is currently being inserted has geometry with the wrong SDO_GTYPE attribute. This is the easiest way to strictly enforce the GTYPE constraints on the spatial data. However, this has the problem of rejecting the whole row when the geometry type does not match the LAYER_GTYPE keyword. And it is also not easy to log these cases as the error is thrown and the database moves on to process the next Insert statement. In some cases, the user might still want to insert the row into the table, but record cthe fact that there is invalid data in the row. Users can then come back and look at all the invalid entries and fix the issues. We will describe a few methods to do this logging and error processing later in this article. Using database check constraints Specifying the layer_gtype keyword is not the only way to constrain the type in a spatial layer. One can also use table level constraints to achieve the same result. With constraints, users get the additional benefit of specifying more complex constraints, such as allowing only points and lines in a layer. However, if only a single geometry type constraint is required, it is better to implement that constraint using the LAYER_GTYPE method as this is more efficient than the check constraint. These constraints can also be enforced with database triggers, and these trigger-based constraints are discussed in a later section. Alter Table CITY_FURNITURE ADD Constraint city_furniture_gtype_ck CHECK ( geom.sdo_gtype in (2002, 2001) ); Insert Into CITY_FURNITURE Values (432432, 'BENCH', SDO_GEOMETRY(2003,2872,NULL, SDO_ELEM_INFO_ARRAY(1, 1003, 3), SDO_ORDINATE_ARRAY(6010548.53, 2091896.34, 6010550.45,2091890.11))); This will fail with the following error: ERROR at line 1: ORA-02290: check Constraint (BOOK.CITY_FURNITURE_GTYPE_CK) violated Similarly, this approach can be useful for an OBJTYPE column check in the CITY_FURNITURE table. Alter Table CITY_FURNITURE ADD Constraint city_furniture_type_ck CHECK ( feattype in (‘BENCH’,’LIGHT’,’TRASHCAN’) ); Insert Into CITY_FURNITURE Values (432432, ‘LIGHTS’, SDO_GEOMETRY(2001,2872, SDO_POINT_TYPE(6010548.53, 2091896.34, NULL), NULL, NULL)); ERROR at line 1: ORA-02290: check Constraint (BOOK.CITY_FURNITURE_TYPE_CK) violated Now these two constraints are checking two independent columns, but what we really need is a more complex check to ensure each value of OBJTYPE has the corresponding SDO_GEOMETRY with the right type. That is, we want to make sure that TRASHCAN and LIGHT types have a point geometry and BENCH has a line geometry. Alter Table CITY_FURNITURE Drop Constraint city_furniture_gtype_ck; Alter Table CITY_FURNITURE Drop Constraint city_furniture_type_ck; Alter Table CITY_FURNITURE ADD Constraint city_furniture_objtype_geom_ck CHECK ( ( (“FEATTYPE”=’TRASHCAN’ Or “FEATTYPE”=’LIGHT’) AND “GEOM”.”SDO_GTYPE”=2001 ) Or (“FEATTYPE”=’BENCH’ AND “GEOM”.”SDO_GTYPE”=2002) /* Else Invalid combination */ ) ; Insert Into CITY_FURNITURE Values (432432, ‘BENCH’, SDO_GEOMETRY(2001,2872, SDO_POINT_TYPE(6010548.53, 2091896.34, NULL), NULL, NULL)); ERROR at line 1: ORA-02290: check Constraint (BOOK.CITY_FURNITURE_TYPE_CK) violated Multiple representations for the same objects In some situations, it is beneficial to have multiple representations for the same geometric feature. For example, an address usually has a point representation for its location. If a footprint of a building is associated with the address, then that footprint will be represented as a polygon. In some cases, a building might have many different point locations associated with it. One point may allow a GIS application to draw an icon for the building depending on its function (for example, a fire station). Another point may allow the building to be labeled with its street address, and finally another one may show an alternate location that is used for main delivery or emergency services entry at the back of the building. Similarly, a land parcel table can have an interior point of the parcel represented as point geometry in addition to the polygon representation. For a visualization application, it is sometimes useful to represent the land parcel as a point. When a map is displayed at a smaller scale (city level), the map will be cluttered if each land parcel is displayed as a polygon. In such cases, if land parcels are displayed as points with a suitable icon, the map will be less cluttered. When the map is displayed at a larger scale (street level), the same land parcel can be displayed as a polygon. Oracle Spatial allows such multiple representations by allowing multiple SDO_GEOMETRY columns in the same table. We first start with the BUILDING_FOOTPRINTS table and alter it to add an additional SDO_GEOMETRY column to allow the street address to be represented at the center of the building via a point feature. We can use a spatial function that can compute a point inside a polygon automatically to populate this column. Summary A spatial database application should be designed just like any other database application. It should follow the same standard data model practices like any other database application. In this article, we introduced a data model that can be used in a city-wide spatial data management system. This data model is used throughout the article to illustrate different aspects of spatial database application development. We discussed the use of triggers and queues to manage spatial data in the database. We also showed how to design database level constraints for spatial data management. We introduced the concepts of database triggers and queues and showed how they can be used for spatial data management in the database. Resources for Article: Further resources on this subject: Remote Job Agent in Oracle 11g Database with Oracle Scheduler [Article] Introduction to Oracle Service Bus & Oracle Service Registry [Article] Configuration, Release and Change Management with Oracle [Article]
Read more
  • 0
  • 0
  • 4982
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-sentiment-analysis-twitter-data-part-1
Janu Verma
21 Jan 2015
4 min read
Save for later

Sentiment Analysis of Twitter Data - Part 1

Janu Verma
21 Jan 2015
4 min read
Twitter represents a fundamentally new instrument to make social measurements. Millions of people voluntarily express their opinions across any topic imaginable — this data source is incredibly valuable for both research and business. There have been numerous studies on this data for sociological, political, economical, and network analytical questions. We can tap the vast amount of data from Twitter to generate “public opinion” towards certain topics by aggregating the individual tweet results over time. Sentiment Analysis aims to determine how a certain person or group reacts to a specific topic. Traditionally, we would run surveys to gather data and do statistical analysis. With Twitter, it works by extracting tweets containing references to the desired topic, computing the sentiment polarity and strength of each tweet, and then aggregating the results for all such tweets. Companies use this information to gather public opinion on their products and services, and make data-informed decisions. We can also track changes in the users’ opinion towards a topic over time, allowing us to identify the events that caused these changes. One of the first studies on Twitter data for sentiment was to study public perception of Obama’s performance as President. Another (fun) example could be the to explore the variation of sentiment regarding the TV series “Game of Thrones.” The unpredictable episode “The Rains of Castamere” resulted in a lot of negative tweets and a peak in the sentiment score. Also, we can look at the geocoded information in the tweets and analyze the relation between location and mood. For example, people in California may be happy about event X, while New Yorkers didn’t like it much. Sentiment analysis employs natural language processing (NLP), text mining and computational linguistics to extract subjective information from the textual data. Applications Sentiment analysis techniques find applications in technology, finance, and research. Some important applications of sentiment analysis are: predicting stocks computing movie-ratings discerning product satisfaction analyzing political or apolitical campaigns Techniques There are broadly two categories of sentiment analysis: Lexical Methods : These techniques employ dictionaries of words annotated with their semantic polarity and sentiment strength. This is then used to calculate a score for the polarity and/or sentiment of the document. Usually this method gives high precision but low recall. Machine Learning Methods: Such techniques require creating a model by training the classifier with labeled examples. This means that you must first gather a dataset with examples for positive, negative and neutral classes, extract the features from the examples and then train the algorithm based on the examples. These methods are used mainly for computing the polarity of the document. The choice of the method heavily depends upon the application, the domain and the language. Using lexicon-based techniques with large dictionaries enables you to achieve very good results. Nevertheless, these techniques require using a lexicon, something which is not always available in all languages. On the other hand, Machine Learning based techniques can deliver good results, but they require obtaining training on labeled data. Here are some examples of companies that use sentiment analysis: AlchemyAPI, based in Denver, is a really cool company that provides resources to do sentiment analysis for an entity on a document or webpage. The Stock Sonar uses sentiment analysis of unstructured text to determine whether online press is being positive or negative towards businesses by identifying lexical sentiment as well as business events. About the Author Janu Verma is a Quantitative Researcher at the Buckler Lab, Cornell University, where he works on problems in bioinformatics and genomics. His background is in mathematics and machine learning and leverages tools from these areas to answer questions in biology. Janu holds a Masters in Theoretical Physics from University of Cambridge in UK, and dropped out from mathematics PhD program (after 3 years) at Kansas State University. He also writes about data science, machine learning and mathematics at Random Inferences. Until Sunday 24th January you can save 50% on our leading Machine Learning titles as we celebrate Machine Learning week. From Python to Spark, and from R to Java, we've got a range of tools and languages covered so you can explore Machine Learning from a range of different perspectives. You can also pick up a free Machine Learning eBook every day this week from our Free Learning page – don’t miss out!
Read more
  • 0
  • 0
  • 4978

article-image-osgi-life-cycle
Packt
07 Mar 2013
5 min read
Save for later

OSGi life cycle

Packt
07 Mar 2013
5 min read
(For more resources related to this topic, see here.) OSGi applications are described as living entities; by this we mean that these applications appear to evolve as the lifecycles of their constituent bundles are lived. The lifecycle layer facilitates this functionality. OSGi bundles are dynamically installed, resolved, started, updated, stopped, and uninstalled. The framework enforces the transitions between states, one cannot directly install a bundle and jump to an Active state without first passing through the resolved and starting states. The transitions between each state are illustrated in the following figure: Installed Bundles came into existence in an OSGi framework in the installed state. A bundle in this state cannot be immediately started, as the preceding diagram depicts that there is no direct transition from the installed state to the starting state. An installed bundle is also not active. There are three possible transitions: the bundle may become resolved, uninstalled, or refreshed. Apache Karaf command To install a bundle in Karaf, issue the osgi:install (bundle:install on Karaf 3.x) command, as follows: karaf@root> osgi:install URLs Having a bundle installed to the OSGi framework does not mean it is ready to be used; next we must resolve its dependencies. Resolved Entering the resolved state requires the framework to ensure that all the dependencies of a bundle have been met. Upon having its dependencies ensured, the bundle is now a candidate to be transitioned to the starting state. A resolved bundle may be refreshed, transitioning the bundle back to the installed state. A resolved bundle may also be transitioned to the uninstalled state. A resolved bundle is not active; however, it is ready to be activated. Apache Karaf command To resolve an installed bundle in Karaf, issue the osgi:resolve (bundle:resolve on Karaf 3.x) command, as follows: karaf@root> osgi:resolve BundleID Starting A resolved bundle may be started. The starting state is transitory; the framework is initializing the resolved bundle into a running active state. In fact, the transition from the starting to active state is implicit. Apache Karaf command To start a resolved bundle in Karaf, issue the osgi:start (bundle:start on Karaf 3.x) command, as follows: karaf@root> osgi:start BundleID Active The bundle is fully resolved, providing and consuming services in the OSGi environment. To perform any more transitions on an active bundle, it must first be stopped. Updating Bundle updates occur when the framework is instructed to re-evaluate a bundle's dependencies; this action is synonymous with refreshing a bundle. When this action occurs, all of the wiring to and from the bundle is broken, so care must be taken before refreshing to avoid starting a bundle storm (one bundle refreshing causes a domino effect of other bundles refreshing). Apache Karaf command To update a bundle in Karaf, issue the osgi:update (bundle:update on Karaf 3.x) command, as follows: karaf@root> osgi:update BundleID [location] The location option allows you to update the bundle via its predefined updated location or to specify a new location to find bundle updates. Stopping Stopping a bundle transitions it from the active to the resolved state. The bundle can be restarted while it remains in the resolved state. Apache Karaf command To stop an active bundle in Karaf, issue the osgi:stop (bundle:stop on Karaf 3.x) command, as follows: karaf@root> osgi:stop BundleID Uninstalled Uninstalling a bundle transitions an installed or resolved bundle out of the OSGi environment; however, the bundle is not removed from the environment! Why is this? While the bundle is no longer available for use, references to the bundle may still exist and used for introspection. To help leverage these states in your bundles, the OSGi specification provides a hook into your bundle state via the Activator interface. Apache Karaf command To uninstall a bundle in Karaf, issue the osgi:uninstall (bundle:uninstall on Karaf 3.x) command, as follows: karaf@root> osgi:uninstall BundleID BundleActivator bundle may optionally declare an Activator class implementing the org.osgi.framework.BundleActivator interface. This class must be referenced in the bundle manifest file via the BundleActivator header. Implementing the activator allows the bundle developer to specify actions to be performed upon starting or stopping a bundle. Generally, such operations include gaining access to or freeing resources, and registering and unregistering services. The entry in manifest.mf will appear as follows: Bundle-Activator: com.packt.osgi.starter.sample.Activator When building with maven-bundle-plugin, the following configuration instruction is added: <Bundle-Activator> com.packt.osgi.starter.sample.Activator </Bundle-Activator> The process can be seen in the following screenshot: Summary: This article covered the various states involved in the OSGi lifecycle. We also learnt about the transitions from one state to another. Resources for Article : Further resources on this subject: Koha's Web Installer, Crontab, and Other Server Configurations [Article] Using the OSGi Bundle Repository in OSGi and Apache Felix 3.0 [Article] Getting Started with Bookshelf Project in Apache Felix [Article]
Read more
  • 0
  • 0
  • 4975

Packt
16 Oct 2009
6 min read
Save for later

Network Configuration—IPv6 with FreeBSD

Packt
16 Oct 2009
6 min read
Several methods were introduced to reduce the usage of IP addresses in the internet including: Classless Interdomain Routing (CIDR): This introduced the death of classful addressing (for example Class A, B, C) by a new subnetting method which is not limited, unlike the classful method. Network Address Translation (NAT): Using NAT you do not need to use public IP addresses on your internal hosts. Using CIDR subnets and NAT only helped IPv4 to live a few years longer, but was not the ultimate cure to the problem. Besides the addressing issues, there were other problems with IPv4 which could not be easily solved. These issues include the following: The size of internet routing tables was growing rapidly and this forced backbone providers to upgrade their networking gears. The IPv4 was very inefficient for high throughput links and did not support QoS by nature. Back in the early 90s, IETF had started a workgroup to solve the deficiencies of the IP protocol. In 1995, the IETF published the initial drafts of IPv6 as the next generation IP. Since then, the protocol has matured enormously and been implemented in many operating systems. IPv6 Facts If you are not familiar with IPv6, here is a very quick look at the difference between IPv4 and IPv6. (For a more detailed insight into IPv6 and its configuration in various operating systems, it is recommended that you read Running IPv6 book by Iljitsch van Beijnum). Fact One—Addressing Addressing in IPv6 is quite different from legacy IPv4 addresses. IPv6 uses 128-bit address space unlike the 32-bit addressing system in IPv4. A typical IPv6 address would look like—2002:a00:1:5353:20a:95ff:fef5:246e Fact Two—Address Types There are 4 types of addresses in IPv6: Unicast: A typical IPv6 address you use on a host. Multicast: Addresses that start with ff:: are equivalent to IPv4 multicast. Anycast: A typical IPv6 address that is used on a router. Reserved: Includes loopback, link-local, site-local, and so on. Fact Three—ARP There is no ARP! MAC to IP mapping is no longer needed as MAC addresses are embedded into IPv6 addresses. Instead, ND is born. ND is used to auto-configure addresses on hosts, duplicated detection, and so on. Fact Four—Interface Configuration If you are new to IPv6, you will be shocked to see an IPv6 address, telling yourself that you are in trouble assigning addresses to interfaces or remembering the addresses. However, it is not all that hard. In most cases, you can have your host autoconfigure IPv6 address on its interfaces. Typically, you should set this up only on your network gateway (router) manually. Using IPv6 Running FreeBSD 7, the kernel is already IPv6 enabled. However, you should manually enable IPv6 in the UserLand, by adding the following line to the /etc/rc.conf configuration file: ipv6_enable="YES" And manually start the appropriate rc script (or reboot the system) for the changes to take effect: # /etc/rc.d/network_ipv6 start This will enable IPv6 on all interfaces that are IPv6 capable. This behavior is changed by modifying the following variable in the /etc/rc.conf file: ipv6_network_interfaces="fxp0 bge0" This will enable IPv6 support on specified interfaces. The default value for this variable is auto. Once you enable IPv6, interfaces will discover the IPv6 enabled routers on the network and build their own IPv6 addresses based on the network prefix they receive from the router. Configuring Interfaces In a typical scenario, IPv6 network stack will automatically look for an IPv6 enabled router on the same network for each interface and try to automatically configure the IPv6 address on the interface. The following is an example of an automatically configured interface(replace the $ with %): # ifconfig ed0 ed0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 00:1c:42:8d:5d:bf inet6 fe80::21c:42ff:fe8d:5dbf$ed0 prefixlen 64 scopeid 0x1 inet 192.168.0.225 netmask 0xffffff00 broadcast 192.168.0.255 inet6 2a01:3c8::21c:42ff:fe8d:5dbf prefixlen 64 autoconf media: Ethernet autoselect (10baseT/UTP) Beside the IPv4 address, there are two IPv6 addresses on the interface. One address begins with fe80:: and identified with the scopeid 0x1 tag, which is called a link-local address. Another address begins with 2a01:3c8::, which is the unicast address of this interface. The unicast address prefix is obtained from the IPv6 router on the network. The whole address is created using the 64 bits Extended Unique Identifier (EUI-64) algorithm, which consists of the hosts MAC address with some minor modifications. The link-local address (that is from the reserved address pool) always starts with fe80:: and is used for local network usage. This can be compared with RFC1819 private addresses that are suitable for local use. The network stack will automatically assign a link-local address to each IPv6 enabled interface, regardless whether an IPv6 router is discovered on the network. This means that in a scenario of a home network or a lab network, you do not need to run an IPv6 router or have a valid IPv6 prefix in order to establish an IPv6 network. All the hosts will be automatically provisioned with a link-local address, so they can exchange IPv6 traffic. The network discovery protocol (NDP) helps the host find the router on the network and then create a unicast address for the interface. NDP is known as the equivalent to ARP protocol in IPv6. The ndp(8) utility is used to control the behavior of this protocol: # ndp -a Neighbor Linklayer Address Netif Expire S Flags 2a01:3c8:: 0:16:cb:98:d4:bf ed0 20s R R 2a01:3c8::21c:42ff:fe8d:5dbf 0:1c:42:8d:5d:bf ed0 permanent R fe80::216:cbff:fe98:d4bf$ed0 0:16:cb:98:d4:bf ed0 23h58m48s S R fe80::21c:42ff:fe8d:5dbf$ed0 0:1c:42:8d:5d:bf ed0 permanent R fe80::1%lo0 (incomplete) lo0 permanent R The above example shows the discovered IPv6 hosts(replace the $ with %). The ed0 interface is connected to an IPv6 enabled network and receives a valid prefix via a router (the first entry of the list). The second entry is the unicast address of the ed0. The third and the fourth entries are link-local address for the router and our host. And the last entry belongs to the local host. As you have seen so far, there are some special (reserved) IPv6 addresses. The following table shows a list of reserved addresses:   Address Name Description :: Unspecified Equivalent to 0.0.0.0 in Pv4 ::1 Loopback address Equivalent to 127.0.0.1 in IPv4 fe80:: Link-local fec0:: Site-local ff00:: Multicast   In case you want to configure the static IPv6 address on an interface, it can be done as in a typical IPv4 scenario: # ifconfig vr0 inet6 2a01:3c8::21c:42ff:dead:beef prefixlen 64 This will manually configure an IP address on the specified interface. Note the prefixlen keyword that is equivalent to subnet mask in IPv4.
Read more
  • 0
  • 0
  • 4973

article-image-improvising-your-infrastructure-as-code-using-ai-tools
Russ McKendrick
11 Jun 2023
6 min read
Save for later

Improvising your Infrastructure as Code using AI-tools

Russ McKendrick
11 Jun 2023
6 min read
In Infrastructure as Code for Beginners, I briefly discussed the role of AI tools such as ChatGPT. Since then, new features have emerged, making this the perfect time to revisit the topic and evaluate whether my initial recommendations have evolved. Let's quickly recap: GPT-powered chat services, including ChatGPT, certainly have their place in Infrastructure as Code (IaC) projects. However, we are not quite at the point where we can ask these tools to create our projects from scratch. The primary reason for this is straightforward: GPT, while impressive, can make mistakes. This is fine, except that when GPT errors, it often does so by inventing information, which it then presents in a highly convincing manner. On the one hand, you have Infrastructure as Code which has been taking away the heavy lifting of managing infrastructure for quite a while; a lot of the tooling is well established even though it is still considered a fast-moving technology to most.On the contrary,  artificial intelligence tools such as ChatGPT show up and quite dramatically change what we thought was possible even just a year ago, impacting how we create, test, and maintain code. Here are just some of the use cases I have been using the two together for in the last few months.●      Code Quality: You can have ChatGPT check your code and make recommendations,●      Cutting Down on Human Error: ChatGPT can double-check and make sure you are not doing something dumb if you ask it to.●      Boosting Productivity: You can ask ChatGTP to add inline comments to your code, update variables, and handle the boring stuff. Remember, we're just scratching the surface here. The combo of AI and IaC is still fresh, and there's a lot of territory left to explore.Improvising your Terraform code with ChatGPTRather than talk anymore, let’s take a look at implementing of the use cases mentioned above, to start with let’s have ChatGPT review an example Terraform block and make some recommendations on how it could be improved.Here is the prompt that we can use: I have the following Terraform code; it creates a Storage Account in Microsoft Azure; resource "azurerm_storage_account" "example" {name                     = "saiacforbeg2022111534"resource_group_name      = azurerm_resource_group.example.namelocation                 = azurerm_resource_group.example.locationaccount_tier             = "Standard"account_replication_type = "GRS"} Could you check it and provide recommendations on how it could be improved? The prompt above made the following recommendations, all of which are valid observations: Image 1: Recommendations that ChatGPT gave me.As well as rewriting the code considering its recommendations, it also recognized that we are deploying an Azure Storage Account and pointed out the following: Additionally, please remember that the storage account names in Azure need to be globally unique and must be between 3 and 24 characters in length, and can include numbers and lowercase letters only. Thus, you might want to consider adding checks to ensure that these constraints are met. Now let’s ask ChatGPT to add some inline comments which explain what is going to the main block of code using the following prompt :Please add inline comments explaining what is happening to the code below … ChatGPT gave the following response:Image 2: Fully commented codeFinally, we can ask ChatGTP to check and make some suggestions about how we could better secure our Storage Account using the following prompt:Image 3:  list of recommendations for improving our Terraform code.Let’s see if ChatGPT could update the code with some of those suggestions with the following prompt and also have a look at the result: Image 4:  recommendations within the context of the original Terraform code. GPT vs Bing vs BardHow do other AI Chatbots handle our original prompt, let’s start with Bing; Image 5: Bing Chat had to say about our Terraform code.Well, that wasn’t helpful at all - let’s now see if Google Bard fares any better:Image 6: Google Bard had to say about our Terraform code.It didn’t just give us a thumbs up, which is a good start. It made similar recommendations to ChatGPT. However, while Bard made some suggestions, they were not as detailed, and it didn’t point out anything outside of what we asked it to, unlike ChatGPT, which recognized we were creating a storage account and gave us some additional pointers.Now don’t write Microsoft off just yet. While Bing didn’t blow me away with its initial responses, Microsoft’s various Copilot services, all of which are launching soon, all of which are GPT4 based, I expect them to be up to the same level of responsiveness of ChatGTP and in the case of GitHub Copilot X I expect it to potentially surpass ChatGTP in terms of usefulness and convenience as it is going to be built directly into your IDE and have all of GitHub to learn from.Given the rate of change, which feels like new tools and features and power is being added every other day now - the next 6 to 12 months is going to be an exciting time for anyone writing Infrastructure as Code as the tools we are using day-to-day start get more direct access to the AI tools we have been discussing.SummaryIn conclusion, this article delved into the process of improving code quality through various means, including seeking recommendations, incorporating inline code comments, and leveraging the capabilities of ChatGPT, Bing, and Google Bard. The demonstration emphasized the importance of continuously enhancing code and actively involving AI-powered tools to identify areas for improvement. Additionally, the article explored how ChatGPT's insights and suggestions were utilized to enhance storage security through stronger and more reliable code implementation. By employing these techniques, developers can achieve code optimization and bolster the overall security of their storage systems.Author BioRuss McKendrick is an experienced DevOps practitioner and system administrator with a passion for automation and containers. He has been working in IT and related industries for the better part of 30 years. During his career, he has had responsibilities in many different sectors, including first-line, second-line, and senior support in client-facing and internal teams for small and large organizations.He works almost exclusively with Linux, using open-source systems and tools across dedicated hardware and virtual machines hosted in public and private clouds at Node4, where he holds the title of practice manager (SRE and DevOps). He also buys way too many records!Author of the book: Infrastructure as Code for Beginners
Read more
  • 0
  • 0
  • 4973
article-image-software-documentation-trac
Packt
21 Oct 2009
4 min read
Save for later

Software Documentation with Trac

Packt
21 Oct 2009
4 min read
Documentation—if there is one word that installs fear in most developers, it must be this one. No one in their right mind would argue the value of documentation, but it is the actual act of writing it that concerns developers so. The secret of creating good documentation is to make the process of doing so as painless as possible, and if we are lucky maybe even attractive, to the developers. The only practical way to achieve that is to reduce friction. The last thing we need when we are in middle of fixing a bug is to wrestle with our word processor, or even worse try to find the right document to update. What's in a name?Throughout the rest of this article, we will refer to various URLs that point to specific areas of our Trac environment, Subversion repository, or WebDAV folders. Whenever you see servername, replace it with your own server name. Making Documentation Easy One of the reasons Trac works so well for managing software development is because it is browser based. Apart from our development environment, the browser, along with our email client, are the next most likely applications we are going to have installed and running on our computer. If access to our Trac environment is only a click away, it stands to reason that we are more likely to use it. We can refer to Trac as a "wiki on steroids" because of the way the developers have integrated the typical features of a wiki throughout the whole product. However, for all the extra features and integration, at its heart Trac is basically just a wiki and this is the main reason why it so useful in helping smooth the documentation process. A wiki is a web application that allows visitors to create and modify its content. Let's expand on that slightly. As well as letting us view content—like a normal website—a wiki lets us create or edit the content as we desire. This could take the form of creating new content, or simply touching up the spelling on something that already exists. While the general idea with a wiki is that anyone can edit them, in practice this can lead to abuse, vandalism, or spam. The obvious solution to this is to involve people to authenticate the edit. Do we really need this security?Yes. Having these security requirements provides us with accountability. We will always be able to see when something is done, but by enforcing security we can see who did it. While this does cause some administrative overhead to create and maintain authentication details for anyone involved with our development projects, the benefits outweigh the costs. Accessing Trac Before we look at how to modify and create pages, let's see how our Trac environment looks to a normal (i.e. unauthenticated) user. To do this we need to open our web browser and enter the URL http://servername/projects/sandbox into the address bar and then press the Enter key. This will take us to the default page (which is actually called WikiStart). When we access our project as an unauthenticated (or anonymous in Trac parlance) user, the majority of it will look and act like a normal website and the wiki in particular seems just like the usual collection of interlinked pages. However, as soon as we authenticate ourselves to Apache (which passes that information on to Trac), it all changes. If we click the Login link in the top right of the page now, we will be presented with our browser's usual authentication dialog box as shown in the following screenshot. Input the proper username and password and click OK. If we enter them correctly we will be taken back to the same page, but this time there will be two differences. Firstly, instead of the login link we will see the text logged in as followed by the username we used and a Logout link. Secondly, if we scroll to the bottom of the page there are some buttons that allow us to modify the page in various ways. Anonymous users have permission to only view wiki pages, while authenticated users have full control. We should try that out now—click the Logout link and scroll down again, and you will see that the buttons are absent.
Read more
  • 0
  • 0
  • 4971

article-image-using-prezi-online-presentation-software-tool
Packt
03 Jul 2013
6 min read
Save for later

Using Prezi - The Online Presentation Software Tool

Packt
03 Jul 2013
6 min read
(For more resources related to this topic, see here.) So, what is Prezi? Prezi is a web-based presentation tool. It has awesome features you can use to create compelling, attention-grabbing, and memorable presentations. We will get to the cool features later. First, let me talk about it being a web-based presentation tool. By "web-based," I mean that you create your presentations online using this application program. You create presentations online, present it from the web, and have a link through which your audience can access it later. You can also use Prezi as a collaboration tool by sharing your Prezi and allowing members of your group to edit the presentation with you in real time. During a Prezi meeting, up to 10 members of your group can make changes on a Prezi as you brainstorm your ideas or discuss your points. So, while it is largely a presentation tool, Prezi can also function as a tool for brainstorming and discussion. This is possible because of the way Prezi has been laid out and the various flowcharts you can build to illustrate your points. In this sense, Prezi also functions as a mind mapping tool. So, Prezi is an online presentation tool that you can also use for brainstorming, mind mapping, and real-time collaboration. It comes with features that you can harness to create highly visual and engaging presentations. Now, there are several presentation tools online, but Prezi is different. Working with most slideshow presentations is like working with a stack of index cards or poster boards. We add text and illustrations on each card, then we present our cards one at a time. First you view the first slide, then the second, and so on, as shown in the following diagram: In slideshow presentations, information is presented one slide at a time Similar to slideshow presentation tools, you use Prezi to present your ideas by adding text and illustrations on a given space. What makes Prezi awesome, however, are the panning and zooming features you should use to make a greater impact on your audience. When it comes to making a presentation using Prezi, imagine yourself in front of a huge white board the size of a wall. You begin by typing text and illustrations on this board. In Prezi, this is referred to as the canvas. It is like having an exhibit, but you would want to guide your viewers' eyes through the texts and illustrations you have placed on your canvas. Which one should they look at first? Next? So, you add numbers to each item. This will be the path your viewers' eyes would take as they travel from one element to another, as shown in the following diagram: In Prezi, you guide your viewer's eyes by adding a numbered path Now imagine you have a video camera. Imagine viewing your whiteboard and focusing on the first item, then moving the video (without turning it off) to view the second item. In Prezi, a similar panning movement occurs as you move from one frame to the next following the path you have indicated, as shown in the following diagram: A Prezi presentation is like viewing your presentation through a video camera, as it pans across the canvas andzooming into focus points at certain times This sense of movement created by panning makes the presentation eye-catching. The motion produced helps to keep your viewer's attention as you move from one item on your presentation to the next. Imagine if, given the preceding diagram, we pan 1-2-4-3 instead of 1-2-3-4. Can you imagine how different that would be? Or how about if we were to place the elements in a straight row? Will it be as attractive? Think of the many different ways you can be creative with this Prezi feature! Installation Prezi is a web-based tool, and it is best to work on it online. You can get started in four easy steps as follows: Go to http://prezi.com/. Click on Sign Up Now located in the middle of the screen, or Sign Up located in the upper right-hand corner. Choose one of the following options for a Prezi license: Public: This is free. You will have 100 MB of storage, which is enough for a few presentations. You can edit and share the Prezi presentations you create. Your presentations will be public. Enjoy: For a fee, you will have 500 MB of storage. On top of that, you have the option to make your presentations private and use your own logo. Subscribers also get premium customer support where you can get answers within one day. Pro: If you upgrade to a Pro license, you'll get 2 GB of storage. In addition to having the option to have private presentations and using your own logo, you will also be able to download Prezi Desktop. You use Prezi Desktop to work offline securely. Student and Teacher Licenses: You can register for an Enjoy Edu license for free if your school has an official website. Use a school e-mail account when signing up. You can create private presentations, use your own logo, and have 500 MB of storage space. If you want the ability to work offline or need more space, you can upgrade to a Pro Edu license for a fee. Decide which option is most appropriate for you and then click on the button that says Try Now. Follow the prompts for signing up. You will be asked for your name, e-mail address, and a password. It may be a good idea for you to write down the e-mail and password you used before you forget. You should also check the box signifying that you agree to Prezi's terms of use. If you have a Facebook account, you might like to log in using your Facebook account information. Just click on Log in with Facebook. Once you have signed up, you can log in and use Prezi any time you wish by going to http://prezi.com/ and putting in your e-mail address and password. And that's it! Now you're ready to create your first Prezi! Summary In this article we saw what is Prezi, its uses, and installation. Resources for Article : Further resources on this subject: Turning your PowerPoint presentation into a Prezi [Article] Getting Started with Impressive Presentations [Article] Implementation of SASS [Article]
Read more
  • 0
  • 0
  • 4969

Packt
16 Jun 2017
9 min read
Save for later

Streaming and the Actor Model – Akka Streams!

Packt
16 Jun 2017
9 min read
In this article by Piyush Mishra, author of the Akka Cookbook, we will learn about the streaming and the actor model with Akka streams. (For more resources related to this topic, see here.) Akka is a popular toolkit designed to ease the pain of dealing with concurrency and distributed systems. It provides easy APIs to create reactive, fault-tolerant, scalable, and concurrent applications, thanks to the actor model. The actor model was introduced by Carl Hewitt in the 70s, and it has been successfully implemented by different programming languages, frameworks, or toolkits, such as Erlang or Akka. The concepts around the actor model are simple. All actors are created inside an actor system. Every actor has a unique address within the actor system, a mailbox, a state (in the case of being a stateful actor) and a behavior. The only way of interacting with an actor is by sending messages to it using its address. Messages will be stored in the mailbox until the actor is ready to process them. Once it is ready, the actor will pick one message at a time and will execute its behavior against the message. At this point, the actor might update its state, create new actors, or send messages to other already-created actors. Akka provides all this and many other features, thanks to the vast ecosystem around the core component, such as Akka Cluster, Akka Cluster Sharding, Akka Persistence, Akka HTTP, or Akka Streams. We will dig a bit more into the later one. Streaming framework and toolkits are gaining momentum lately. This is motivated by the massive number of connected devices that are generating new data constantly that needs to be consumed, processed, analyzed, and stored. This is basically the idea of Internet of Things (IoT) or the newer term Internet of Everything. Some time ago, the Akka team decided that they could build a Streaming library leveraging all the power of Akka and the actor model: Akka Streams. Akka Streams uses Akka actors as its foundation to provide a set of easy APIs to create back-pressured streams. Each stream consists of one or more sources, zero or more flows, and one or more sinks. All these different modules are also known as stages in the Akka Streams terminology. The best way to understand how a stream works is to think about it as a graph. Each stage (source, flow, or sink) has zero or more input ports and zero or more output ports. For instance, a source has zero input ports and one output port. A flow has one input port and one output port. And finally, a sink has one input port and zero output ports. To have a runnable stream, we need to ensure that all ports of all our stages are connected. Only then, we can run our stream to process some elements: Akka Streams provides a rich set of predefined stages to cover the most common streaming functions. However, if a use case requires a new custom stage, it is also possible to create it from scratch or extend an existing one. The full list of predefined stages can be found at http://doc.akka.io/docs/akka/current/scala/stream/stages-overview.html. Now that we know about the different components Akka Streams provides, it is a good moment to introduce the actor materializer. As we mentioned earlier, Akka is the foundation of Akka Streams. This means the code you define in the high-level API is eventually run inside an actor. The actor materializer is the entity responsible to create these low-level actors. By default, all processing stages get created within the same actor. This means only one element at a time can be processed by your stream. It is also possible to indicate that you want to have a different actor per stage, therefore having the possibility to process multiple messages at the same time. You can indicate this to the materializer by calling the async method in the proper stage. There are also asynchronous predefined stages. For performance reasons, Akka Streams batches messages when pushing them to the next stage to reduce overhead. After this quick introduction, let's start putting together some code to create and run a stream. We will use the Scala build tool (famously known as sbt) to retrieve the Akka dependencies and run our code. To begin with, we need a build.sbt file with the following content: name := "akka-async-streams" version := "1.0" scalaVersion := "2.11.7" libraryDependencies += "com.typesafe.akka" % "akka-actor_2.11" % "2.4.17" libraryDependencies += "com.typesafe.akka" % "akka-stream_2.11" % "2.4.17" Once we have the file ready, we need to run sbt update to let sbt fetch the required dependencies. Our first stream will push a list of words, capitalize each of them, and log the resulting values. This can easily be achieved by doing the following: implicit val actorSystem = ActorSystem() implicit val actorMaterializer = ActorMaterializer() val stream = Source(List("hello","from","akka","streams!")) .map(_.capitalize) .to(Sink.foreach(actorSystem.log.info)) stream.run() In this small code snippet, we can see how our stream has one source with a list of strings, one flow that is capitalizing each stream, and finally one sink logging the result. If we run our code, we should see the following in the output: [INFO] [default-akka.actor.default-dispatcher-3] [akka.actor.ActorSystemImpl(default)] Hello [INFO] [default-akka.actor.default-dispatcher-3] [akka.actor.ActorSystemImpl(default)] From [INFO] [default-akka.actor.default-dispatcher-3] [akka.actor.ActorSystemImpl(default)] Akka [INFO] [default-akka.actor.default-dispatcher-3] [akka.actor.ActorSystemImpl(default)] Streams! The execution of this stream is happening synchronously and ordered. In our next example, we will do the same stream; however, we can see how all stages are modular: implicit val actorSystem = ActorSystem() implicit val actorMaterializer = ActorMaterializer() val source = Source(List("hello","from","akka","streams!")) val sink = Sink.foreach(actorSystem.log.info) val capitalizer = Flow[String].map(_.capitalize) val stream = source.via(capitalizer).to(sink) stream.run() In this code snippet, we can see how stages can be treated as immutable modules. We see that we can use the via helper method to provide a flow stage in a stream. This stream is still running synchronously. To run it asynchronously, we can take advantage of the mapAsync flow. For this, let's create a small actor that will do the capitalization for us: class Capitalizer extends Actor with ActorLogging { def receive = { case str : String => log.info(s"Capitalizing $str") sender ! str.capitalize } } Once we have our actor defined, we can set up our asynchronous stream. For this, we will create a round robin pool of capitalizer actors. Then, we will use the ask pattern to send a message to an actor and wait for a response. This happens using the operator? The stream definition will be something like this: implicit val actorSystem = ActorSystem() implicit val actorMaterializer = ActorMaterializer() implicit val askTimeout = Timeout(5 seconds) val capitalizer = actorSystem.actorOf(Props[Capitalizer].withRouter(RoundRobinPool(10))) val source = Source(List("hello","from","akka","streams!")) val sink = Sink.foreach(actorSystem.log.info) val flow = Flow[String].mapAsync(parallelism = 5)(elem => (capitalizer ? elem).mapTo[String]) val stream = source.via(flow).to(sink) stream.run() If we execute this small piece of code, we can see something similar: [INFO] [default-akka.actor.default-dispatcher-16] [akka://default/user/$a/$a] Capitalizing hello [INFO] [default-akka.actor.default-dispatcher-15] [akka://default/user/$a/$b] Capitalizing from [INFO] [default-akka.actor.default-dispatcher-6] [akka://default/user/$a/$c] Capitalizing akka [INFO] [default-akka.actor.default-dispatcher-14] [akka://default/user/$a/$d] Capitalizing streams! [INFO] [default-akka.actor.default-dispatcher-14] [akka.actor.ActorSystemImpl(default)] Hello [INFO] [default-akka.actor.default-dispatcher-14] [akka.actor.ActorSystemImpl(default)] From [INFO] [default-akka.actor.default-dispatcher-14] [akka.actor.ActorSystemImpl(default)] Akka [INFO] [default-akka.actor.default-dispatcher-14] [akka.actor.ActorSystemImpl(default)] Streams! We can see how each word is being processed by a different capitalizer actor ($a/$b/$c/$d) and by different threads (default-dispatcher 16,15,6 and 14). Even if these executions are happening asynchronously in the pool of actors, the stream is still maintaining the order of the elements. If we do not need to maintain order and we are looking for a faster approach, where an element can be pushed to the next stage in the stream as soon as it is ready, we can use mapAsyncUnordered: implicit val actorSystem = ActorSystem() implicit val actorMaterializer = ActorMaterializer() implicit val askTimeout = Timeout(5 seconds) val capitalizer = actorSystem.actorOf(Props[Capitalizer].withRouter(RoundRobinPool(10))) val source = Source(List("hello","from","akka","streams!")) val sink = Sink.foreach(actorSystem.log.info) val flow = Flow[String].mapAsyncUnordered(parallelism = 5)(elem => (capitalizer ? elem).mapTo[String]) val stream = source.via(flow).to(sink) stream.run() When running this code, we can see that the order is not preserved and the capitalized words arrive to the sink differently every time we execute our code. Consider the following example: [INFO] [default-akka.actor.default-dispatcher-10] [akka://default/user/$a/$b] Capitalizing from [INFO] [default-akka.actor.default-dispatcher-4] [akka://default/user/$a/$d] Capitalizing streams! [INFO] [default-akka.actor.default-dispatcher-13] [akka://default/user/$a/$c] Capitalizing akka [INFO] [default-akka.actor.default-dispatcher-14] [akka://default/user/$a/$a] Capitalizing hello [INFO] [default-akka.actor.default-dispatcher-12] [akka.actor.ActorSystemImpl(default)] Akka [INFO] [default-akka.actor.default-dispatcher-12] [akka.actor.ActorSystemImpl(default)] From [INFO] [default-akka.actor.default-dispatcher-12] [akka.actor.ActorSystemImpl(default)] Hello [INFO] [default-akka.actor.default-dispatcher-12] [akka.actor.ActorSystemImpl(default)] Streams! Akka Streams also provides a graph DSL to define your stream. In this DSL, it is possible to connect stages just using the ~> operator: implicit val actorSystem = ActorSystem() implicit val actorMaterializer = ActorMaterializer() implicit val askTimeout = Timeout(5 seconds) val capitalizer = actorSystem.actorOf(Props[Capitalizer].withRouter(RoundRobinPool(10))) val graph = RunnableGraph.fromGraph(GraphDSL.create() { implicit b => import GraphDSL.Implicits._ val source = Source(List("hello","from","akka","streams!")) val sink = Sink.foreach(actorSystem.log.info) val flow = Flow[String].mapAsyncUnordered(parallelism = 5)(elem => (capitalizer ? elem).mapTo[String]) source ~> flow ~> sink ClosedShape }) graph.run() These code snippets show only a few features of the vast available options inside the Akka Streams framework. Actors can be seamlessly integrated with streams. This brings a whole new set of possibilities to process things in a stream fashion. We have seen how we can preserve or avoid order of elements, either synchronously or asynchronously. In addition, we saw how to use the graph DSL to define our stream. Summary In this article, we covered the concept of the actor model and the core components of Akka. We also described the stages in Akka Streams and created an example code for stream. If you want to learn more about Akka, Akka Streams, and all other modules around them, you can find useful and handy recipes like these ones in the Akka Cookbook at https://www.packtpub.com/application-development/akka-cookbook.  Resources for Article: Further resources on this subject: Creating First Akka Application [article] Working with Entities in Google Web Toolkit 2 [article] Skinner's Toolkit for Plone 3 Theming (Part 1) [article]
Read more
  • 1
  • 0
  • 4966
article-image-megaman-clone-unity-part-2
Travis and
10 Jun 2015
6 min read
Save for later

Megaman Clone with Unity Part 2

Travis and
10 Jun 2015
6 min read
Creating a Weapon You will remember that back in Part 1 we made our simple MegaMan clone. Let's take this project further. So, first off let's create a weapon. Now, we're not going into minute details like creating an actual weapon for our hero, but let's create a bullet, or else it's going to be hard to shoot enemies! Create a sphere called "Bullet," and change all of its scale values to 0.2, attaching a new material to it that will be yellow. Make sure afterwards to make the Bullet a Prefab by dragging it into the Project Assets folder. Lastly, remove the sphere collider, and add a circle collider 2D to it. Now that we have our bullet, let's create a new script called "Weapon", and attach it to our Player Object. We'll also create another new script called "Bullet" and attach it to our Bullet Prefab. In fact we didn't do it in the last post, but let's actually make the Player object a prefab. Now first, open the PlayerMovement script and make a quick edit. Ok, so we have created a new enum state called Direction, and an associated property called playerDirection, that is going to keep track of what way our player is currently facing. We also created a property, because nothing else but our PlayerMovement script should change our players direction. Also this stops it from appearing in the inspector, which if it was there, could eventually start cluttering things if our designers are not really supposed to be touching that. Lastly, in our MovePlayer method called every update, we add a simple if statement to keep track of what way our player moved last. Note that it is not affected at 0, this is because we want to know the last direction moving, so if our player is at a standstill, we still want to shoot the previous way clicked. Alright, let's open our Bullet.cs script and quickly make some edits to it! So we now have our bullet that will move in a direction based on its own direction state. All we need is a part to manage all of these interactions. This will be our weapon script, so let's open that now! Ok, we have what is essentially a manager of these two together. This one will wait for a user's input, create the bullet, and then set its direction depending on the players current direction. We use the Fire1 button so that this can be changed later in the Input manager and work on other controllers easily. Now, we do want to point out something with our connection between the playerMovement class and the bulletDirection class. First, we have a very tight coupling on these classes, which isn't great, but for the continuation of this post, we're going to skip it. But if you to wish know more about this we suggest researching delegates and events, as well as decoupling in Unity. For now though, this will do. Creating an Enemy Next let’s create an enemy for this bullet to interact with. So let's create a cube, make him red with a material, and then give him the tag "Enemy" as well as the name "Enemy". Take off the box collider, and attach a box collider 2D, as well as a rigidbody2D. Lastly, make this enemy a prefab. It should look like the following in the Inspector. Now to make sure our player and bullet don't bump each other anymore, let's quickly take that out of the physicsManager. First, create three layers, "Bullet", "Player", and "Enemy". Each of these three game objects should be put on their respective layers. Now in the PhysicsManager under Edit _> Project Settings -> Physics 2D, make sure that the player and bullet classes are NOT checked, so they no longer respond to each other. Okay, now let's create an "Enemy" script and attach it to the Enemy game object. In here, we have a very simple script that just contains a health int, and a method to adjust the health of our enemy. Realistically our player class should have a very similar set up, but for the sake of scope, we can just do this for our enemy. Also, when our enemy class takes enough damage, we destroy that game object. Now we're going to have to change our Bullet script as well to know what to do with this class. We've added a couple of things. First, we now have a damage int at the top of our class that is used to measure the damage this bullet will do to our enemy. We could for example, hold down the shoot button, which increases the damage of our bullet. For this, we'll just keep it at a base amount. Next, we add the OnCollisionEnter2D method, which is going to handle what to do if our bullet interacts with an enemy. If the collided with object is an enemy, our bullet will call the Damage method in the enemy class, and then destroy itself afterwards. In honesty, we could actually put that destroy outside the if statement so that no matter what the bullet hit it would destroy itself. So now if we try our game we have an enemy in the game world who after two hits will actually die. Yes I know he's not really in any danger right now, but this is a great start for finding hittable targets! If this project continued, the next thing added should be a simple enemy movement script, some weapons perhaps for our enemies, and then some simple level design! For more Unity game development tutorials visit our dedicated Unity page here. About the Authors Denny is a Mobile Application Developer at Canadian Tire Development Operations. While working, Denny regularly uses Unity to create in-store experiences, but also works on other technologies like Famous, Phaser.IO, LibGDX, and CreateJS when creating game-like apps. He also enjoys making non-game mobile apps, but who cares about that, am I right? Travis is a Software Engineer, living in the bitter region of Winnipeg, Canada. His work and hobbies include Game Development with Unity or Phaser.IO, as well as Mobile App Development. He can enjoy a good video game or two, but only if he knows he'll win!
Read more
  • 0
  • 0
  • 4962

article-image-android-30-application-development-gps-locations-and-maps
Packt
22 Jul 2011
7 min read
Save for later

Android 3.0 Application Development: GPS, Locations, and Maps

Packt
22 Jul 2011
7 min read
  Android 3.0 Application Development Cookbook Design and develop rich smartphone and tablet applications for Android 3.0         Introduction For managing location based information, Android provides the android.location package which in turn gives us the LocationManager class that gives us access to location based functions such as the latitude and longitude of a device's position. Tracking a device over time is made equally convenient and the LocationListener class monitors changes in location as they occur. Listening for location changes is only a part of the story, as Google provides APIs for managing Google Maps data and displaying and manipulating maps through the use of the MapView and MapController classes. These powerful tools require us to sign up with Google first, and once done enable us to zoom in and out of maps, pan to any location that we are looking for, and when we want to, include application information on a map, and even add our own layers to maps and mark locations on a Google map. Detecting a device's location Android locations are expressed in terms of latitude and longitude coordinates. The default format is degrees. The Location object can also be used to store a time-stamp and other information such as speed and distance traveled. Although obtaining a device's last known location does not always yield the most accurate information, it is often the first reading that we may want. It is fast, simple to employ, and makes a good introduction to the LocationManager. <uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" /> How to do it... Use the TextView provided in the main.xml file and give it a resource ID: android:id="@+id/text_view" Declare a TextView as a class-wide field in the Java activity code: TextView textView; Then, find it in the usual way, from within the onCreate() method: textView = (TextView) findViewById(R.id.text_view); Next, and still within onCreate(), declare and define our LocationManager: LocationManager manager = (LocationManager) getSystemService(Context.LOCATION_SERVICE); Then, to retrieve the last known location using GPS and display this in the text view, add these lines: Location loc = manager.getLastKnownLocation(LocationManager.GPS_PROVIDER); textView.setText("latitude: " + loc.getLatitude() + "nlongitude: " + loc.getLongitude()); Run the code on a handset or emulator to obtain its location: How it works... The use of a LocationManager to obtain the device's last known location is very straightforward. As with other system services, we obtained it with getSystemService() and the getLastKnownLocation() method returns the Location object itself, which can be further queried to provide latitude and longitude coordinates. We could have done more with the Location object, for example Location.getAltitude() will return altitude and getDistance(Location) and getBearing(Location) will return distance and bearing to another Location. It is possible to send mock locations to an emulator using the DDMS perspective in Eclipse: Before sending location data this way, make sure that you have set the emulator to allow mock locations under Settings | Applications | Development. It is worth noting that although use of the getLastKnownLocation() method may not always be accurate, particularly if the device has been switched off for some time, it does have the advantage of yielding almost immediate results. There's more... Using GPS to obtain a location has a couple of drawbacks. Firstly, it does not work indoors; and secondly, it is very demanding on the battery. Location can be determined by comparing cell tower signal strengths, and although this method is not as accurate, it works well indoors and is much more considerate to the device's battery. Obtaining a location with a network provider The network provider is set up in exactly the same way as the previous GPS example, simply exchange the Location declaration with: Location loc = manager.getLastKnownLocation(LocationManager.NETWORK_PROVIDER); You will also need to change, or amend, the permission in the manifest file with: <uses-permission android_name="android.permission.ACCESS_COURSE_LOCATION" /> Listening for location changes Obtaining the last known location as we did in the previous recipe is all well and good and handy for retrieving a Location quickly, but it can be unreliable if the handset has been switched off or if the user is on the move. Ideally we want to be able to detect location changes as they happen and to do this we employ a LocationListener. In this recipe we will create a simple application that keeps track of a mobile device's movements. Getting ready This task can be performed most easily by starting where the previous one left off. If you have not completed that task yet, do so now—it is very short—then return here. If you have already completed the recipe then simply open it up to proceed. How to do it... First, move the declaration of our LocationManager so that it is a class-wide field: LocationManager manager; In the main Java activity code, before the TextView.setText() call, add the following three lines: LocationListener listener = new MyLocationListener(); manager.requestLocationUpdates(LocationManager.GPS_PROVIDER, 30000, 50, listener); Location location = manager.getLastKnownLocation(LocationManager.GPS_PROVIDER); Now create an inner class called MyLocationListener that implements LocationListener: LocationListener: public class MyLocationListener implements LocationListener { } Eclipse will most likely insist that you add some unimplemented methods and you should do so. For now, only complete one of them, the onLocationChanged() callback: @Override public void onLocationChanged(Location l) { textView.setText("/n/nlatitude: " + l.getLatitude() + "nlongitude: " + l.getLongitude()); } Leave the others as they are: @Override public void onProviderDisabled(String provider) {} @Override public void onProviderEnabled(String provider) {} @Override public void onStatusChanged(String provider, int status, Bundle extras) {} If you want to test this code on an emulator, then go right ahead. However, this code will create a serious drain on the battery of a handset, and it is wise to switch our listener off when it is not needed. Here we have used the activity's onPause() and onResume() functions to control this. You may wish to include these statements in any part of your activity's life cycle that suits your application's purpose: @Override protected void onResume() { super.onResume(); manager.requestLocationUpdates(LocationManager.GPS_PROVIDER, 30000, 50, listener); } @Override protected void onPause() { super.onPause(); manager.removeUpdates(this); } If you have not already tested this application, do so now. You will need to move around if you are testing it on a real device, or send mock locations to an emulator to see the code in action: How it works... In this recipe we used the LocationManager to provide location updates roughly every 30 seconds (30000 milliseconds) or whenever the location changed by more than 50 meters. We say 'roughly' because these values work only as a guide and the actual frequency of updates often varies from the values we set. Nevertheless, setting these two parameters of the requestLocationUpdates() method to high values can make a big difference to the amount of battery power the GPS provider consumes. Hopefully the use of the provider and the LocationListener as the other two parameters is self explanatory. The LocationListener operates very much as other listeners do and the purpose of the onProviderEnabled() and onProviderDisabled() should be clear. The onStatusChanged() method is called whenever a provider becomes unavailable after a period of availability or vice versa. The int, status can represent 0 = OUT_OF_SERVICE, 1 = TEMPORARILY_UNAVAILABLE, or 2 = AVAILABLE.  
Read more
  • 0
  • 0
  • 4961
Modal Close icon
Modal Close icon