Managing Multimedia and Unstructured Data in the Oracle Database: A revolutionary approach to understanding, managing, and delivering digital objects, assets, and all types of data

Throughout this book, most chapters will cover the usage of metadata. With unstructured data management, metadata is crucial. It is the data that describes the unstructured data and gives meaning to it. Each type of unstructured data object has its own metadata. It might be as simple as a filename, or as complex as a complete set of relational records. Without metadata the unstructured data loses meaning.

The metadata is primarily used for searching. Without it, it's not possible to construct a multimedia warehouse. It is also used for assigning a description. A person might see a photo of a plant. The metadata might have a description of what that photo is, giving meaning and context to the photo.

The metadata is also used to relate unstructured data objects, which in turn adds intelligence and structure to it. It is also used to store information about the object like its name, when it was created, who created it, and who modified it.

The metadata can be used to represent any knowledge about the unstructured object. It's typically stored in a structured format. Currently the trend is to use XML, but this has not always been the case. Additionally, metadata can be matched to data in relational databases or NoSQL databases.

As will be shown in the following chapters, the metadata usage can be rich, varied, and complex. At the moment because of limitations in computer technology, metadata is crucial for most systems that want to extensively use unstructured data. A computer if asked the question, find me the video with the picture of the person John in it, would have great difficulty answering it. Likewise, a question asking, find me all audio files with a lyre bird singing after sunset, would be equally hard to answer. By having a human operator attach metadata with this information in it, then while searching multimedia with that information, the questions raised can be answered.

Unfortunately, the need to manually attach metadata is a time consuming and costly exercise. A number of sites are investigating crowd sourcing to resolve it (see Chapter 3, The Multimedia Warehouse) or just bringing in a number of people to go through and identify the unstructured data.

As computer technology improves and new algorithms are discovered, the need to store metadata will disappear. Computers are already good at facial recognition and can convert speech to text. They do have major limitations and still struggle in complex situations that humans do easily. It is envisaged that in the next 20 years technology will improve to the point where algorithms will become commonplace that will be able to identify objects and people in a video or photo, and understand sounds and complex speech in audio files. When this point is reached, the need for metadata will be reduced and constrained to a smaller, more tightly controlled subset. The metadata will always exist and always be needed.

As the veil over the unstructured data is slowly removed, and as knowledge and understanding grows, so will the use of metadata. As covered in the previous point, the use will change and diminish over time, and the market for its use will grow. For example, if the current market represented 100 units, and if multimedia represented 30 percent that would be 30 units. If its usage over time dropped to 5 percent that would be 5 units. But if the growth of the market expanded to 10,000 units, 5% would be 500 units, which is five times bigger than the current market. So even though the need will be reduced, the market as it grows will demand an increasing usage for metadata.

The uses for metadata will start to strain relational databases, and object relational databases will be pushed to their limits to identify and handle the changing complexities of it. Time-based structures (effectively four-dimensional) will be needed. Oracle's flashback capabilities will need to be ramped up in data warehouses to handle large-scale, complex queries. The fuzzy data structures, which are needed to handle the vagaries of some multimedia types, struggle to be easily represented and queried against in most databases. Neural structures are another story altogether and most computer systems can't even cope with the basic handling of them. It's feasible in concept to attach a neural network as a metadata to an object type, which details how to recognize and handle components within it⁽⁷⁾.

In reviewing this book, an important question was raised. And that was, what is the best term to describe the concept of storing and delivering digital information? On investigation, a number of terms that closely fit the mark were discovered, though none truly described the concept that was trying to be expressed.

The following are a list of some of the terms discovered and reviewed, including definitions found on the Internet.

Each of the preceding definitions are correct, but the issue is that none truly conveys the meaning behind what it is to manage the unstructured data and deliver it. Each definition is restrictive and not adaptive to the changing digital technology. Most assume a digital image is a photo or document, and all assume they are owned. As will be shown further, these assumptions do not stand up on a closer scrutiny.

What did stand out was that most definitions conveyed the idea of representation, that is the digital information is meant to symbolize something, be it a photo, document, or video.

So which term should be used? After reviewing all terms the one that seems to have the most potential is a digital object. This is the term that will be used throughout most of the book. It is far easier to use an existing term that people are familiar with than it is to create a new one or define an acronym.

It is then important to accurately define what a digital object actually is. With technology changing, any classic definition we give today is likely to be out of date within a couple of years. The standard perception that the general public has of a digital object is a photograph taken by a digital camera. As will be explained later, a digital photograph is just a subset of type Picture. In fact, when looking at digital objects we are looking at ways of representing data, which is ultimately used by one of our traditional five senses.

When looking at the types of digital objects available they can be broken up as shown in the following table:

These are the traditional object types used throughout the world, but one needs to address the need of what types of images will exist in 50 years time. It is nearly impossible to predict this, so to accurately define a digital object we need to look at how we as humans deal with digital objects and use this to future proof our definition. This involves looking at the senses humans use for viewing digital objects and then expand on this.

So let's redefine the definition of a digital object to the following:

A digital object is a representation of anything, stored in binary format, to be used by our senses.

Why to be used by our senses? If there is no intention of use for a digital object, it can be classified as a digital file. A Windows DLL, a Unix executable, or security attributes are all digital representations of something, but they are not digital objects because they are not used by our senses. They are used by computers for the management of data. By specifying that the binary representation has to be used by our senses, then the boundaries of use for that digital object are captured and can then be further defined.

The traditional view is that we have five senses: sight, sound, touch, taste, and smell. When looking in greater detail at these senses we can break them down as shown in the following table:

There is not much difference between taste and smell, as both involve chemical reactions. Interestingly, we can actually taste with our nose⁽¹⁵⁾. When it comes to sound we can actually feel certain low vibration sounds. For watching movies on DVDs, this is an important experience and part of the entertainment value. In this case the deep bass, which is emitted by certain speakers, is felt by the body through touch. So one digital object can be used by multiple senses.

By equating it to a sense we can resolve a number of real world problems associated with defining an image. For example, a document that can be viewed or read using sight, can be converted to Braille and then read by touch. For those familiar with the TV show Red Dwarf, in that series they even explored the concept of reading a book using smell.

Just because we currently are not using one of our senses for viewing a digital object, it does not mean it should be excluded. A good example is taste. Currently it is very hard to simulate taste in a digital sense, but this doesn't mean that in ten years' time the concept of artificial taste will not be invented.

A digital object can be broken down into image types. Each image type can be further broken into image subtypes. We can then apply conversion and transformation rules to each subtype to modify the digital object.

By breaking down the senses into their core concepts and then equating them to traditional image concepts, it now becomes possible to identify traditional object types and then define them.

A digital object does not need to have any meaning associated with it, nor does it have to represent a real world scene (which is the traditional view of a photograph). A picture of an abstract painting is a digital object and the white noise of an empty TV channel can be classified as a digital object.

For simplicity we will maintain a digital object as having to be stored in the binary format. Though there are audio and video formats that use analogue signals, these formats can be expressed in a binary digital format. Even when looking at artificial intelligence and the use of neural networks, this can be represented in a binary format.

Digital objects can be composed of two core types and the dimension of time. By combining these core types into different combinations, a variety of base types can be created.

The core types are as follows:

The use of the dimension of time is very fluid and varies based on how it is used. Video uses a very strict definition of time. Animated GIFs use a very simple time-based sequence, whereas heraldry uses a very loose definition of time. The use of time is covered later in greater detail.

When expanding this definition to handle three-dimensional objects, the concept of an artifact is introduced. This is an object created from physical materials, but created digitally. An example is a model created from a three-dimensional printer using resins and glue.

For each object type we introduce object subtypes. For example, we can define a photograph as a real world representation of a picture. A line drawing is a hand drawing. The CGI is a computer-generated image.

Picture

A picture is a two-dimensional representation of anything. A picture can be viewed using all senses. A picture is defined as having a width and height assuming the picture is rectangular. For non-rectangular pictures the width and height describe the upper boundary lengths of the picture.

The following are the examples of subtypes:

• Geo-raster

• Photo

• Art

• Line drawing

• Montage

• 3D view using a set of 2D (but still 2D)

• Stereoscopic image

When looking at these definitions it can be seen that the definition for a document and video can be expressed using the terms of a picture and an audio. By adding a rule set we can create new digital image types. The two new rule definitions as used previously are as follows:

Using new rules we can create new digital object types based on the core picture and audio image types.

The following are examples of non-traditional digital object types:

It is possible for a digital object to be categorized into multiple object types. This is because the line on what actually constitutes a digital object can change depending on how it is delivered. For example, an MP3 file is classified as an audio type. If it is delivered using the Real Player server and streamed to a client, it is treated as a video type. Another example is an animated GIF, which is a time-related set of images enclosed within a repeating cycle. An animated GIF is by definition a video, yet for ease of delivery, it is delivered as a specialized GIF (that is, a type of static picture).

This means it is important to separate the storage of the digital object from the delivery mechanism. The delivery mechanism might involve a virtual change of the digital object. The digital object exists in two (or possibly more) states and it isn't until the object is delivered that its true state is determined. When this happens it is called a virtual digital object.

In a perfect world there is no difference between a digital object and its delivery mechanism. But because of Internet standards that limit what can be seen (for example, browsers by default only view JPEG and GIF images and not TIFF ones) and due to limitations in network bandwidth and cost of delivery, virtual digital objects have had to be created to address these issues. These issues are subject to current environmental constraints and will change over time. HTML5 is attempting to define a set of supported video standards. This is covered further in Chapter 3, The Multimedia Warehouse.

One goal in this book is to describe how to deliver a digital object. This is covered in great detail in Chapter 6, Delivery Techniques. At the moment we have classified what a digital object is, but have not defined what it is to actually deliver it.

We expand on the original definition and add the following:

Only when that digital object has been successfully consumed by one of our senses can it be considered to be delivered.

This means a photograph viewed on the computer screen has been delivered. A DVD streamed to a computer terminal has been delivered and a document viewed and read has been delivered.

It is not important that money has been transacted when delivering a digital image. Buying a digital image is an optional part of the delivery process.

But what about the scenario where an audio file is cut to CD and then shipped to a customer? What if that customer does not listen to it? By the preceding definition it has not been consumed therefore it has not been delivered. Common sense indicates that the image has been delivered. From a traditional consumer viewpoint it is on actual receipt of the digital image that the image can be considered to be delivered. This view is now starting to conflict with new e-commerce concepts starting to appear on the Internet. That is, consumers are now only being charged for use of a digital image only when it has been consumed and not when they have received it.

So when defining consumption of an image and ensuring that definition is future proof, we have to be careful that our traditional viewpoint of commerce does not interfere with that definition. With e-commerce the rules are changing, consumer habits are changing, and new ideas for image delivery are being tried.

At this point we will leave the definition as it is. In Chapter 6, Delivery Techniques, we will explore this concept in greater detail. Here we will be looking at who the consumer is and who the producer is. With e-commerce our traditional perspectives need to be challenged.

This section is an introduction to the methods available for managing and manipulating a digital object. When working with digital objects it soon becomes apparent that techniques have to be utilized to view and understand what they actually are. The digital object itself can contain other digital objects and only by processing it can these other objects be discovered.

The photograph is also referred to as a picture, but the proper usage is a digital image.

It can be taken by a digital camera or can be scanned in. A photograph can have the metadata embedded in it (common formats include EXIF, IPTC, Adobe XML, Dicom). A photo can be of type JPEG, TIFF, PNG. There are well over 300 other types. Some camera manufacturers use a raw option when storing their digital images (two of the most common formats being DNG and NEF).

The photo is stored in the Oracle multimedia ORDSYS.ORDIMAGE data type. More complex photos can be stored in the ORDDICOM data type along with other multimedia types.

A photo can also be of type Georaster, in which case it's best stored using the Oracle Spatial Georaster data type.

A photo can be defined as a two-dimensional object composed of binary data. The photo is typically stored in compressed format using compression software built for that image type.

A video is a time-based set of two-dimensional photographs with optional audio. A video can contain metadata. It can also optionally contain an audio track (audio type) and a caption (document type). The video can be compressed and photographs can be extracted from the image. Common examples include MPEG, Divx, AVI, and QuickTime.

In Oracle multimedia, a video is stored using the ORDSYS.ORDVIDEO data type.

An audio image is a time-based collection of analog-based sounds. An audio image can be compressed. It can also optionally contain a caption (document type).

In Oracle multimedia, audio is stored using the ORDSYS.ORDAUDIO data type.

It is a set of two-dimensional pictures that conform to a well-defined set. A document can also contain within it all image types. As a result, a document is stored as binary not character. Microsoft Word and Adobe PDF are two well known examples. There are over 3000 examples of documents found in the marketplace.

A document can be indexed using Oracle text.

In Oracle multimedia, a document is stored using the ORDSYS.ORDDOC data type.

Text is a document that is not binary but composed of character data only. It can contain structured data (the two best known examples being relational and XML). Depending on the type, it is determined where to store the data. It can be stored in an XML type, an Oracle table, a CLOB, or a varchar field. It can be indexed using Oracle text.

It is a three-dimensional representation of an object. Though still in its infancy, some cameras can create a 3D view of an object. It can also be referred to as a blueprint, equating to a three-dimensional drawing used by architects⁽¹⁶⁾.

In Oracle multimedia, an artifact is stored using the ORDSYS.ORDSOURCE data type.

The multimedia is not just limited to the types mentioned. It can include anything. In the next decade we will be seeing new types of multimedia containing very large amounts of data. Some of these will be based on life sciences and simulation. Individual multimedia files will be on an average over a 1 TB in size.

For those familiar with VMware, it is feasible (but not currently practical) to store whole VMware instances as a multimedia type. These can be of any size. As more sites move down the virtualization path, the ability to create many installs will be simplified and organizations will be creating more of them. In the next decade we will see computers with a large number of cores and very large amounts of memory being able to host one VMware instance per user. Thus going down the path of each user having their own client computer, which is centrally stored and managed. Once more we see the rules for tuning and management change. Smart use of virtualization will ensure that the CPU use is fairly distributed. But as the number of these virtualizations grow, they will need to be managed and archived. And the most logical place to put them is in a database.

Other smaller sized types can include e-mail messages, flash files, and executables.

It is a set of one or more multimedia types stored in the one type. A ZIP, RAR, or TAR file can contain a mixture of multimedia. Further multimedia can then be extracted. A Dicom file can contain multiple multimedia types. Certain photographic file formats can contain multiple images within them. A GIF can be animated, a TIFF can contain multiple images, and a JPEG can contain other JPEGs within it. How the multimedia is going to be used best determines how it is stored.

A composite type is different to a product group or a container.

A composite type introduces the concept of multiple originals. Our traditional notion of an image needs to extend to deal with a multimedia type that is related to other multimedia. A good example is with a DICOM image. This is typically an image that contains information about a medical patient. It can contain patient history, x-rays, ultrasound, and scans. Each is a different image type, but together they all represent the one patient. If we view the patient as an object, then the object is a digital image composed of multiple original with each one being another multimedia type. Another example is a museum painting. The one painting can have multiple photographs taken of it. It might have an associated video showing how it was painted and an audio commentary of it by the artist. Each is a separate image but together they create one image with multiple originals in it. Another example is when a photographer takes a mosaic picture. This is a set of photographs of a scene that can be stitched together to create a new picture (like a jigsaw puzzle). Each image is still treated separately.

For a composite type, one digital image is chosen as the representative image and used as the thumbnail. Depending on the context in which the image is used or accessed, this representative image can dynamically change.

Composite types are best handled using the Oracle database's object/relational capabilities.

It is used to describe the fact that a file type can have multiple encoding algorithms used within it. A video file of type AVI is a container because different compression formats can be used within it. This is covered in greater detail in Chapter 2, Understanding Digital Objects.

A ZIP file is a specialized composite type. The goal is compression. ZIP is now used in the vernacular, even though there are other products that can do the same task. Some of these include Winrar, Unix Tar, and Unix Gzip. With ZIP the idea is to create one or more large files containing all the other files and to compress them within it. This is useful for backups or delivery/transfer of large number of images between computer systems.

Within Oracle there are a number of methods for dealing with a ZIP file. The context or how it's designed to be used determines what the ZIP file actually is. The following highlights three different uses of a ZIP file:

The metadata is a text data associated with a digital object for the purpose of searching and providing structured or semi-structured information about the digital object. Metadata is covered in greater detail in Chapter 3, The Multimedia Warehouse. In Oracle metadata can be stored in tables.

For a multimedia type, the NULL equivalent should be discussed. It's possible for the digital images associated with metadata to exist, but the actual multimedia component to not yet exist. For example, a museum has information about an object that needs to be photographed and stored in the database. The object could be a painting, a vase, a person, or any general collection object. They first store the metadata about the object in the database. At a later time the object is scanned, photographed, or a video is taken of it. The digital image is then associated with the initial metadata.

Because the metadata for the object exists but its associated multimedia does not, this is considered the NULL case for the multimedia type. The definition of NULL is related to the potential of what the image could become and avoids confusing a NULL image with one that is empty or blank.

Images stored in the database can be directly linked with metadata. In the one transaction an image can be manipulated, a thumbnail of that image created, and all associated metadata modified. Related information is kept in sync. If an image is stored in a file system, it is possible for external processes to delete or modify that image, causing the image itself to either become orphaned or lose synchronicity with its corresponding relational data. Another common issue is web quality images losing their associated thumbnails, meaning web page displays become broken.

Oracle multimedia, which extends control over images, allows images to be manipulated inside the database. They can be resized, copied, converted, and rotated. This simplifies management of them and allows for the one programming environment (PL/SQL or Java).

Moving multimedia is simplified as only one object is being moved. When deleting any multimedia all associated thumbnails and metadata is deleted. Management becomes simpler and less prone to error, especially on recovery and when doing general database maintenance.

If all images are stored in a directory, fine grained control is not possible. That is, it is not possible to restrict access of the images to individual users. Once users can gain access to an image in the directory they can access all of them (this is based on the assumption of using Digest Authentication).

By storing an image in the database, fine grained security becomes possible. Access to an image can be restricted to individual users and it also becomes possible to achieve the following:

Using Oracle security it becomes possible to attach roles to images and introduce fine grained access on them. Security can be configured so that a user can access a thumbnail but not the original. Full auditing of who accessed each image and how they accessed it can be tracked. Auditing can also be included to keep track of network capacity used per user, making it possible to track and then charge for network usage.

The one backup program that is used to backup the database will also backup the images. This simplifies the backup process. In the event of failure the whole database can be recovered to the last committed transaction.

The traditional behavior for backing up a filesystem is to back it up daily or weekly. This means in the event of failure the filesystem on recovery will be out of sync with the database by at least one day.

So by having the images in the database only one backup program is required and in the event of failure only one recovery procedure is needed.

Another advantage that can be seen comes when using some of the more advanced database features such as standby databases and replication. Images are automatically replicated if the advanced replication option is used, and for disaster recovery situations, image data is automatically transferred to a standby database.

The Oracle database is designed to handle backing up and recovering very large volumes of data. Using RMAN, incremental backups ensure only the changed blocks are backed up. Backup and recovery can be done in parallel. The database supports full rollback. Recovery is done until the last committed transaction, meaning no data is lost. This is important as it ensures that when the database is recovered all multimedia and associated metadata match. If a filesystem was used to store the multimedia and a database to store the metadata, when failure occurs it becomes possible for the metadata to become out of sync with the files recovered on the filesystem.

Though the latest generation of SANs can do high volume and high-speed backups of data, there is still no way to guarantee complete consistency between the database on the SAN. With changes in technology, a number of SANs now support real-time block level replication and can ensure consistency. This capability is vendor and database specific.

All the data is in one location. The digital image becomes an object and can be accessed using the one query. Client server and web applications can access the one image and retrieve its associated metadata using the one SQL statement. Image management is also subject to the same transactional rules as relational data. Using PLSQL and/or Java, the one query can access a variety of multimedia types in the one query. For example, a photo with its associated metadata and video can be retrieved in the one simple query. It's also possible to do a query that not only retrieves the metadata, but also performs a spatial query to do analysis of it.

The simplistic nature of these highly complex tasks makes it a powerful option to use.

An image stored in the database can be indexed. If an image is a document it can be thematically searched and gists (summaries) can be extracted from it.

An image can be converted from one format to another. Metadata can be extracted from it. It can be copied, re-sized, and the image quality controlled.

When it comes to managing and controlling the images in the database, the Oracle database offers the greatest in flexibility. Sets of images can be deleted, updated, or copied as easy as it is to write a query.

Images can be linked together and metadata can be easily attached to them. All data related to an image or set of images can logically co-exist.

This adds flexibility, which gives a DBA and developer greater control over managing and working with the images.

The Oracle database has various built-in features, which when used makes it easier to manage and deliver multimedia:

Even though the focus should always be to store the images in the database first, this experience is still to be accepted in the marketplace. The attitude is still to store it in the filesystem. Most management when confronted with the idea of putting images in the database invariably come up with the same set of fears that first appeared over ten years ago when database vendors first tried to push the idea of storing images in the database and failed. What's different now is that the rules have changed, the technology has changed and experience has shown that a lot of the previously raised issues are not valid anymore.

Isn't it slower to retrieve an image from the database compared to a filesystem?

This might have been true ten years ago on older disk systems, but with improvements in disk technology this issue has subsequently disappeared. Tests have shown that it is just as fast to retrieve an image from a database as it is from a disk filesystem.

In addition, by using optional caching technology, it is possible to cache frequently accessed images thus improving the time to retrieve them.

Fine grained control over where an image is stored and how it's accessed is possible. A thumbnail can be stored on a local disk and cached in the SGA to ensure the fastest possible speed for retrieval. The original can be stored on lower speed disks. The architecture of the Oracle database is one that inherently supports scalability. This makes it simpler to develop applications that load and deliver images.

Oracle's new Securefile Lobs offer speeds nearly twice as fast as previous versions, for loading and retrieval.

Doesn't it take more storage to put in an image in the database compared to a file system?

Yes it does. The storage format used in the database adds extra overheads to manage locking and to reserve storage for growth. In addition, Oracle puts indexes on images to improve the time it takes to retrieve and manipulate them.

Though there is extra storage required, it is not significant in the overall storage requirements. When tens of thousands of images are stored in the database, the extra required is dwarfed by the overall size of the images. It is also fair to say that disk is cheaper than it once was, and when it comes to database management, the strategy now is to sacrifice storage for performance. When dealing with relational data it is now common practice to add additional indexes and use locally managed tablespaces to improve performance. These extra features come at an additional storage cost. In some cases with data warehouses just for storing relational data, the rule of thumb is to factor in eight times the raw storage to handle all the additional overheads.

So increasing the storage requirements in the database by storing images in it, only ensures that the image data is retrieved optimally and consistently.

Isn't it more complicated and time consuming having to put images into the database and retrieve them compared to a filesystem?

This was exactly the same argument used fifteen years ago when relational databases first appeared. But at that time the argument was concerned with storing data in what was known as flat files versus putting it into a relational database. Time has shown that the overheads of putting data into a relational database offer more benefits and ultimately greater control than when storing it in a flat file. The same argument can be applied to images. Yes there is some programming overhead to put them in, but the advantages gained from having them in the database (as explained previously) is greater than when not having them in the database.

So when it comes to storing and managing those digital assets, keep in mind that ultimately it is easier, safer, and better to keep them stored in a database.