Data cleansing

As covered, there are two key steps when loading a digital object. They are to load the digital object in and to match existing metadata to it. The ordering can be done either way and the match of the existing metadata is optional.

When the digital object is loaded, it needs to be processed. This includes creating derivatives as well as watermarking or general image cleanup (cropping, sharpening, adjusting, censoring).

Once the meta is attached to the digital object, it might need to be cleansed. The concept is similar to what happens in a data warehouse. Some basic cleansing processes include:

• Converting varchar (sets of characters) dates into proper dates. A date might be stored in a varchar field. The dates might be of a mixed format, such as 12th Jan 2010, 10/12/08, Jan 15th 2000. They need to be translated into a standard date format.

• Converting varchar numbers into numbers.

• Ensuring there are no orphaned relationships (all keys storing relationships match to valid digital...