What is a file type?
Let's not go as far deep as "What is a file?", but before we start, let's take a look at file types. File types are determined by the contents of the files themselves, and are used to allow the opening program to be chosen wisely. In Microsoft Windows, file extension globbing is the sole method of identifying file types. Users must provide a common phrase at the end of files, and the files would be searched by their name, in turn providing the correct icon and program.
Things are a little different in Ubuntu. Of course, globbing (the most basic method) is present for file types, but Ubuntu has a few other tricks up its sleeve. One of these is magic numbers. The magic number of a binary file is the first few bytes, which identify the file type. The definition of a "magic" number has somewhat loosened in recent years; it can now mean any piece of data, generally near the beginning of a file, that can be used to uniquely identify the type.
Another, more powerful, but too rarely used feature, is XML namespace matching. Without this feature, all XML files wouldn't be able to be more specifically identified, with the exception of extension globbing, of course. Namespace matching allows for quick detection of a XML-based format based on not only the namespace, but also the root element. For example, XHTML files (application/xhtml+xml) can not only be matched by an xhtml file extension, but also by its namespace URI (http://www.w3.org/1999/xhtml) and its root element (html).
How are file types detected?
In Ubuntu, programs such as Nautilus use the shared-mime-database as the sole location for file type information. Unfortunately, other Gnome facilities such as file open and save dialogs only use extension globbing, and are independent from the MIME database. These databases are stored in a similar way to how programs can be located in four tiers, /bin, /usr/bin, /usr/local/bin and ~/bin. These databases can be found in the following directories:
Like the program tiers, it is generally agreed that only MIME types installed from Ubuntu packages should be located in the first level. System-wide changes by the user or programs installed via make install are placed in the second tier, while changes local to the user are in the third.
The directories inside these MIME databases represent MIME groups, for example ./video for video/* MIME types, and ./application for application/* types. Not all of these directories may exist; they'll be created on demand for file types. In these directories, there are multiple XML files, each named by their MIME suffix. They contain nodes with information about magic numbers, extension globs, parent types, child and alias types, and the file type description (often in multiple languages).
The update-mime-database command, invoked manually or as a trigger opened when packages are changed, draws upon the information in these files and turns them into fast-seeking formats that aren't as friendly as XML. These real databases are in the following files:
- aliases: alternate names for MIME types
- generic-icons: system icons to be used for files
- globs: extension globbing without priority values (deprecated)
- globs2: extension globbing with priority values (current)
- icons: custom icons for odd file types
- magic: magic number database
- mime.cache: master cache with the entire database
- subclasses: child file types
- treemagic: detection of directory structures
- types: a list of MIME types
- XMLnamespaces: detection through XML namespaces and elements
A long time ago, when I was learning about the MIME database, I used Bless to directly edit these files to create changes, but was always confused by my changes immediately disappearing. This is because the information is converted one way from the XML files to the cache files.
The structure of the XML files
Before we use programs to modify the MIME database for convenience, here's a quick breakdown of the format of the XML files in the database. The root element is mime-info, with the shared MIME info namespace:
This root element contains any number of mime-type nodes, providing detection information about a file type. You could even have an empty mime-info node, but that isn't productive at all.
The following are a selection of the most important elements that can be found in mime-type nodes:
- glob nodes with a simple wildcard glob in a pattern attribute. A weight attribute from 0 to 100 is optional, and defaults to 50:
<glob pattern="*.mkv" weight="55"/>
- glob-deleteall and magic-deleteall nodes, which clear any cascading of globs or magic numbers from previously parsed files and starts afresh
- magic nodes with an optional priority attribute from 0 to 100 (again defaulting to 50). These contain match nodes, which define rules for matching using magic numbers. These are the attributes to be used with match elements:
- type: one of string, host16, host32, big16, big32, little16, little32 or byte
- offset: where to check for the magic, using a single numeric offset or a range notated start:end
- value: the value to match with (numeric for any type other than string)
- mask: an optional attribute, this can be used for more detailed matches by running a bitwise AND on the potential match before testing. The value is either numeric (in the type specified) or strings, which are hexadecimal values all starting with 0x
<match type="string" offset="0" value="DVDVIDEO">
- alias nodes, with a type attribute specifying alternate or deprecated MIME types that are equivalent
- sub-class-of nodes, with a type attribute specifying the parent MIME type
- comment, acronym and expanded-acronym nodes that help describe the file type to people; xml:lang attributes can be used to distinguish language
- root-XML elements which determine types using XML namespaces have namespaceURI and localName (root element) attributes
Here's an example XML source file that uses a couple of these features (this file type is bogus, I just created it for the example):
<comment xml:lang="en-AU">DML source document</comment>
<expanded-acronym xml:lang="en-AU">Delan's Markup Language</expanded-acronym>
<root-XML namespaceURI="http://azabani.com/dml" localName="dml"/>
Assogiate: a GUI editor for the Gnome MIME database
Assogiate is a neat little program that allows you to create and modify file types, modifying the database in a very user-friendly and quick way. It can access the user database, ~/.local/share/mime, or the system override database, /usr/local/share/mime. Changes are not, however, placed in XML files with the file name structure of the MIME type, instead they are placed in ./packages/Override.xml allowing for a memory of the user-changed file types.
Assogiate can be found in the Ubuntu universe repository:
sudo apt-get install assogiate
In the case that it is not, you can download and compile it:
http://azabani.com/files/apps/assogiate-0.2.1.tar.gz| tar xvz
cd assogiate-0.2.1; ./configure; make; sudo make install
You aren't allowed to change the system override database without running the program as a privileged user, so always run it as root:
In the Assogiate window, you can use the toolbar buttons to add and modify selected file types, remove and revert changes, or search for file types. The left pane allows you to narrow your view to groups of MIME types, or user modified types.
Adding and editing file types
The process for these two actions is very similar. When you are in the Edit Type dialog, you can edit canonical information, alias and parent types, globbing, magic numbers and XML namespace matching each in its own tab.
Bless: a useful GUI-based hex editor
Bless is a very powerful hex editor for inspecting and modifying binary files. A very important use of this program is in assisting in finding magic numbers for file types. It also can be found in the Ubuntu repositories as bless:
sudo apt-get install bless
To inspect a file, simply pass the file as the first argument:
For example, in the image above (a Quicktime mov container video), common candiate for a magic number could be located at offset 4, and is the string "ftypqt". Always check in multiple files of a known type to make sure that the magic number is consistent. Once a magic number has been identified using Bless, Assogiate can now be used to add the magic number:
Once information about a file type is altered, the changes take effect immediately. If you are adding a type, you can reload a directory with a file of the type in Nautilus and see the icon update, or right click the file and choose Properties to clarify the file type.
Example: adding a magic number for Matroska video
This is a good example because by default, Ubuntu only matches Matroska video with extension globbing and a magic number that no longer works. In this example, we'll research the magic number for this file type, and use assogiate to allow Ubuntu to detect the file type of Matroska video without using an extension on the file.
Firstly, let's open two Matroska files in Bless:
bless Owl\ City:\ Fireflies
bless Remo\ Giazotto's\ Adagio\ in\ G\ minor
Looking at the lower half of the window, you can see that the two Matroska video files share the first four bytes, which is a convenient place for a magic number. Note that the "Show little endian decoding" checkbox is not checked; we will be using a 32-bit big endian value.
Opening Assogiate, search for Matroska and open the editor for the video/x-matroska file type:
In the File contents tab, click Add. The priority should be higher than 50 as this is a more certain way of determining the file type; but changing this is essentially optional. The value to be added is a big32 value of 440786851 at the very start of the file:
That's it! Ubuntu (and programs that use the MIME database, such as Nautilus) will now detect Matroska videos using their magic number, without the need for an extension.
Example: adding a magic number for .IFO files
It has always annoyed me that Ubuntu didn't have the magic number for the DVD structure files (IFO files); I couldn't open them automatically in SMPlayer without using the "Open with Other Application" dialog. Even then, it seemed messy as the files were then matched by their extension even though the .BUP files were identical and weren't matched. In this example, I will create a new file type with the magic, so .IFO and .BUP files are detected.
To start off, open Assogiate:
Create a new file type. We must choose a MIME type name; there is no formal name for this format so I chose video/x-dvd-information:
In Bless, I observed that these files shared a magic number where the files all start with the string "DVDVIDEO". In the File contents tab, add the magic number:
The changes will take effect immediately and that's all that's needed to allow these files to be recognised. However, upon opening the file, Ubuntu won't know what program to use. You can choose the application to open with, or use Ubuntu Tweak to associate groups of MIME types easily in one place.
Enter Ubuntu Tweak
Ubuntu Tweak is an all-in-one solution for modifying various settings about your computer. It handles everything from software sources to package cleaning, session control to theme settings, power management to templates and security settings, even file type associations. Note that Ubuntu Tweak cannot be used to add or modify file type identification info; that can be done with Assogiate. Ubuntu Tweak allows you to associate a default program to a MIME type.
To install Ubuntu Tweak, download the package file or add the Ubuntu Tweak PPA for automatic updates. First, authenticate the signature:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys e260f5b0
Then you can add the following line to /etc/apt/sources.list: (replace the RELEASE with your distribution codename, e.g. lucid)
Finally, install the program:
sudo apt-get update; sudo apt-get install ubuntu-tweak
The file/program associations are per-user; run Ubuntu Tweak as yourself:
All that needs to be done to associate an application for your account against a MIME type is to visit the File Type Manager page, and associate the program with the file type just as you would in Nautilus. Again, the changes take effect immediately, though if it doesn't, your local association database may be overriding the changes.
Congratulations! You now know everything you need to add, modify and associate file types, understand how the MIME database works, and have the power of Ubuntu at your hands. The instructions here apply to many Linux distributions that share similar software without a change, so this tutorial applies to Linux on the whole as well.
If you have read this article you may be interested to view :
- Compiling and Running Handbrake in Ubuntu
- Control of File Types in Ubuntu
- Network Based Ubuntu Installations
- Ubuntu 9.10: How To Upgrade
- Install GNOME-Shell on Ubuntu 9.10 Karmic Koala
- Five Years of Ubuntu
- Ubuntu User Interface Tweaks
- What's New In Ubuntu 9.10 "Karmic Koala"
- Securely Encrypt Removable Media with Ubuntu
- Folding @ Home on Ubuntu: Cancer Research Made Easy
- Securing Network Services with FreeBSD Jails
- Create a Local Ubuntu Repository using Apt-Mirror and Apt-Cacher