Funambol development

Exclusive offer: get 50% off this eBook here
Funambol Mobile Open Source

Funambol Mobile Open Source — Save 50%

Sync your email and other data with mobile devices using Funambol 7.1

$23.99    $12.00
by Stefano Fornari | December 2009 | Networking & Telephony Open Source

This article by Stefano Fornari explains how to develop extensions for the Funambol DS Service so that a developer can create integrations with other backend datasources.

The following sections present several concepts related to how Funambol can be used to build mobile applications. Before digging into the details of Funambol development, it is useful to describe some basic concepts of data synchronization, as it is the foundation for Funambol applications and services.

Data synchronization

All mobile devices—handheld computers, mobile phones, pagers, and laptops—need to synchronize their data with the server where the information is stored. This ability to access and update information on the fly is the key to the pervasive nature of mobile computing. Yet, today almost every device uses a different technology for data synchronization.

Data synchronization is helpful for:

  • Propagating updates between a growing number of applications
  • Overcoming the limitations of mobile devices and wireless connections
  • Maximizing the user experience by minimizing data access latency
  • Keeping scalability of the infrastructure in an environment where the number of devices (clients) and connections tend to increase considerably
  • Understanding the requirements of mobile applications, providing a user experience that is helpful, and not an obstacle, for mobile tasks

Data synchronization is the process of making two sets of data look identical, as shown in the following figure:

Funambol Mobile Open Source

This involves many techniques, which will be discussed in the following sections. The most important are:

  • ID handling
  • Change detection
  • Modification exchange
  • Conflict detection
  • Conflict resolution
  • Slow and fast synchronization

ID handling

At first glance, ID handling seems like a pretty straightforward process that requires little or no attention. However, ID handling is an important aspect of the synchronization process and is not trivial.

In some cases a piece of data is identifiable by a subset of its content fields. For example, in the case of a contact entry, the concatenation of a first name and last name uniquely selects an entry in the directory. In other cases, the ID is represented by a particular field specifically introduced for that purpose. For example, in a Sales Force Automation mobile application, an order is identified by an order number or ID. The way in which an item ID is generated is not predetermined and it may be application or even device specific.

In an enterprise system, data is stored in a centralized database, which is shared by many users. Each single item is recognized by the system because of a unique global ID. In some cases, two sets of data (the order on a client and the order on a server) represent the same information (the order made by the customer) but they differ. What could be done to reconcile client and server IDs to make the information consistent? Many approaches can be chosen:

  • Client and server agree on an ID scheme (a convention on how to generate IDs must be defined and used).
  • Each client generates globally unique IDs (GUIDs) and the server accepts client-generated IDs.
  • The server generates GUIDs and each client accepts those IDs.
  • Client and server generate their own IDs and a mapping is kept between the two. Client-side IDs are called Locally Unique Identifiers (LUID) and server-side IDs are called Globally Unique Identifiers (GUID). The mapping between local and global identifiers is referred as LUID-GUID mapping. The SyncML specifications prescribe the use of LUID-GUID mapping technique, which allows maximum freedom to client implementations.

Change detection

Change detection is the process of identifying the data that was modified after a particular point in time that is, the last synchronization. This is usually achieved by using additional information such as timestamp and state information. For example, a possible database enabled for efficient change detection is shown in the following table:

ID

First name

Last name

Telephone

State

Last_update

12

John

Doe

+1 650 5050403

N

2008-04-02 13:22

13

Mike

Smith

+1 469 4322045

D

2008-04-01 17:32

14

Vincent

Brown

+1 329 2662203

U

2008-03-21 17:29

However, sometimes legacy databases do not provide the information needed to accomplish efficient change detection. As a result, the matter becomes more complicated and alternative methods must be adopted (based on content comparison, for instance). This is one of the most important aspects to consider when writing a Funambol extension, because the synchronization engine needs to know what's changed from a point in time.

Modification exchange

A key component of a data synchronization infrastructure is the way modifications are exchanged between client and server. This involves the definition of a synchronization protocol that client and server use to initiate and execute a synchronization session. In addition to the exchange modification method, a synchronization protocol must also define a set of supported modification commands. The minimal set of modification commands are as follows:

  • Add
  • Replace
  • Delete

Conflict detection

Let's assume that two users synchronize their local contacts database with a central server in the morning before going to the office. After synchronization, the contacts on their smartphones are exactly the same. Let's now assume that they update the telephone number for "John Doe" entry and one of them makes a mistake and enters a different number. What will happen the next morning when they both synchronize again? Which of the two new versions of the "John Doe" record should be taken and stored into the server? This condition is called conflict and the server has the duty of identifying and resolving it.

Funambol detects a conflict by means of a synchronization matrix shown in the following table:

Database A →

↓ Database B

New

Deleted

Updated

Synchronized

/

Unchanged

Not Existing

New

C

C

C

C

B

Deleted

C

X

C

D

X

Updated

C

C

C

B

B

Synchronized/Unchanged

C

D

A

=

B

Not Existing

A

X

A

A

X

As both users synchronize with the central database, we can consider what happens between the server database and one of the client databases at a time. Let's call Database A, as the client database and Database B, as the server database. The symbols in the synchronization matrix have the following meaning:

  • X: Do nothing
  • A: Item A replaces item B
  • B: Item B replaces item A
  • C: Conflict
  • D: Delete the item from the source(s) containing it

Conflict resolution

Once a conflict arises and is detected, proper action must be taken. Different policies can be applied. Let's see some of them:

  • User decides: The user is notified of the conflict condition and decides what to do.
  • Client wins: The server silently replaces conflicting items with the ones sent by the client.
  • Server wins: The client has to replace conflicting items with the ones from the server.
  • Timestamp based: The last modified (in time) item wins.
  • Last/first in wins: The last/first arrived item wins.
  • Merge: Try to merge the changes, at least when there is no direct conflict. Consider the case of a vcard, where two concurrent modifications have been applied to two different fields. There is a conflict at the card level, but the two changes can be merged so that both clients can then have a valid version of the card. This is the best example of the case when the change is not directly conflicting.
  • Do not resolve.

Note that Funambol adopts a special merging policy that guarantees that the user does not lose data. The server always tries to merge if possible. When a conflict cannot be resolved with merging (for example, there are conflicting changes on the same field), the value in the last synchronization wins over the older synchronizations to meet the expectation of the user who is synchronizing. In this way, when the users who applied previous changes receive the new updates all devices will be in sync.

Synchronization modes: Full or fast

There are many modes to carry out the synchronization process. The main distinction is between fast and full synchronization. Fast synchronization involves only the items changed since the last synchronization between two devices. Of course, this is an optimized process that relies on the fact that, the devices were fully synchronized at some point in the past; this way, the state at the beginning of the sync operation is well known and sound. When this condition is not met (for instance, the mobile device has been reset and lost the timestamp of the last synchronization), a full synchronization must be performed. In a full synchronization, the client sends its entire database to the server, which compares it with its local database and returns the modifications that must be applied to be up-to-date again.

Both fast and full synchronization modes can be performed in one of the following manners:

  • Client-to-server: The server updates its database with client modifications, but sends no server-side modifications
  • Server-to-client: The client updates its database with server modifications, but sends no client-side modifications
  • Two-way: The client and server exchange their modifications and both databases are updated accordingly

Extending Funambol

The Funambol platform can be extended in many areas to integrate Funambol with existing systems and environments. The most common integration use cases and the Funambol modules involved are:

  • Officer: Integrating with an external authentication and authorization service
  • SyncSource: Integrating with an external datasource to address client specific issues
  • Synclet: Adding pre or postprocessing to a SyncML message
  • Admin WS: Integrating with an external management tool

These are illustrated in the following diagram:

Funambol Mobile Open Source

Funambol extensions are distributed and deployed as Funambol modules. This section describes the structure of a Funambol module, while the following sections describe each of these listed scenarios.

A Funambol module represents the means by which developers can extend the Funambol server. A module is a packaged set of files containing Java classes, installation scripts, configuration files, initialization SQL scripts, components, and so on, used by the installation procedure to embed extensions into the server core.

For more information on how to install Funambol modules, see the Funambol Installation and Administration Guide.

Funambol Mobile Open Source Sync your email and other data with mobile devices using Funambol 7.1
Published: December 2009
eBook Price: $23.99
Book Price: $39.99
See more
Select your format and quantity:

Building a Funambol module

A Funambol module is a ZIP package named using the following convention:

<module-name>-<major-version>.<minor-version>.s4j

Here <module-name> is the name of the module without spaces and using lowercase characters only, and <major/minor-version> are the major and minor version numbers. A new version of a module with a minor version number change must be backward compatible, while changes in the major version number imply that a migration may be required.

The package must have the structure illustrated in the following diagram:

Funambol Mobile Open Source

In the previous diagram, entries ending with a '/' represent directories and filenames in italic are given as examples (in a real package they will be replaced with real filenames).

The module classes are packaged in a main JAR file called <modulename>.jar.

Configuration files are stored under the package directory config, creating subdirectories as needed.

Even if it is not mandatory, usually SyncSource instance configuration files are stored under a subtree in the form <module-id>/<connector-id>/<sourcetype-id>, which is the convention used by the Funambol Administration Tool when creating a new SyncSource instance.

The directory install contains install.xml, an Apache Ant script that is called when the module is installed; this is the hook where a module developer can insert module specific installation tasks. Installation specific files can be organized in subdirectories under install.

If the module requires a custom database schema, the scripts to create, drop, and initialize the database are stored under the sql/<database> directory, where  <database> is the name of the DBMS as listed in the install.properties file.

Finally, the exclude directory is used to store files that will be temporarily used by the installation procedure, but will not be (automatically) copied. These can be used by the module installation script for any purpose.

Modules, connectors, listeners, and SyncSource types

As mentioned, a module is a container for anything related to one or more server extensions, which are used by the engine to communicate and integrate with external systems. These extensions are usually specific with backend to be integrated. A specific case of such an extension is when the main purpose is to connect to an external datasource and in this case the module is called a connector. In other words, a connector is an extension of the server, intended to support the synchronization of a particular datasource.

To access the datasource, the connector must provide a SyncSource type. A SyncSource type represents the template from which an instance of a SyncSource can be created. For example, the FileSystemSyncSource type made available by Funambol is the means used by the server to synchronize data stored in the file system. However, it does not represent a particular directory to synchronize. In order to synchronize a specific directory (for instance /data/contacts) a real SyncSource instance must be created and configured with the desired directory. You can think of a SyncSource type as a class and of a SyncSource as an instance.

An additional (but optional) component that a connector can provide is called listener. This component is in charge of detecting changes in the backend so that the server can trigger a device to synchronize the changes.

Funambol provides a module called foundation that contains basic functionality, libraries, and classes required by all custom modules. This module must not be removed from any Funambol installation.

Registering modules, connectors, and SyncSource types

Modules, connectors, and SyncSource types are registered by filling the following database tables:

  • fnbl_module for module information
  • fnbl_connector  for connector information
  • fnbl_sync_source_type for SyncSource type information
  • fnbl_connector_source_type  for connector-SyncSource type associations
  • fnbl_module_connector  for module-connector association

The last two tables are used to create the hierarchy module-connector-SyncSource type that can be seen in the Funambol Administration Tool.

Let's consider the registration of the foundation module. When Funambol is installed, the foundation module is also installed. It brings a connector called FunambolFoundationConnector which, in turn, contains the SyncSource types PIM Contact SyncSource, PIM Calendar SyncSource, FileSystem SyncSource, and SIF SyncSource as shown in the following screenshot:

Funambol Mobile Open Source

This hierarchy is obtained with the following SQL commands:

  1. Module registration:
    insert into fnbl_module (id, name, description) 
    values('foundation','foundation','Foundation');
  2. SyncConnector registration:
    insert into fnbl_connector(id, name, description) 
    values('foundation','FunambolFoundationConnector',
    'Funambol Foundation Connector');
  3. The Foundation Connector belongs to the foundation module:
    insert into fnbl_module_connector(module, connector) 
    values('foundation','foundation');
  4. The SyncSource Type registration:
    insert into fnbl_sync_source_type(id, description, class, admin_class) 
    values('contact-foundation','PIM Contact SyncSource',
    'com.funambol.foundation.engine.source. PIMContactSyncSource',
    'com.funambol.foundation.admin. PIMContactSyncSourceConfigPanel');
    insert into fnbl_sync_source_type(id, description, class,admin_class)
    values('calendar-foundation','PIM Calendar SyncSource',
    'com.funambol.foundation.engine.source. PIMCalendarSyncSource',
    'com.funambol.foundation.admin. PIMCalendarSyncSourceConfigPanel');
    insert into fnbl_sync_source_type(id, description, class, admin_class)
    values('fs-foundation','FileSystem SyncSource',
    'com.funambol.foundation.engine.source. FileSystemSyncSource',
    'com.funambol.foundation.admin. FileSystemSyncSourceConfigPanel');
    insert into fnbl_sync_source_type(id, description, class, admin_class)
    values('sif-fs-foundation','SIF SyncSource',
    'com.funambol.foundation.engine.source.SIFSyncSource',
    'com.funambol.foundation.admin. SIFSyncSourceConfigPanel');
  5. Finally, the SyncSource type belongs to the Foundation Connector:
    insert into fnbl_connector_source_type(connector, sourcetype) 
    values('foundation','contact-foundation');
    insert into fnbl_connector_source_type(connector, sourcetype)
    values('foundation','calendar-foundation');
    insert into fnbl_connector_source_type(connector, sourcetype)
    values('foundation','fs-foundation');
    insert into fnbl_connector_source_type(connector, sourcetype)
    values('foundation','sif-fs-foundation');


Two classes are specified for each SyncSource type registration—The class (for example, com.funambol.foundation.engine.source.FileSystemSyncSource) that actually implements the SyncSource interface, and the admin_class that is used to create a new SyncSource instance and to configure it in the Funambol Administration Tool.

Summary

This article helps developers who want to extend the Funambol DS Service to integrate with their own backend datasources. This article presented some key concepts of the synchronization process, which provided us with the background information needed to understand some of the details of the Funambol API and framework.

 

If you have read this article you may be interested to view :

Funambol Mobile Open Source Sync your email and other data with mobile devices using Funambol 7.1
Published: December 2009
eBook Price: $23.99
Book Price: $39.99
See more
Select your format and quantity:

About the Author :


Stefano Fornari

Stefano Fornari is a co-founder and CTO of Funambol, Inc., the leading provider of open source mobile cloud sync and push email solutions for billions of phones. Stefano had several years of software development experience before starting the Funambol open source project in 2003. He was one of the main contributors of the project and was also the project manager. Today, Stefano is in charge of the engineering team of Funambol. Prior to his development work on Funambol, Fornari was the chief architect at Stigma Online where he played a key role in the development of the company’s flagship portal product, SolWeb Intra. He has also held positions at Compaq, where he was an advisor on wireless technologies in PDAs, and at Art Technology Group (ATG) as a consultant. He holds an M.S. Degree in Computer Science.

Books From Packt


GlassFish Administration
GlassFish Administration

Apache Roller 4.0 – Beginner's Guide
Apache Roller 4.0 – Beginner's Guide

Tomcat 6 Developer's Guide
Tomcat 6 Developer's Guide

Spring Persistence with Hibernate
Spring Persistence with Hibernate

Asterisk 1.6
Asterisk 1.6

Beginning OpenVPN 2.0.9
Beginning OpenVPN 2.0.9

Zabbix 1.6 Network Monitoring [RAW]
    Zabbix 1.6 Network Monitoring [RAW]

FreePBX 2.5 Powerful Telephony Solutions
FreePBX 2.5 Powerful Telephony Solutions


No votes yet

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
K
Y
p
Z
R
e
Enter the code without spaces and pay attention to upper/lower case.
Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software