Extending Applications in Tcl

Exclusive offer: get 50% off this eBook here
Tcl 8.5 Network Programming

Tcl 8.5 Network Programming — Save 50%

Build network-aware applications using Tcl, a powerful dynamic programming language

$29.99    $15.00
by Piotr Beltowski Wojciech Kocjan | July 2010 | Open Source

In this article by Wojciech Kocjan and Piotr Beltowski, authors of the book Tcl 8.5 Network Programming, we will see how to create extensible applications using Tcl-based modules.

(For more resources on Tcl, see here.)

Very often applications need to be extended with additional functionalities. In many cases it is also necessary to run different code on various types of machines.

A good solution to this problem is to introduce extensibility to our application. We'll want to allow additional modules to be deployed to specified clients—this means delivering modules to clients, keeping them up to date, and handling loading and unloading of modules.

The following sections will introduce how Starkits and Tcl's VFS can be used to easily create modules for our application, and how these can be used to deploy additional code.

We'll also introduce a simple implementation of handling modules, both on server side and client side. This requires creating code for downloading modules themselves, loading them, and the mechanism for telling clients what actions they should be performing.

The server will be responsible for telling clients when they should load or unload any of the modules. They will also inform clients if any of the modules are out of date. Clients will use this information to make sure modules that are loaded are consistent, with what server expects.

The following steps will be performed by the client:

  • The client sends a list of loaded modules and a list of all available modules along with their MD5 checksums to server
  • The server responds with the list of modules to load, unload, and which need to be downloaded
  • The client downloads modules that it needs to download; if the client already has downloaded all modules it needs, no action is taken
  • The client unloads modules that it should unload and loads modules that it should load; if the client has already loaded all modules it should have, no action is taken

Handling security is an additional consideration when creating pluggable applications. Clients should check if files are authentic whenever they download any code, which will be run on the client. This can be achieved using SSL and Certificate Authorities.

Starkits as extensions

One of the features Tcl offers is its VFS and using single file archives for staging entire archives. We've used these technologies to create our clients as single file executables.

Now we'll reuse similar mechanisms to create modules for our application. All modules will simply be a Starkit VFS—this will allow embedding all types of files, creating complex scripts, and easily manage whatever a module contains.

Our modules will have their MD5 checksum calculated, similar to the automatic update feature. This will allow the application to easily check whether a module needs to be updated or not. In order to deliver modules to a client, the server will allow downloading modules, similar to retrieving binaries for automatic update.

Building modules

Our modules will be built similar to how binaries were built, with a minor exception. Modules will have their MD5 checksum stored as the first 32 bytes of the file; this does not cause problems for MK4 VFS package and allows easy checking of whether a package is up to date or not.

The source code for each module will be stored in the mod-<modulename> directories. For the purpose of this implementation, clients will source the load.tcl script when loading the package and will source the unload.tcl script when unloading it. It will be up to the module to initialize and finalize itself properly.

The directory and file structure in the following screenshot show how files for modules are laid out.

Extending Applications in TCL

It is based on previous example and only the mod-comm and mod-helloworld directories are new.

We'll need to modify the build.tcl script and the add code responsible for building of modules. We can do this after the initial binaries have been built.

First let's create the modules directory:

file mkdir modules

Then we'll iterate over each module to build and create name of the target file:

foreach module {helloworld comm} {
set modfile [file join modules $module.kit]

W e'll start off by creating a 32 byte file:

set fh [open $modfile w]
fconfigure $fh -translation binary
puts -nonewline $fh [string repeat "\0" 32]
close $fh

Next we'll create the VFS, copy the source code of the module, and unmount it:

vfs::mk4::Mount $modfile $modfile
docopy mod-$module $modfile
vfs::unmount $modfile

Now, calculate the MD5 checksum of the newly created module:

set md5 [md5::md5 -hex -file $modfile]

And set the first 32 bytes to the checksum:

    set fh [open $modfile r+]
fconfigure $fh -translation binary
seek $fh 0 start
puts -nonewline $fh $md5
close $fh
}

We'll create two modules—helloworld and comm. The first one will simply log "Hello world!" every 30 seconds. The second one will set up a comm interface that can be used for testing and debugging purposes.

Let's start with creating the mod-helloworld/load.tcl script which will be responsible for initializing the helloworld module:

csa::log::info "Loading helloworld module"

namespace eval helloworld {}

proc helloworld::hello {} {
csa::log::info "Hello world!"
after cancel helloworld::hello
after 30000 helloworld::hello
}

helloworld::hello

Our module will log the information that it has been loaded ,and write out hello world to the log. The helloworld::hello command also schedules itself to be run every 30 seconds.

The mod-helloworld/unload.tcl script that cleans up the module looks like this:

csa::log::info "Unloading helloworld module"

after cancel helloworld::hello

namespace delete helloworld

This will log information about the unloading of a module, cancel the next invocation of the helloworld::hello command, and remove the entire helloworld namespace.

Implementing the comm module is also simple. The mod-comm/load.tcl script is as follows:

csa::log::info "Loading comm module"

package require comm

comm::comm configure -port 1992 -listen 1 -local 1

This script simply loads the comm package, sets it up to listen on port 1992, and only accepts connections on the local interface.

Unloading the package (in mod-comm/load.tcl) will configure the comm interface not to listen for incoming connections:

csa::log::info "Unloading comm module"

comm::comm configure -listen 0

As the comm package cannot be simply unloaded, the best solution is for load.tcl to configure it to listen for connections and unload.tcl to disable listening.

Server side

The server side of extensibility needs to perform several activities. First of all we need to track which clients should be using which modules. The second function is providing clients with modules to download. The third functionality is telling clients which modules they need to fetch from the server, and which ones that they need to load or unload from the environment.

Let's start off with adding initialization of the modules directory to our server. We need to add it to src-server/main.tcl:

set csa::binariesdirectory [file join [pwd] binaries]
set csa::modulesdirectory [file join [pwd] modules]
set csa::datadirectory [file join [pwd] data]

We'll also need to load an additional script for handling this functionality:

csa::log::debug "Sourcing remaining files"
source [file join $starkit::topdir commapi.tcl]
source [file join $starkit::topdir database.tcl]
source [file join $starkit::topdir clientrequest.tcl]
source [file join $starkit::topdir autoupdate.tcl]
source [file join $starkit::topdir clientmodules.tcl]

Next we'll also need to modify src-server/database.tcl to add support for storing the modules list. We'll need to add a new table definition to script that creates all tables:

CREATE TABLE clientmodules (
client CHAR(36) NOT NULL,
module VARCHAR(255) NOT NULL
);

In order to work on the data we'll also need commands to add or remove a module for a specified client:

proc csa::setClientModule {client module enabled} {
if {[llength [db eval \
{SELECT guid FROM clients WHERE guid=$client
AND status=1}]] == 0} {
return false
}

db eval {DELETE FROM clientmodules WHERE
client=$client AND module=$module}

if {$enabled} {
db eval {INSERT INTO clientmodules (client, module)
VALUES($client, $module)}
}

return true
}

Our command starts off by checking if a client exists and returns immediately if it does not. In the next step, we delete any existing entries and insert a new row if we've been asked to enable a particular module for a specified client.

We'll also need to be able to list modules associated with a particular client, which means executing a simple SQL query to list modules:

proc csa::getClientModules {client} {
return [lsort [db eval {SELECT module
FROM clientmodules WHERE client=$client}]]
}

Then we'll need to create the clientmodules.tcl file that will have functionality related to handling modules, and providing them to clients.

The first thing required is a function to read MD5 checksums from modules. We'll first check if the file exists and return an empty string if it does not, otherwise we'll read and return the first 32 bytes of the file:

proc csa::getModuleMD5 {name} {
variable modulesdirectory

set filename [file join $modulesdirectory $name]

if {![file exists $filename]} {
return ""
}
set fh [open $filename r]
fconfigure $fh -translation binary
set md5 [read $fh 32]
close $fh
return $md5
}

Next we'll create a function that takes a client identifier, queries the database for modules for that client and returns a list of module-md5sum pairs, which can be treated as a dictionary – where the key is the module name and the value is its md5 checksum.

proc csa::getClientModulesMD5 {client} {
set rc [list]
foreach module [getClientModules $client] {
lappend rc $module [getModuleMD5 $module]
}
return $rc
}

Another function will handle requests for a particular module. Similar to how it is implemented for automatic updates, we'll provide files only from a single directory and handle cases where a file does not exist, and register a prefix for the requests in TclHttpd:

proc csa::handleClientModule {sock suffix} {
variable modulesdirectory
set filename [file join $modulesdirectory \
[file tail $suffix]]

log::debug "handleClientModule: File name: $filename"
if {[file exists $filename]} {
Httpd_ReturnFile $sock application/octet-stream \
$filename
} else {
log::warn "handleClientModule: $filename not found"
Httpd_Error $sock 404
}
}

Url_PrefixInstall /client/module csa::handleClientModule

For communication with the clients, we'll reuse the protocol for requesting jobs.

We also need a function that given a client identifier, request dictionary, and name of the response variable will provide information to the client. It will also return whether a client should be provided with a list of jobs or not. If a client will need to download new or updated modules first, we do not need to provide a list of jobs as the client will need to have updated modules first.

Let's start by making sure that both the modules available on the client and the list of loaded modules has been sent to the client:

proc csa::csaHandleClientModules {guid req responsevar} {
upvar 1 $responsevar response

set ok true

if {[dict exists $req availableModules]
&& [dict exists $req loadedModules]} {

Then we copy the values to local variables for convenience, and get a list of modules that a client should have along with their MD5 checksums.

set rAvailable [dict get $req availableModules]
set rLoaded [dict get $req loadedModules]
set lAvailable [getClientModulesMD5 $guid]

We'll also create a list of actions we want to pass back to the client— the list of modules it needs to download and the list of modules to load and unload. By default, all lists are empty and we'll add items only if we detect that the client should perform actions:

set downloadList [list]
set loadList [list]
set unloadList [list]

As the first step, we'll iterate over the modules that the client should have and check if it has them—if the client either does not have a module or its checksum differs, we'll tell the client to download it.

foreach {module md5} $lAvailable {
if {(![dict exists $rAvailable $module]) ||
([dict get $rAvailable $module] != $md5)} {
lappend downloadList $module
}

After this we check if the client has already loaded this module. If not, we'll tell him to load the module.

if {[lsearch -exact $rLoaded $module] < 0} {
lappend loadList $module
}
}

We'll also iterate over the modules that the client currently has loaded and if any of them should not be loaded according to our list, we'll tell the client to unload it.


foreach module $rLoaded {
if {![dict exists $lAvailable $module]} {
lappend unloadList $module
}
}

Once we've conducted our comparison, we tell the agent what should be done. If he needs to download any modules, we only return this information and return that there is no point in providing the list of jobs to perform:

if {[llength $downloadList] > 0} {
dict set response moduleDownload \
$downloadList
set ok false
} else {

Otherwise if all modules on the client are updated, we provide a list of modules to load or unload if this is needed.

if {[llength $loadList] > 0} {
dict set response moduleLoad \
$loadList
}

if {[llength $unloadList] > 0} {
dict set response moduleUnload \
$unloadList
}
}

Finally, we return whether or not we should provide the client with a list of jobs to perform or not:

}

return $ok
}

Now we'll need to modify the csa::handleClientProtocol command in the src-server/clientrequest.tcl file to invoke our newly created csa::csaHandleClientModules command:

if {[csaHandleClientModules \
$guid $req response]} {
# only specify jobs if client
# has all the modules

if {[dict exists $req joblimit]} {
set joblimit [dict get $req joblimit]
} else {
set joblimit 10
}

dict set response jobs \
[getJobs $guid $joblimit]
log::debug "handleClientProtocol: Jobs:\
[llength [dict get $response jobs]]"
}

This will cause jobs to be added only if the csaHandleClientModules command returned true.

We can also modify the csa::apihandle command in the src-server/commapi.tcl file to allow adding or removing a module from a client. The following needs to be added inside the main switch responsible for handling commands:

switch -- $cmd {
addClientModule {
lassign $command cmd client module
return [setClientModule $client $module 1]
}
removeClientModule {
lassign $command cmd client module
return [setClientModule $client $module 0]
}

These commands simply invoke the csa::setClientModule command created earlier.

Tcl 8.5 Network Programming Build network-aware applications using Tcl, a powerful dynamic programming language
Published: July 2010
eBook Price: $29.99
Book Price: $49.99
See more
Select your format and quantity:

(For more resources on Tcl, see here.)

Handling modules on client

The first thing that we'll need to implement on the client is support for handling modules—storage, loading and unloading, and keeping the state of the currently loaded modules.

Let's start off with adding initialization of the modules directory to our server. We need to add it to src-client/main.tcl:

set csa::datadirectory [file join [pwd] data]
set csa::modulesdirectory [file join [pwd] data modules]
file mkdir $csa::datadirectory
file mkdir $csa::modulesdirectory

Next we'll need to add loading of clientmodules.tcl, which will hold all scripts related to handling modules:

source [file join $starkit::topdir fileinfo.tcl]
source [file join $starkit::topdir autoupdate.tcl]
source [file join $starkit::topdir database.tcl]
source [file join $starkit::topdir client.tcl]
source [file join $starkit::topdir clientmodules.tcl]

The clientmodules.tcl file will hold all scripts related to modules. We'll start off by creating the code for getting the MD5 checksum of a module. Implementation on the client will use the modulemd5cache array for caching checksums of a module.

Caching checksums is needed because of the way that VFS is implemented, reading MD5 checksum is not possible for currently loaded modules. Also, this will speed up operations and as the client will download updates on its own, it is safe to keep MD5 checksums cached here.

proc csa::getModuleMD5 {module} {
variable modulesdirectory
variable modulemd5cache
if {![info exists modulemd5cache($module)]} {

If we don't know the checksum of the requested module we'll initialize it. If the file exists, then read the first 32 bytes and store it in the cache. If the file does not exist, we set it to an empty string:

set filename [file join $modulesdirectory $module]
if {[file exists $filename]} {
set fh [open $filename r]
fconfigure $fh -translation binary
set modulemd5cache($module) [read $fh 32]
close $fh
} else {
set modulemd5cache($module) ""
}
}
return $modulemd5cache($module)
}

We'll also need a procedure that removes the MD5 cache for a specified module:

proc csa::cleanModuleCache {module} {
variable modulemd5cache

catch {unset modulemd5cache($module)}
}

This will be needed in case we download a new version as we need to make sure our client does not use an older version of the checksum.

Now, we'll create a function to list all modules that are available locally to this client. We'll simply use the glob command to list items in the modules directory matching *.kit:

proc csa::getAvailableModules {} {
variable modulesdirectory

return [lsort [glob -nocomplain -tails \
-directory $modulesdirectory *.kit]]
}

Similar to the server-side implementation, we'll also create a function that returns all local modules along with their MD5 checksums.

proc csa::getAvailableModulesMD5 {} {
set rc [list]
foreach module [getAvailableModules] {
lappend rc $module [getModuleMD5 $module]
}
return $rc
}

Let's implement the loading of a particular module. The command returns whether the package has been successfully loaded or not.

We'll store all currently loaded modules in the loadedmodule array—loading package first checks if the specified package has currently been loaded. If it has been, we return the following code immediately:

proc csa::loadModule {module} {
variable loadedmodule
variable modulesdirectory

set filename [file join $modulesdirectory $module]

if {[info exists loadedmodule($module)]} {
return true
}

Then we check if a module exists. If it does not, its checksum is empty. In this case, too, we exit:

if {[getModuleMD5 $module] == ""} {
return false
}

Assuming the module exists we'll now mount it as read only and initialize the loadedmodule value for the specified module to the output of mounting the VFS, and source the VFS's load.tcl script:

set loadedmodule($module) \
[vfs::mk4::Mount $filename $filename -readonly]

if {[catch {
uplevel #0 [list \
source [file join $filename load.tcl] \
]
} error]} {

If sourcing the script fails, we log an error and unload the module. We'll also

log::error "Error loading module $module: $error"
unloadModule $module true
return false
}

return true
}

Now let's create the command to unload a module. It returns whether the package needed to be unloaded or not. Unloading a package works in similar way to loading it. We start off by checking if it is currently loaded:

proc csa::unloadModule {module} {
variable loadedmodule
variable modulesdirectory

set filename [file join $modulesdirectory $module]

if {![info exists loadedmodule($module)]} {
return false
}

If a package is currently loaded, we source the unload.tcl script. If it fails, we log an error, but continue anyway.

if {[catch {
uplevel #0 [list \
source [file join $filename unload.tcl] \
]
} error]} {
log::error "Error unloading module $module: $error"
}

Then we try to unmount the VFS. If that fails, we again log an error:

if {[catch {
vfs::unmount $filename
} error]} {
log::error "Error unloading module $module: $error"
}

Finally, we unset loadedmodule for a specified module and return:

unset loadedmodule($module)

return true
}

We can also create a function to list all currently loaded modules:

proc csa::getLoadedModules {} {
variable loadedmodule
return [lsort [array names loadedmodule]]
}

We simply return keys for the loadedmodule array since they correspond to the currently loaded modules.

Communication with server

Now that the csa client can handle modules, we'll need to implement retrieving information about them from the server.

It will need to send a list of all available modules, along with their checksums, and a list of currently loaded ones. Then the server will tell it which modules to download, load, and unload.

In order to do that we'll need to modify the csa::requestJobs, and csa:: requestJobsDone commands in the src-client/client.tcl file.

The first thing is that in csa::requestJobs we need to add a feature for sending the list of modules:

set url "http://${hostname}:8981/client/protocol"
set req [dict create]

dict set req guid $::csa::guid
dict set req password $::csa::password

dict set req availableModules \
[getAvailableModulesMD5]

dict set req loadedModules \
[getLoadedModules]

This will cause information about modules to be sent.

Next, in the csa::requestJobsDone command we need to handle the processing of the response related to modules:

if {$ok} {
if {![processModulesResponse $response]} {
after cancel csa::requestJobs
after 3600000 csa::requestJobs
log::info "requestJobsDone: Rescheduling next\
communications for in 1 hour from now"
}

# evaluate jobs to perform if any are available
if {[dict exists $response jobs]
&& ([llength [dict get $response jobs]] > 0)} {
runJobs [dict get $response jobs]
}
} else {
log::error "requestJobsDone: Server returned error"
}

What we do is invoke the csa::processModulesResponse command with the response. If it returns false, we schedule the next communication for after one hour from now. The command will return false if new modules need to be downloaded—in this case, we schedule the next communication in one hour in case downloading the modules fails.

We now need to write the procedure for handling part of the response related to modules.

We start by checking if a list of modules to download has been returned. If it has, we initialize the download of all modules and return.

proc csa::processModulesResponse {response} {
if {[dict exists $response moduleDownload]} {
downloadModules [dict get $response moduleDownload]
return false
}

Now we check if there are any modules to load or unload. We then iterate over the lists and load or unload the modules appropriately:

if {[dict exists $response moduleLoad]} {
foreach module [dict get $response moduleLoad] {
log::info "Loading module $module"
loadModule $module
}
}
if {[dict exists $response moduleUnload]} {
foreach module [dict get $response moduleUnload] {
log::info "Unloading module $module"
unloadModule $module
}
}

We'll implement downloading one or more modules asynchronously. The idea is that the csa::downloadModules command will start the download of the first module in the list, and pass any remaining modules as arguments to the csa:: downloadModulesDone callback, which will be invoked when HTTP request is done. Then we'll write the module to disk and if there are any other modules pending for download, we'll invoke the csa::downloadModules command again with remaining modules. This will then be done until all modules have been downloaded.

Let's start by creating the csa::downloadModules command . It gets the first module from the list and stores any remaining modules. Next we create the URL to the module and unload this module, if it is currently loaded. Unloading will allow us to write to that module later on.

proc csa::downloadModules {modules} {
variable hostname

set module [lindex $modules 0]
set modules [lrange $modules 1 end]

log::debug "downloadModules: Downloading $module; \
[llength $modules] modules remaining"

set url http://${hostname}:8981/client/module/$module
unloadModule $module

Next we try to initialize an HTTP request to specified URL. We provide the callback command that gets both current module and list of remaining modules passed as arguments.

In case of any errors, we log them and initialize the next communication with the server in one minute.

if {[catch {
http::geturl $url \
-timeout 900000 -binary 1 -command [list \
csa::downloadModulesDone $module $modules]
}]} {
log::error "downloadModules: Unable to \
download module $module"

# set up next request for jobs sooner
after cancel csa::requestJobs
after 60000 csa::requestJobs
}
}

Once the download of a module has been completed, our callback is invoked. We start off by checking for errors in the HTTP transaction.

proc csa::downloadModulesDone {module modules token} {
variable modulesdirectory

if {[http::status $token] != "ok"} {
log::error "downloadModulesDone: Invalid HTTP status"
} elseif {[http::ncode $token] == 404} {
log::warn "downloadModulesDone:\
Server does not provide module $module"
} elseif {[http::ncode $token] != 200} {
log::error "downloadModulesDone: Invalid HTTP code"
} else {

If no error has occurred, we write the module to disk. We then clear the cache of this module's checksum and clean up the HTTP token.

log::info "downloadModulesDone: Module $module \
downloaded"
set filename [file join $modulesdirectory $module]
set fh [open $filename w]
fconfigure $fh -translation binary
puts -nonewline $fh [http::data $token]
close $fh

cleanModuleCache $module

http::cleanup $token

Finally, we check if there are any additional modules to download. If there are, we retrieve them. If there are no more modules to download, we initialize the next communication as soon as possible to request which modules to load and get jobs to perform.

if {[llength $modules] > 0} {
downloadModules $modules
} else {
log::info "downloadModulesDone: All modules \
have been downloaded"
after cancel csa::requestJobs
after idle csa::requestJobs
}
return
}

Regardless of the presence of any additional modules, we also return from the procedure here.

The following code will get run if we ran into any problems. It will clean up the HTTP token and set up the next communication for one minute from now.

http::cleanup $token
after cancel csa::requestJobs
after 60000 csa::requestJobs
return
}

Summary

In this article we saw how modules can be added to allow additional code to be deployed to clients on demand.


Further resources on this subject:


Tcl 8.5 Network Programming Build network-aware applications using Tcl, a powerful dynamic programming language
Published: July 2010
eBook Price: $29.99
Book Price: $49.99
See more
Select your format and quantity:

About the Author :


Piotr Beltowski

Piotr Beltowski is an IT and software engineer, co-author of several US patents and publications. Master of Science in telecommunications, his technical experience includes various aspects of telecommunication, like designing IP networks or implementing support of network protocols. Piotr's experience as Level 3 customer support lead makes him focused on finding simple solutions to complex technical issues.

His professional career includes working in international corporations such as IBM and Ericsson.

Wojciech Kocjan

Wojciech Kocjan is a system administrator and programmer with 10 years of experience. His work experience includes several years of using Nagios for enterprise IT infrastructure monitoring. He also has experience in large variety of devices and servers, routers, Linux, Solaris, AIX servers and i5/OS mainframes. His programming experience includes multiple languages (such as Java, Ruby, Python, and Perl) and focuses on web applications as well as client-server solutions.

Books From Packt

jQuery 1.4 Reference Guide
jQuery 1.4 Reference Guide

Zabbix 1.8 Network Monitoring
Zabbix 1.8 Network Monitoring

Spring Security 3
Spring Security 3

FreeSWITCH 1.0.6
FreeSWITCH 1.0.6

JasperReports 3.6 Development Cookbook
JasperReports 3.6 Development Cookbook

Blender 2.49 Scripting
Blender 2.49 Scripting

Hacking Vim 7.2
Hacking Vim 7.2

Moodle 1.9 Extension Development
Moodle 1.9 Extension Development

Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software