Organizing a Virtual Filesystem

(For more resources related to this topic, see here.)

Files are the building blocks of any computer system. This article deals with portable handling of read-only application resources, and provides recipes to store the application data.

Let us briefly consider the problems covered in this article. The first one is the access to application data files. Often, application data for desktop operating systems resides in the same folder as the executable file. With Android, things get a little more complicated. The application files are packaged in the .apk file, and we simply cannot use the standard fopen()-like functions, or the std::ifstream and std::ofstream classes.

The second problem results from the different rules for the filenames and paths. Windows and Linux-based systems use different path separator characters, and provide different low-level file access APIs.

The third problem comes from the fact that file I/O operations can easily become the slowest part in the whole application. User experience can become problematic if interaction lags are involved. To avoid delays, we should perform the I/O on a separate thread and handle the results of the Read() operation on yet another thread. To implement this, we have all the tools required.

We start with abstract I/O interfaces, implement a portable .zip archives handling approach, and proceed to asynchronous resources loading.

Abstracting file streams

File I/O APIs differ slightly between Windows and Android (POSIX) operating systems, and we have to hide these differences behind a consistent set of C++ interfaces.

Getting ready

Please make sure you are familiar with the UNIX concept of the file and memory mapping. Wikipedia may be a good start (

How to do it...

  1. From now on, our programs will read input data using the following simple interface. The base class iObject is used to add an intrusive reference counter to instances of this class:

    class iIStream: public iObject { public: virtual void Seek( const uint64 Position ) = 0; virtual uint64 Read( void* Buf, const uint64 Size ) = 0; virtual bool Eof() const = 0; virtual uint64 GetSize() const = 0; virtual uint64 GetPos() const = 0;

    The following are a few methods that take advantage of memory-mapped files:

    virtual const ubyte* MapStream() const = 0; virtual const ubyte* MapStreamFromCurrentPos() const = 0; };

    This interface supports both memory-mapped access using the MapStream() and MapStreamFromCurrentPos() member functions, and sequential access with the BlockRead() and Seek() methods.

  2. To write some data to the storage, we use an output stream interface, as follows (again, the base class iObject is used to add a reference counter):

    class iOStream: public iObject { public: virtual void Seek( const uint64 Position ) = 0; virtual uint64 GetFilePos() const = 0; virtual uint64 Write( const void* B, const uint64 Size ) = 0; };

  3. The Seek(), GetFileSize(), GetFilePos(), and filename-related methods of the iIStream interface can be implemented in a single class called FileMapper:

    class FileMapper: public iIStream { public: explicit FileMapper( clPtr<iRawFile> File ); virtual ~FileMapper(); virtual std::string GetVirtualFileName() const{
    return FFile->GetVirtualFileName(); } virtual std::string GetFileName() const{ return FFile->GetFileName(); }

  4. Read a continuous block of data from this stream and return the number of bytes actually read:

    virtual uint64 BlockRead( void* Buf, const uint64 Size ) { uint64 RealSize =( Size > GetBytesLeft() ) ? GetBytesLeft() : Size;

  5. Return zero if we have already read everything:

    if ( RealSize < 0 ) { return 0; } memcpy( Buf, ( FFile->GetFileData() + FPosition ),static_cast<size_t>
    ( RealSize ) );

  6. Advance the current position and return the number of copied bytes:

    FPosition += RealSize; return RealSize; } virtual void Seek( const uint64 Position ) { FPosition = Position; } virtual uint64 GetFileSize() const { return FFile->GetFileSize(); } virtual uint64 GetFilePos() const { return FPosition; } virtual bool Eof() const { return ( FPosition >= GetFileSize() ); } virtual const ubyte* MapStream() const { return FFile->GetFileData(); } virtual const ubyte* MapStreamFromCurrentPos() const { return ( FFile->GetFileData() + FPosition ); } private: clPtr<iRawFile> FFile; uint64 FPosition; };

  7. The FileMapper uses the following iRawFile interface to abstract the data access:

    class iRawFile: public iObject { public: iRawFile() {}; virtual ~iRawFile() {}; void SetVirtualFileName( const std::string& VFName );void
    SetFileName( const std::string& FName );std::string GetVirtualFileName() const; std::string GetFileName(); virtual const ubyte* GetFileData() const = 0; virtual uint64 GetFileSize() const = 0; protected: std::string FFileName; std::string FVirtualFileName; };

Along with the trivial GetFileName() and SetFileName() methods implemented here, in the following recipes we implement the GetFileData() and GetFileSize() methods.

How it works...

The iIStream::BlockRead() method is useful when handling non-seekable streams. For the fastest access possible, we use memory-mapped files implemented in the following recipe. The MapStream() and MapStreamFromCurrentPos() methods are there to provide access to memory-mapped files in a convenient way. These methods return a pointer to the memory where your file, or a part of it, is mapped to. The iOStream::Write() method works similar to the standard ofstream::write() function. Refer to the project 1_AbstractStreams for the full source code of this and the following recipe.

There's more...

The important problem while programming for multiple platforms, in our case for Windows and Linux-based Android, is the conversion of filenames.

We define the following PATH_SEPARATOR constant, using OS-specific macros, to determine the path separator character in the following way:

#if defined( _WIN32 ) const char PATH_SEPARATOR = '\\'; #else const char PATH_SEPARATOR = '/'; #endif

The following simple function helps us to make sure we use valid filenames for our operating system:

inline std::string Arch_FixFileName(const std::string& VName) { std::string s( VName ); std::replace( s.begin(), s.end(), '\\', PATH_SEPARATOR ); std::replace( s.begin(), s.end(), '/', PATH_SEPARATOR ); return s; }

See also

  • Implementing portable memory-mapped files
  • Working with in-memory files

Implementing portable memory-mapped files

Modern operating systems provide a powerful mechanism called the memory-mapped files. In short, it allows us to map the contents of the file into the application address space. In practice, this means we can treat files as usual arrays and access them using C pointers.

Getting ready

To understand the implementation of the interfaces from the previous recipe we recommend to read about memory mapping. The overview of this mechanism implementation in Windows can be found on the MSDN page at

To find out more about memory mapping, the reader may refer to the mmap() function documentation.

How to do it...

  1. In Windows, memory-mapped files are created using the CreateFileMapping() and MapViewOfFile() API calls. Android uses the mmap() function, which works pretty much the same way. Here we declare the RawFile class implementing the iRawFile interface.

    RawFile holds a pointer to a memory-mapped file and its size:

    ubyte* FFileData; uint64 FSize;

  2. For the Windows version, we use two handles for the file and memory-mapping object, and for the Android, we use only the file handle:

    #ifdef _WIN32 HANDLE FMapFile; HANDLE FMapHandle; #else int FFileHandle; #endif

  3. We use the following function to open the file and create the memory mapping:

    bool RawFile::Open( const string& FileName,const string& VirtualFileName ) {

  4. At first, we need to obtain a valid file descriptor associated with the file:

    #ifdef OS_WINDOWS FMapFile = (void*)CreateFileA( FFileName.c_str(),GENERIC_READ,
    FILE_FLAG_RANDOM_ACCESS,NULL ); #else FFileHandle = open( FileName.c_str(), O_RDONLY ); if ( FFileHandle == -1 ) { FFileData = NULL; FSize = 0; } #endif

  5. Using the file descriptor, we can create a file mapping. Here we omit error checks for the sake of clarity. However, the example in the supplementary materials contains more error checks:

    #ifdef OS_WINDOWS FMapHandle = (void*)CreateFileMapping( ( HANDLE )FMapFile,NULL, PAGE_READONLY,
    0, 0, NULL ); FFileData = (Lubyte*)MapViewOfFile((HANDLE)FMapHandle,FILE_MAP_READ, 0,0,0 ); DWORD dwSizeLow = 0, dwSizeHigh = 0; dwSizeLow = ::GetFileSize( FMapFile, &dwSizeHigh ); FSize = ((uint64)dwSizeHigh << 32) | (uint64)dwSizeLow; #else struct stat FileInfo; fstat( FFileHandle, &FileInfo ); FSize = static_cast<uint64>( FileInfo.st_size ); FFileData = (Lubyte*) mmap(NULL, FSize, PROT_READ,MAP_PRIVATE, FFileHandle, 0); close( FFileHandle ); #endif return true; }

  6. The correct deinitialization function closes all the handles:

    bool RawFile::Close() { #ifdef OS_WINDOWS if ( FFileData ) UnmapViewOfFile( FFileData ); if ( FMapHandle ) CloseHandle( (HANDLE)FMapHandle ); CloseHandle( (HANDLE)FMapFile ); #else if ( FFileData ) munmap( (void*)FFileData, FSize ); #endif return true; }

  7. The main functions of the iRawFile interface, GetFileData and GetFileSize, have trivial implementation here:

    virtual const ubyte* GetFileData() { return FFileData; } virtual uint64 GetFileSize() { return FSize; }

How it works...

To use the RawFile class we create an instance and wrap it into a FileMapper class instance:

clPtr<RawFile> F = new RawFile(); F->Open("SomeFileName"); clPtr<FileMapper> FM = new FileMapper(F);

The FM object can be used with any function supporting the iIStream interface. The hierarchy of all our iRawFile implementations looks like what is shown in the following figure:

Implementing file writers

Quite frequently, our application might want to store some of its data on the disk. Another typical use case we have already encountered is the downloading of some file from the network into a memory buffer. Here, we implement two variations of the iOStream interface for the ordinary and in-memory files.

How to do it...

  1. Let us derive the FileWriter class from the iOStream interface. We add the Open() and Close() member functions on top of the iOStream interface and carefully implement the Write() operation. Our output stream implementation does not use memory-mapped files and uses ordinary file descriptors, as shown in the following code:

    class FileWriter: public iOStream { public: FileWriter(): FPosition( 0 ) {} virtual ~FileWriter() { Close(); } bool Open( const std::string& FileName ) { FFileName = FileName;

  2. We split Android and Windows-specific code paths using defines:

    #ifdef _WIN32 FMapFile = CreateFile( FFileName.c_str(),GENERIC_WRITE, FILE_SHARE_READ,
    NULL, CREATE_ALWAYS,FILE_ATTRIBUTE_NORMAL, NULL ); return !( FMapFile == ( void* )INVALID_HANDLE_VALUE ); #else FMapFile = open( FFileName.c_str(), O_WRONLY|O_CREAT ); FPosition = 0; return !( FMapFile == -1 ); #endif }

  3. The same technique is used in the other methods. The difference between both OS systems is is trivial, so we decided to keep everything inside a single class and separate the code using defines:

    void Close() { #ifdef _WIN32 CloseHandle( FMapFile ); #else if ( FMapFile != -1 ) { close( FMapFile ); } #endif } virtual std::string GetFileName() const { return FFileName; } virtual uint64 GetFilePos() const { return FPosition; } virtual void Seek( const uint64 Position ) { #ifdef _WIN32 SetFilePointerEx( FMapFile,*reinterpret_cast<const LARGE_INTEGER*>(
    &Position ),NULL, FILE_BEGIN ); #else if ( FMapFile != -1 ) { lseek( FMapFile, Position, SEEK_SET ); } #endif FPosition = Position; }

    However, things may get more complex if you decide to support more operating systems. It can be a good refactoring exercise.

    virtual uint64 Write( const void* Buf, const uint64 Size ) { #ifdef _WIN32 DWORD written; WriteFile( FMapFile, Buf, DWORD( Size ),&written, NULL ); #else if ( FMapFile != -1 ) { write( FMapFile, Buf, Size ); } #endif FPosition += Size; return Size; } private: std::string FFileName; #ifdef _WIN32 HANDLE FMapFile; #else int FMapFile; #endif uint64 FPosition; };

How it works…

Now we can also present an implementation of the iOStream that stores everything in a memory block. To store arbitrary data in a memory block, we declare the Blob class, as shown in the following code:

class Blob: public iObject { public: Blob(); virtual ~Blob();

Set the blob data pointer to some external memory block:

void SetExternalData( void* Ptr, size_t Sz );

Direct access to data inside this blob:

void* GetData(); …

Get the current size of the blob:

size_t GetSize() const;

Check if this blob is responsible for managing the dynamic memory it uses:

bool OwnsData() const; …

Increase the size of the blob and add more data to it. This method is very useful in a network downloader:

bool AppendBytes( void* Data, size_t Size ); … };

There are lots of other methods in this class. You can find the full source code in the Blob.h file. We use this Blob class, and declare the MemFileWriter class, which implements our iOStream interface, in the following way:

class MemFileWriter: public iOStream { public: MemFileWriter(clPtr<Blob> Container);

Change the absolute position inside a file, where new data will be written to:

virtual void Seek( const uint64 Position ) { if ( Position > FContainer->GetSize() ) {

Check if we are allowed to resize the blob:

if ( Position > FMaxSize - 1 ) { return; }

And try to resize it:

if ( !FContainer->SafeResize(static_cast<size_t>( Position ) + 1 )) { return; } } FPosition = Position; }

Write data to the current position of this file:

virtual uint64 Write( const void* Buf, const uint64 Size ) { uint64 ThisPos = FPosition;

Ensure there is enough space:

Seek( ThisPos + Size ); if ( FPosition + Size > FMaxSize ) { return 0; } void* DestPtr = ( void* )( &( ( ( ubyte* )(FContainer->GetData() )
)[ThisPos] ) );

Write the actual data:

memcpy( DestPtr, Buf, static_cast<size_t>( Size ) ); return Size; } } private: … };

We omit the trivial implementations of GetFileName(), GetFilePos(), GetMaxSize(), SetContainer(), GetContainer(), GetMaxSize(), and SetMaxSize() member functions, along with fields declarations. You will find the full source code of them in the code bundle of the book.

See also

  • Working with in-memory files

Working with in-memory files

Sometimes it is very convenient to be able to treat some arbitrary in-memory runtime generated data as if it were in a file. As an example, let's consider using a JPEG image downloaded from a photo hosting, as an OpenGL texture. We do not need to save it into the internal storage, as it is a waste of CPU time. We also do not want to write separate code for loading images from memory. Since we have our abstract iIStream and iRawFile interfaces, we just implement the latter to support memory blocks as the data source.

Getting ready

In the previous recipes, we already used the Blob class, which is a simple wrapper around a void* buffer.

How to do it...

  1. Our iRawFile interface consists of two methods: GetFileData() and GetFileSize(). We just delegate these calls to an instance of Blob:

    class ManagedMemRawFile: public iRawFile { public: ManagedMemRawFile(): FBlob( NULL ) {} virtual const ubyte* GetFileData() const { return ( const ubyte* )FBlob->GetData(); } virtual uint64 GetFileSize() const { return FBlob->GetSize(); } void SetBlob( const clPtr<Blob>& Ptr ) { FBlob = Ptr; } private: clPtr<Blob> FBlob; };

  2. Sometimes it is useful to avoid the overhead of using a Blob object, and for such cases we provide another class, MemRawFile, that holds a raw pointer to a memory block and optionally takes care of the memory allocation:

    class MemRawFile: public iRawFile { public: virtual const ubyte* GetFileData() const { return (const ubyte*) FBuffer; } virtual uint64 GetFileSize() const { return FBufferSize; } void CreateFromString( const std::string& InString ); void CreateFromBuffer( const void* Buf, uint64 Size ); void CreateFromManagedBuffer( const void* Buf, uint64 Size ); private: bool FOwnsBuffer; const void* FBuffer; uint64 FBufferSize; };

How it works...

We use the MemRawFile as an adapter for the memory block extracted from a .zip file and ManagedMemRawFile as the container for data downloaded from photo sites.

Implementing mount points

It is convenient to access all of the application's resources as if they all were in the same folder tree, no matter where they actually come from—from an actual file, a .zip archive on disk, or an in-memory archive downloaded over a network. Let us implement an abstraction layer for this kind of access.

Getting ready

We assume that the reader is familiar with the concepts of NTFS reparse points (, UNIX symbolic links (, and directory mounting procedures (

How to do it...

  1. Our folders tree will consist of abstract mount points. A single mount point can correspond to a path to an existing OS folder, a .zip archive on disk, a path inside a .zip archive, or it can even represent a removed network path.

    Try to extend the proposed framework with network paths mount points.

    class iMountPoint: public iObject { public:

  2. Check if the file exists at this mount point:

    virtual bool FileExists( const string& VName ) const = 0;

  3. Convert a virtual filename, which is the name of this file in our folders tree, to a full filename behind this mount point:

    virtual string MapName( const string& VName ) const = 0;

  4. We will need to create a file reader that can be used with the FileMapper class, for the specified virtual file inside this mount point:

    virtual clPtr<iRawFile> CreateReader(const string& Name ) const = 0; };

  5. For physical folders we provide a simple implementation that creates instances of the FileMapper class with the reference to iRawFile:

    class PhysicalMountPoint: public iMountPoint { public: explicit PhysicalMountPoint(const std::string& PhysicalName); virtual bool FileExists(const std::string& VirtualName ) const { return FS_FileExistsPhys( MapName( VirtualName ) ); } virtual std::string MapName(const std::string& VirtualName ) const { return ( FS_IsFullPath( VirtualName ) ) ?VirtualName : (
    FPhysicalName + VirtualName ); }

  6. Create the reader to access the data inside this mount point:

    virtual clPtr<iRawFile> CreateReader(const std::string& VirtualName ) const { std::string PhysName = FS_IsFullPath( VirtualName ) ?VirtualName :
    MapName( VirtualName ); clPtr<RawFile> File = new RawFile(); return !File->Open( FS_ValidatePath( PhysName ),VirtualName ) ? NULL : File; } private: std::string FPhysicalName; };

  7. The collection of mount points will be called FileSystem, as shown in the following code:

    class FileSystem: public iObject { public: void Mount( const std::string& PhysicalPath ); void AddAlias(const std::string& Src,const std::string& Prefix ); std::string VirtualNameToPhysical(const std::string& Path ) const; bool FileExists( const std::string& Name ) const; private: std::vector< clPtr<iMountPoint> > FMountPoints; };

How it works...

The MapName() member function transforms a given virtual filename into a form that can be passed to the CreateReader() method.

The FS_IsFullPath() function checks if the path starts with the / character on Android, or contains the :\ substring on Windows. The Str_AddTrailingChar() function ensures we have a path separator at the end of the given path.

The FileSystem object acts as a container of the mount points, and redirects the file reader creation to the appropriate points. The Mount method determines the type of the mount point. If the PhysicalPath ends with either .zip or .apk substrings, an instance of the ArchiveMountPoint class is created, otherwise the PhysicalMountPoint class is instantiated. The FileExists() method iterates the active mount points and calls the iMountPoint::FileExists() method. The VirtualNameToPhysical() function finds the appropriate mount point and calls the iMountPoint::MapName() method for the filename to make it usable with the underlying OS I/O functions. Here we omit the trivial details of the FMountPoints vector management.

There's more...

Using our FileSystem::AddAlias method, we can create a special mount point that decorates a filename:

class AliasMountPoint: public iMountPoint { public: AliasMountPoint( const clPtr<iMountPoint>& Src ); virtual ~AliasMountPoint();

Set the alias path:

void SetAlias( const std::string& Alias ) { FAlias = Alias; Str_AddTrailingChar( &FAlias, PATH_SEPARATOR ); } … virtual clPtr<iRawFile> CreateReader(const std::string& VirtualName ) const { return FMP->CreateReader( FAlias + VirtualName ); } private:

Set a prefix to be appended to each file in this mount point:

std::string FAlias;

Set a pointer to another mount point, which is hidden behind the alias:

clPtr<iMountPoint> FMP; };

This decorator class will add the FAlias string before any filename passed into it. This simple mount point is useful when developing for both Android and Windows, because in Android .apk, the files reside lower in the folder hierarchy than they do in a Windows development folder. Later we determine the folder, where our Android application resides, and mount it using the AliasMountPoint class.

As a reminder, the following is the class diagram of our iMountPoint interface and its implementations:

See also

  • Decompressing files from a .zip archives

Enumerating files in the .zip archives

To incorporate the contents of a .zip file seamlessly into our filesystem, we need to read the archive contents and be able to access each file individually. Since we are developing our own file I/O library, we use the iIStream interface to access .zip files. The NDK provides a way to read the .apk assets from your C++ application (see usr/include/android/asset_manager.h in your NDK folder). However, it is only available on Android 2.3, and will make debugging of file access in your game more complex on a desktop computer without an emulator. To make our native code portable to previous Android versions and other mobile operating systems, we will craft our own assets reader.

Android applications are distributed as .apk packages, which are basically just renamed .zip archives, containing a special folder structure and metadata inside them.

Getting ready

We use the zlib library and the MiniZIP project to access the content of a .zip archive. The most recent versions can be downloaded from

How to do it...

  1. The zlib library is designed to be extensible. It does not assume every developer uses only the fopen() calls or the std::ifstream interface. To read the data from our own containers with the iIStream interface, we cast the iIStream instances to the void* pointers and write a set of routines that are passed to zlib. These routines resemble the standard fopen()-like interface and essentially only redirect the zlib to our iIStream classes:

    static voidpf ZCALLBACK zip_fopen( voidpf opaque,
    const void* filename, int mode ) { ( ( iIStream* )opaque )->Seek( 0 ); return opaque; }

  2. Read compressed data from a .zip file. This indirection actually allows to access archives inside the other archives:

    static uLong ZCALLBACK zip_fread( voidpf opaque, voidpf stream,void* buf,
    uLong size ) { iIStream* S = ( iIStream* )stream; int64_t CanRead = ( int64 )size; int64_t Sz = S->GetFileSize(); int64_t Ps = S->GetFilePos(); if ( CanRead + Ps >= Sz ) { CanRead = Sz - Ps; } if ( CanRead > 0 ) { S->BlockRead( buf, (uint64_t)CanRead ); } else { CanRead = 0; } return ( uLong )CanRead; }

  3. Return the current position inside a .zip file:

    static ZPOS64_T ZCALLBACK zip_ftell( voidpf opaque, voidpf stream ) { return ( ZPOS64_T )( ( iIStream* )stream )->GetFilePos(); }

  4. Advance to the specified position. The offset value is relative to the current position (SEEK_CUR), file start (SEEK_SET), or file end (SEEK_END):

    static long ZCALLBACK zip_fseek ( voidpf opaque, voidpf stream,ZPOS64_T offset,
    int origin ) { iIStream* S = ( iIStream* )stream; int64 NewPos = ( int64 )offset; int64 Sz = ( int64 )S->GetFileSize(); switch ( origin ) { case ZLIB_FILEFUNC_SEEK_CUR: NewPos += ( int64 )S->GetFilePos(); break; case ZLIB_FILEFUNC_SEEK_END: NewPos = Sz - 1 - NewPos; break; case ZLIB_FILEFUNC_SEEK_SET: break; default: return -1; } if ( NewPos >= 0 && ( NewPos < Sz ) ) { S->Seek( ( uint64 )NewPos ); } else { return -1; } return 0; }

  5. We do not close or handle errors, so the fclose() and ferror() callbacks are empty:

    static int ZCALLBACK zip_fclose(voidpf op, voidpf s) { return 0; } static int ZCALLBACK zip_ferror(voidpf op, voidpf s) { return 0; }

  6. Finally, the pointers to all functions are stored in the zlib_filefunc64_def structure that is passed instead of the usual FILE* to all functions of MiniZIP. We write a simple routine to fill this structure, as shown in the following code:

    void fill_functions( iIStream* Stream, zlib_filefunc64_def* f ) { f->zopen64_file = zip_fopen; f->zread_file = zip_fread; f->zwrite_file = NULL; f->ztell64_file = zip_ftell; f->zseek64_file = zip_fseek; f->zclose_file = zip_fclose; f->zerror_file = zip_ferror; f->opaque = Stream; }

  7. Once we have implemented the fopen() interface, we can provide the code snippet to enumerate the files in the archive represented by the iIStream object. This is one of the two essential functions in the ArchiveReader class:

    bool ArchiveReader::Enumerate_ZIP() { iIStream* TheSource = FSourceFile; zlib_filefunc64_def ffunc; fill_functions( TheSource, &ffunc ); unzFile uf = unzOpen2_64( "", &ffunc ); unz_global_info64 gi; int err = unzGetGlobalInfo64( uf, &gi );

  8. Iterate through all the files in this archive:

    for ( uLong i = 0; i < gi.number_entry; i++ ) { char filename_inzip[256]; unz_file_info64 file_info; err = unzGetCurrentFileInfo64( uf, &file_info,filename_inzip, sizeof(
    filename_inzip ),NULL, 0, NULL, 0 ); if ( err != UNZ_OK ) { break; } if ( ( i + 1 ) < gi.number_entry ) { err = unzGoToNextFile( uf ); }

  9. Store the encountered filenames in a vector of our own structures:

    sFileInfo Info; std::string TheName = Arch_FixFileName(filename_inzip); Info.FCompressedSize = file_info.compressed_size; Info.FSize = file_info.uncompressed_size; FFileInfos.push_back( Info ); FFileNames.push_back( TheName ); } unzClose( uf ); return true; }

  10. The array of sFileInfo structures is stored in the ArchiveReader instances:

    class ArchiveReader: public iObject { public: ArchiveReader(); virtual ~ArchiveReader();

  11. Assign the source stream and enumerate the files:

    bool OpenArchive( const clPtr<iIStream>& Source );

  12. Extract a single file from the archive into the FOut stream. This means we can extract compressed files directly into the memory:

    bool ExtractSingleFile( const std::string& FName,const std::string&
    Password,const clPtr
    <iOStream>& FOut );

  13. Free everything and optionally close the source stream:

    bool CloseArchive();

  14. Check if such a file exists in the archive:

    bool FileExists( const std::string& FileName ) const { return ( GetFileIdx( FileName ) > -1 ); } …

  15. The following code is the sFileInfo structure mentioned in the preceding point, that defines where a file is located inside a .zip archive:

    struct sFileInfo {

  16. First, we need an offset to the file data inside the archive:

    uint64 FOffset;

  17. Then we need a size of the uncompressed file:

    uint64 FSize;

  18. And a size of the compressed file, to let the zlib library know when to stop decoding:

    uint64 FCompressedSize;

  19. Don't forget a pointer to the compressed data itself:

    void* FSourceData; }; … };

We do not provide the complete source for the ArchiveReader class, however, do encourage you to look into the accompanying source code. The second essential function, the ExtractSingleFile(), is presented in the following recipe.

How it works...

We use the ArchiveReader class to write the ArchiveMountPoint that provides seamless access to the contents of a .zip file:

class ArchiveMountPoint: public iMountPoint { public: ArchiveMountPoint( const clPtr<ArchiveReader>& R );

Create a reader interface to access the content of the archive:

virtual clPtr<iRawFile> CreateReader( const std::string& VirtualName ) const { std::string FName = Arch_FixFileName( VirtualName ); MemRawFile* File = new MemRawFile(); File->SetFileName( VirtualName ); File->SetVirtualFileName( VirtualName ); const void* DataPtr = FReader->GetFileData( FName ); uint64 FileSize = FReader->GetFileSize( FName ); File->CreateFromManagedBuffer( DataPtr, FileSize ); return File; }

Check if a specified file exists inside this archive mount point:

virtual bool FileExists(const std::string& VirtualName ) const { return FReader->FileExists(Arch_FixFileName(VirtualName)); } virtual std::string MapName(const std::string& VirtualName ) const { return VirtualName; } private: clPtr<ArchiveReader> FReader; };

The ArchiveReader class takes care of the memory management and returns a ready-to-use instance of MemRawFile.

Decompressing files from the .zip archives

We have the Enumerate_ZIP() function to iterate through individual files inside a .zip archive, and now it is time to extract its contents.

Getting ready

This code uses the same set of fopen()-like functions from the previous recipe.

How to do it...

  1. The following helper function does the job of file extraction and is used in the ArchiveReader::ExtractSingleFile() method:

    int ExtractCurrentFile_ZIP( unzFile uf,
    const char* password, const clPtr<iOStream>& fout ) { char filename_inzip[256]; int err = UNZ_OK; void* buf; uInt size_buf; unz_file_info64 file_info; err = unzGetCurrentFileInfo64( uf, &file_info,filename_inzip, sizeof(
    filename_inzip ),NULL, 0, NULL, 0 ); if ( err != UNZ_OK ) { return err; } uint64_t file_size = ( uint64_t )file_info.uncompressed_size; uint64_t total_bytes = 0; unsigned char _buf[WRITEBUFFERSIZE]; size_buf = WRITEBUFFERSIZE; buf = ( void* )_buf; if ( buf == NULL ) { return UNZ_INTERNALERROR; }

  2. Pass the supplied password to the zlib library:

    err = unzOpenCurrentFilePassword( uf, password );

  3. The following is the actual decompression loop:

    do { err = unzReadCurrentFile( uf, buf, size_buf ); if ( err < 0 ) { break; } if ( err > 0 ) { total_bytes += err; fout->Write( buf, err ); } } while ( err > 0 ); int close_err = unzCloseCurrentFile ( uf ); … }

  4. And the ExtractSingleFile() function performs the extraction of a single file from an archive into an output stream:

    bool ArchiveReader::ExtractSingleFile( const string& FName,const string& Password,
    const clPtr<iOStream>& FOut ) { int err = UNZ_OK; LString ZipName = FName; std::replace ( ZipName.begin(), ZipName.end(), '\\', '/' ); clPtr<iIStream> TheSource = FSourceFile; TheSource->Seek(0);

  5. Decompress the data through the following code:

    zlib_filefunc64_def ffunc; fill_functions( FSourceFile.GetInternalPtr(), &ffunc ); unzFile uf = unzOpen2_64( "", &ffunc ); if ( unzLocateFile( uf, ZipName.c_str(), 0) != UNZ_OK ) { return false; } err = ExtractCurrentFile_ZIP( uf,Password.empty() ? NULL : Password.c_str(),
    FOut ); unzClose( uf ); return ( err == UNZ_OK ); }

How it works...

The ExtractSingleFile() method uses the zlib and MiniZIP libraries. In the accompanying material, we have included the libcompress.c and libcompress.h files that contain the amalgamated zlib, MiniZIP, and libbzip2 sources.

The 2_MountPoints example contains the test.cpp file with the code to iterate an archive file:

clPtr<RawFile> File = new RawFile(); File->Open( "", "" ); clPtr<ArchiveReader> a = new ArchiveReader(); a->OpenArchive( new FileMapper(File) );

The ArchiveReader instance contains all the information about the contents of the file.

Loading resources asynchronously

The preface of this book tells us we are going to develop an asynchronous resources loading system in this article. We have completed all of the preparations for this. We are now equipped with secure memory management, task queues, and finally, the FileSystem abstraction with archive file support.

What we want to do now is to combine all of this code to implement a seemingly simple thing: create an application that renders a textured quad and updates its texture on-the-fly. An application starts, a white quad appears on the screen, and then, as soon as the texture file has loaded from disk, the quad's texture changes. This is relatively easy to do—we just run the LoadImage task that we implement here, and as soon as this task completes, we get the completion event on the main thread, which also owns an event queue. We cannot get away with a single mutex to update the texture data, because when we use the OpenGL texture objects, all of the rendering state must be changed only in the same thread that created the texture—in our main thread.

How to do it...

  1. Here we build the foundation for the resources management. We need the concept of a bitmap stored in a memory. It is implemented in the Bitmap class, as shown in the following code:

    class Bitmap: public iObject { public: Bitmap( const int W, const int H) { size_t Size = W * H * 3; if ( !Size ) { return; } FWidth = W; FHeight = H; FBitmapData = (ubyte*)malloc( Size ); memset(FBitmapData, 0xFF, Size); } virtual ~Bitmap() { free(FBitmapData); } void Load2DImage( clPtr<iIStream> Stream ) { free( FBitmapData ); FBitmapData = read_bmp_mem(Stream->MapStream(), &FWidth, &FHeight ); } …

  2. Image dimensions and raw pixel data are set as follows:

    int FWidth; int FHeight;

  3. Here we use a C-style array:

    ubyte* FBitmapData; };

    The read_bmp_mem() function is used once again, but this time the memory buffer comes from an iIStream object.

    We add the Texture class to handle all of the OpenGL complexities, but right now we simply render the instance of a Bitmap class.

  4. Next, we implement the asynchronous loading operation:

    class LoadOp_Image: public iTask { public: LoadOp_Image( clPtr<Bitmap> Bmp, clPtr<iIStream> IStream ):FBmp( Bmp ),
    FStream( IStream ) {} virtual void Run() { FBmp->Load2DImage( FStream ); g_Events->EnqueueCapsule(new LoadCompleteCapsule(FBmp) ); } private: clPtr<Bitmap> FBmp; clPtr<iIStream> FStream; };

  5. The LoadCompleteCapsule class is a iAsyncCapsule-derived class that has the overriden Run() method:

    class LoadCompleteCapsule: public iAsyncCapsule { public: LoadCompleteCapsule(clPtr<Bitmap> Bmp): FBmp(Bmp) {} virtual void Invoke() { // … copy FBmp to g_FrameBuffer … } private: clPtr<Bitmap> FBmp; };

  6. To load a Bitmap object, we implement the following function:

    clPtr<Bitmap> LoadImg( const std::string& FileName ) { clPtr<iIStream> IStream = g_FS->CreateReader(FileName); clPtr<Bitmap> Bmp = new Bitmap(1, 1); g_Loader->AddTask( new LoadOp_Image( Bmp, IStream ) ); return Bmp; }

  7. We use three global objects: the filesystem g_FS, the event queue g_Events, and the loader thread g_Loader. We initialize them at the beginning of our program. At first, we start FileSystem:

    g_FS = new FileSystem(); g_FS->Mount(".");

  8. The iAsyncQueue and WorkerThread objects are created:

    g_Events = new iAsyncQueue(); g_Loader = new WorkerThread(); g_Loader->Start( iThread::Priority_Normal );

  9. Finally, we can load the bitmap:

    clPtr<Bitmap> Bmp = LoadImg("test.bmp");

At this point Bmp is a ready-to-use object that will be automatically updated on another thread. Of course, it is not thread-safe to use the Bmp->FBitmapData, since it might be destroyed while we read it, or only partially updated. To overcome these difficulties, we have to introduce so-called proxy objects.

There's more

The complete example can be found in 3_AsyncTextures. It implements the asynchronous images loading technique described in this recipe.

Storing application data

An application should be able to save its temporary and persistent data. Sometimes data should be written into a folder on external storage accessible by other applications. Let's find out how to get the path to this folder on Android and Windows, and do this in a portable way.

Getting ready

If your Android smartphone unmounts its external storage while connected to a desktop computer, make sure you disconnect it and wait for the storage to be remounted.

How to do it...

  1. We need to write some Java code to accomplish this task. First, we will ask the Environment for the external storage directory and its suffix, so we can distinguish our data from other applications:

    protected String GetDefaultExternalStoragePrefix() { String Suffix = "/external_sd/Android/data/"; return Environment.getExternalStorageDirectory().getPath() +Suffix +
    getApplication().getPackageName(); }

    The Suffix value can be chosen at will. You can use whatever value you desire.

  2. This is quite simple; however, we have to perform some additional checks to make sure this path is really there. On some devices, for example, without external storage, it will be unavailable.

    String ExternalStoragePrefix = GetDefaultExternalStoragePrefix(); String state = Environment.getExternalStorageState();

  3. Check, if the storage is mounted and can be written to:

    if ( !Environment.MEDIA_MOUNTED.equals( state ) ||
    Environment.MEDIA_MOUNTED_READ_ONLY.equals( state ) ) { ExternalStoragePrefix = this.getDir(getApplication().getPackageName(),
    MODE_PRIVATE).getPath(); }

  4. Check if the storage is writable:

    try { new File( ExternalStoragePrefix ).mkdirs(); File F = new File(ExternalStoragePrefix + "/engine.log" ); F.createNewFile(); F.delete(); } catch (IOException e) { Log.e( "App6", "Falling back to internal storage" ); ExternalStoragePrefix = this.getDir(getApplication().getPackageName(),
    MODE_PRIVATE).getPath(); }

  5. Pass the path to our C++ code:

    OnCreateNative( ExternalStoragePrefix ); public static native void OnCreateNative(StringExternalStorage);

How it works...

Native code implements the JNI call OnCreateNative() this way:

extern std::string g_ExternalStorage; extern "C" { JNIEXPORT void JNICALLJava_com_packtpub_ndkcookbook_app6_App6Activity_
OnCreateNative(JNIEnv* env, jobject obj, jstring Path ) { g_ExternalStorage = ConvertJString( env, Path ); OnStart(); } }

There is also a small helper function to convert Java strings to std::string, which we will use frequently:

std::string ConvertJString(JNIEnv* env, jstring str) { if ( !str ) std::string(); const jsize len = env->GetStringUTFLength(str); const char* strChars = env->GetStringUTFChars(str,(jboolean *)0); std::string Result(strChars, len); env->ReleaseStringUTFChars(str, strChars); return Result; }

Check the application 6_StoringApplicationData from the code bundle of the book. On Android, it will output a line similar to the following into the system log:

I/App6 (27043): External storage path:/storage/emulated/0/external_sd/

On Windows, it will print the following into the application console:

External storage path: C:\Users\Author\Documents\ndkcookbook\App6

There's more...

Don't forget to add the WRITE_EXTERNAL_STORAGE permission to your AndroidManifest.xml for your application to be able to write to the external storage:


Otherwise, the previous code will always fall back to the internal storage.


This article showed us asynchronous file handling, resource proxies, and resource compression. Android program packages are simply archive files with an .apk extension, compressed with a familiar ZIP algorithm. To allow reading the application's resource files directly from .apk, we have to decompress the .zip format using the zlib library. Another important topic covered is the virtual filesystem concept, which allows us to abstract the underlying OS files and folders structure, and share resources between Android and PC versions of our application.

Resources for Article :

You've been reading an excerpt of:

Android NDK Game Development Cookbook

Explore Title