Building Android (Must know)

Exclusive offer: get 50% off this eBook here
Instant Android Systems Development How-to [Instant]

Instant Android Systems Development How-to [Instant] — Save 50%

Get your hands dirty with Android Systems Development through carefully thought-out source code examples with this book and ebook

$14.99    $7.50
by Earlence Fernandes | September 2013 | Open Source

This article by Earlence Fernandes, the author of the book Instant Android Systems Development How-to, sets up your build computer and instructs you on how to download and build the Android operating system from scratch.

(For more resources related to this topic, see here.)

Getting ready

You need Ubuntu 10.04 LTS or later (Mac OS X is also supported by the build system, but we will be using Ubuntu for this article). This is the supported build operating system, and the one for which you will get the most help from the online community. In my examples, I use Ubuntu 11.04, which is also reasonably well supported. You need approximately 6 GB of free space for the Android code files. For a complete build, you need 25 GB of free space. If you are using Linux in a virtual machine, make sure the RAM or the swap size is at least 16 GB, and you have 30 GB of disk space to complete the build.

As of Android Versions 2.3 (Gingerbread) and later, building the system is only possible on 64-bit computers. Using 32-bit machines is still possible if you work with Froyo (Android 2.2). However, you can still build later versions on a 32-bit computer using a few "hacks" on the build scripts that I will describe later.

The following steps outline the process needed to set up a build environment and compile the Android framework and kernel:

  • Setting up a build environment
  • Downloading the Android framework sources
  • Building the Android framework
  • Building a custom kernel

In general, your (Ubuntu Linux) build computer needs the following:

  • Git 1.7 or newer (GIT is a source code management tool), JDK 6 to build Gingerbread and later versions, or JDK 5 to build Froyo and older versions
  • Python 2.5 – 2.7
  • GNU Make 3.81 – 3.82

How to do it...

We will first set up the build environment with the help of the following steps:

All of the following steps are targeted towards 64-bit Ubuntu.

  1. Install the required JDK by executing the following command:

    JDK6
    sudo add-apt-repository "deb http: //archive.canonical.com/ lucid
    partner"
    sudo apt-get update
    sudo apt-get install sun-java6-jdk
    JDK5
    sudo add-apt-repository "deb http: //archive.ubuntu.com/ubuntu
    hardy main multiverse"
    sudo add-apt-repository "deb http: //archive.ubuntu.com/ubuntu
    hardy-updates main multiverse"
    sudo apt-get update
    sudo apt-get install sun-java5-jdk

  2. Install the required library dependencies:

    sudo apt-get install git-core gnupg flex bison gperf build-
    essential \
    zip curl zlib1g-dev libc6-dev lib32ncurses5-dev ia32-libs \
    x11proto-core-dev libx11-dev lib32readline5-dev lib32z-dev \
    libgl1-mesa-dev g++-multilib mingw32 tofrodos python-markdown \
    libxml2-utils xsltproc

  3. [OPTIONAL]. On Ubuntu 10.10, a symlink is not created between libGL.so.1 and libGL.so, which sometimes causes the build process to fail:

    sudo ln -s /usr/lib32/mesa/libGL.so.1 /usr/lib32/mesa/libGL.so

  4. [OPTIONAL] On Ubuntu 11.10, an extra dependency is

    sudo apt-get install libx11-dev:i386

  5. Now, we will download the Android sources from Google's repository.
  6. Install repo. Make sure you have a /bin directory and that it exists in your PATH variable:

    mkdir ~/bin
    PATH=~/bin:$PATH
    curl https: //dl-ssl.google.com/dl/googlesource/git-repo/repo > ~/
    bin/repo
    chmod a+x ~/bin/repo

    Repo is a python script used to download the Android sources, among other tasks. It is designed to work on top of GIT.

  7. Initialize repo.
  8. In this step, you need to decide the branch of the Android source you wish to download. If you wish to make use of Gerrit, which is the source code reviewing tool used, make sure you have a live Google mail address. You will be prompted to use this e-mail address when repo initializes.
  9. Create a working directory on your local machine. We will call this

    mkdir android_src
    cd android_src

  10. The following command will initialize repo to download the "master" branch:

    repo init -u https://android.googlesource.com/platform/manifest

  11. The following command will initialize repo to download the Gingerbread 2.3.4 branch:

    repo init -u https: //android.googlesource.com/platform/manifest
    -b android-2.3.4_r1

    The -b switch is used to specify the branch you wish to download.

  12. Once repo is configured, we are ready to obtain the source files. The format of the command is as follows:

    repo sync -jX

    -jX is optional, and is used for parallel fetch.

  13. The following command will sync all the necessary source files for the Android framework. Note that these steps are only to download the Android framework files.Kernel download is a separate process.

    repo sync -j16

    The source code access is anonymous, that is, you do not need to be registered with Google to be able to download the source code. The servers allocate a fixed quota to each IP address that accesses the source code. This is to protect the servers against excessive download traffic. If you happen to be behind a NAT and share an IP address with others, who also wish to download the code, you may encounter error messages from the source code servers warning about excessive usage. In this case, you can solve the problem with authenticated access. In this method, you get a separate quota based on your user ID, generated by the password generator system. The password generator and associated instructions are available at https://android.googlesource.com/new-password.

  14. Once you have obtained a user ID/password and set up your system appropriately, you can force authentication by using the following command:

    repo init -u https://android.googlesource.com/a/platform/manifest

    Notice the /a/ in the URI. This indicates authenticated access.

    Proxy issues

    If you are downloading from behind a proxy, set the following environment variables:

    export HTTP_PROXY=http://<proxy_user_id>:<proxy_
    password>@<proxy_server>:<proxy_port>
    export HTTPS_PROXY=http://<proxy_user_
    id>:<proxy_password>@<proxy_server>:<proxy_port>

Next, we describe the steps needed to build the Android framework sources:

  1. Initialize the terminal environment.
  2. Certain build-time tools need to be included in your current terminal environment. So, navigate to your source directory:

    cd android_src/
    source build/envsetup.sh

    The sources can be built for various targets. Each target descriptor has the BUILD-BUILDTYPE format:

    • BUILD: Refers to a specific combination of the source code for a certain device. For example, full_maguro targets Galaxy Nexus or generic targets the emulator.
    • BUILDTYPE: This can be one of the following three values:
      • user: Suitable for production builds
      • userdebug: Similar to user, with with root access in ADB for easier debugging
      • eng: Development build only
  3. We will be building for the emulator in our current example. Issue the following command to do so:

    lunch full-eng

    To actually build the code, we will use make. The format is as follows:

    make -jX

    Where X indicates the number of parallel builds. The usual rule is: X is the number of CPU cores + 2.

    This is an experimental formula, and the reader should feel free to test it with different values.

  4. To build the code:

    make -j6

    Now, we must wait till the build is complete. Depending on your system's specifications, this can take anywhere between 20 minutes and 1 hour. At the end of a successful build, the output looks similar to the following (note that this may vary depending on your target):

    ...
    target Dex: SystemUI
    Copying: out/target/common/obj/APPS/SystemUI_intermediates/
    noproguard.classes.dex
    target Package: SystemUI (out/target/product/generic/obj/APPS/
    SystemUI_intermediates/package.apk)
    'out/target/common/obj/APPS/SystemUI_intermediates//classes.dex'
    as 'classes.dex'...
    Install: out/target/product/generic/system/app/SystemUI.apk
    Finding NOTICE files: out/target/product/generic/obj/NOTICE_FILES/
    hash-timestamp
    Combining NOTICE files: out/target/product/generic/obj/NOTICE.html
    Target system fs image: out/target/product/generic/obj/PACKAGING/
    systemimage_intermediates/system.img
    Install system fs image: out/target/product/generic/system.img
    Installed file list: out/target/product/generic/installed-files.
    txt
    DroidDoc took 440 sec. to write docs to out/target/common/docs/
    doc-comment-check

    A better check for a successful build is to examine the newly created files inside the following directory.

    The build produces a few main files inside android_src/out/target/product/<DEVICE>/, which are as follows:

    • system.img: The system image file
    • boot.img: Contains the kernel
    • recovery.img: Contains code for recovery partition of the device

    In the case of an emulator build, the preceding files will appear at android_src/out/target/product/generic/.

  5. Now, we can test our build simply by issuing the emulator command:

    emulator

    This launches an Android emulator, as shown in the following screenshot, running the code we've just built:

    The code we've downloaded contains prebuilt Linux kernels for each supported target. If you only wish to change the framework files, you can use the prebuilt kernels, which are automatically included in the build images. If you are making specific changes to the kernel, you will have to obtain a specific kernel and build it separately (shown here), which is explained later:

    Faster Builds – CCACHE

    The framework code contains C language and Java code. The majority of the C language code exists as shared objects that are built during the build process. If you issue the make clean command, which deletes all the built code (simply deleting the build output directory has the same effect as well) and then rebuild, it will take a significant amount of time. If no changes were made to these shared libraries, the build time can be sped up with CCACHE, which is a compiler cache.

  6. In the root of the source directory android_src/, use the following command:

    export USE_CCACHE=1
    export CCACHE_DIR=<PATH_TO_YOUR_CCACHE_DIR>

    To set a cache size:

    prebuilt/linux-x86/ccache/ccache -M 50G

    This reserves a cache size of 50 GB.

    To watch how the cache is used during the build process, use the following command (navigate to your source directory in another terminal):

    watch -n1 -d prebuilt/linux-x86/ccache/ccache -s

    In this part, we will obtain the sources and build the goldfish emulator kernel. Building kernels for devices is done in a similar way.

    goldfish is the name of the kernel modified for the Android QEMU-based emulator.

  7. Get the kernel sources:

    Create a subdirectory of android_src:

    mkdir kernel_code
    cd kernel_code
    git clone https: //android.googlesource.com/kernel/goldfish.git
    git branch -r

    This will clone goldfish.git into a folder named goldfish (created automatically) and then list the remote branches available. The output should look like the following (this is seen after the execution of the git branch):

    origin/HEAD -> origin/master
    origin/android-goldfish-2.6.29
    origin/linux-goldfish-3.0-wip
    origin/master

    Here, in the following command, we notice origin/android-goldfish-2.6.29, which is the kernel we wish to obtain:

    cd goldfish
    git checkout --track -b android-goldfish-2.6.29 origin/android-
    goldfish-2.6.29

    This will obtain the kernel code:

    1. Set up the build environment.
    2. We need to initialize the terminal environment by updating the system PATH variable to point to a cross compiler which will be used to compile the Linux kernel. This cross compiler is already available as a prebuilt binary distributed with the Android framework sources:

      export PATH=<PATH_TO_YOUR_ANDROID_SRC_DIR>/prebuilt/
      linux-x86/toolchain/arm-eabi-4.4.3/bin:$PATH

    3. Run an emulator (you may choose to run the emulator with the system image that we just built earlier. We need this to obtain the kernel configuration file. Instead of manually configuring it, we choose to pull the config file of a running kernel.

      Make sure ADB is still in your path. It will be in your PATH variable if you haven't closed the terminal window since building the framework code, otherwise execute the following steps sequentially.

      (Note that you have to change directory to ANDROID_SRC to execute the following command).

      source build/envsetup.sh
      lunch full_eng
      adb pull /proc/config.gz
      gunzip config.gz
      cp config .config

      The preceding command will copy the config file of the running kernel into our kernel build tree.

    4. Start the compilation process:

      export ARCH=arm
      export SUBARCH=arm
      make

      If the following comes up:

      Misc devices (MISC_DEVICES) [Y/n/?] y
      Android pmem allocator (ANDROID_PMEM) [Y/n] y
      Enclosure Services (ENCLOSURE_SERVICES) [N/y/?] n
      Kernel Debugger Core (KERNEL_DEBUGGER_CORE) [N/y/?] n
      UID based statistics tracking exported to /proc/uid_stat
      (UID_STAT) [N/y] n
      Virtual Device for QEMU tracing (QEMU_TRACE) [Y/n/?] y
      Virtual Device for QEMU pipes (QEMU_PIPE) [N/y/?] (NEW)

      Enter y as the answer. This is some additional Android-specific configuration needed for the build.

    5. Now we have to wait till the build is complete. The final lines of the build output should look like the following (note that this can change depending on your target):

      ...
      LD vmlinux
      SYSMAP System.map
      SYSMAP .tmp_System.map
      OBJCOPY arch/arm/boot/Image
      Kernel: arch/arm/boot/Image is ready
      AS arch/arm/boot/compressed/head.o
      GZIP arch/arm/boot/compressed/piggy.gz
      AS arch/arm/boot/compressed/piggy.o
      CC arch/arm/boot/compressed/misc.o
      LD arch/arm/boot/compressed/vmlinux
      OBJCOPY arch/arm/boot/zImage
      Kernel: arch/arm/boot/zImage is ready

      As the last line states, the new zImage is available inside arch/arm/ boot/.

    6. To test it, we boot the emulator with this newly built image.
    7. Copy zImage to an appropriate directory. I just copied it to android_src/:

      emulator -kernel zImage

    8. To verify that the emulator is indeed running our kernel, use the following command:

      adb shell
      # cat /proc/version

      The output will look like:

      Linux version 2.6.29-gef9c64a (earlence@earlence-
      Satellite-L650) (gcc version 4.4.3 (GCC) ) #1 Mon Jun 4
      16:35:00 CEST 2012

      This is our custom kernel, since we observe the custom build string (earlence@earlence-Satellite-L650) present as well as the time of the compilation. The build string will be the name of your computer.

    9. Once the emulator has booted up, you will see a window similar to the following:

Following are the steps required to build the framework on a 32-bit system:

  1. Make the following simple changes to build Gingerbread on 32-bit Ubuntu. Note that these steps assume that you have set up the system for a Froyo build. Assuming a Froyo build computer setup, the following steps guide you on incrementally making changes such that Gingerbread and later builds are possible. To set up for Froyo, please follow the steps explained at http://source.android.com/source/initializing.html. In build/core/main.mk, change ifneq (64,$(findstring 64,$(build_arch))) to ifneq (i686,$(findstring i686,$(build_arch))).

    Note that there are two changes on that line.

  2. In the following files:
    • external/clearsilver/cgi/Android.mk
    • external/clearsilver/java-jni/Android.mk
    • external/clearsilver/util/Android.mk
    • external/clearsilver/cs/Android.mk

    change:
    LOCAL_CFLAGS += -m64
    LOCAL_LDFLAGS += -m64

    to:
    LOCAL_CFLAGS += -m32
    LOCAL_LDFLAGS += -m32

  3. Install the following packages (in addition to the packages you must have installed for the Froyo build):

    sudo apt-get install lib64z1-dev libc6-dev-amd64 g++-multilib
    lib64stdc++6

  4. Install Java 1.6 using the following command:

    sudo apt-get install sun-java6-jdk

Summary

The Android build system is a combination of several standard tools and custom wrappers. Repo is one such wrapper script that takes care of GIT operations and makes it easier for us to work with the Android sources.

The kernel trees are maintained separately from the framework source trees. Hence, if you need to make customizations to a particular kernel, you will have to download and build it separately. The keen reader may be wondering how we are able to run the emulator if we never built a kernel in when we just compiled the framework. Android framework sources include prebuilt binaries for certain targets. These binaries are located in the /prebuilt directory under the framework source root directory.

The kernel build process is more or less the same as building kernels for desktop systems.

There are only a few Android-specific compilation switches, which we have shown to be easily configurable given an existing configuration file for the intended target.

The sources consist of C/C++ and Java code. The framework does not include the kernel sources, as these are maintained in a separate GIT tree. In the next recipe, we will explain the framework code organization. It is important to understand how and where to make changes while developing custom builds.

Resources for Article:


Further resources on this subject:


 

Instant Android Systems Development How-to [Instant] Get your hands dirty with Android Systems Development through carefully thought-out source code examples with this book and ebook
Published: May 2013
eBook Price: $14.99
See more
Select your format and quantity:

About the Author :


Earlence Fernandes

Earlence Fernandes is pursuing a Ph.D. degree in Systems Security from the University of Michigan, Ann Arbor, USA. Previously, he was a Scientific Programmer at Vrije Universiteit, Amsterdam, where he built several Android research prototypes for various cutting-edge security mechanisms. He also co-authored several scientific papers on the topic. He has been playing with Android since 2008 and is an active member of the Android Google Groups for platform internals.

Books From Packt


 Android 3.0 Application Development Cookbook
Android 3.0 Application Development Cookbook

 Android Database Programming
Android Database Programming

Android 4: New Features for Application Development
Android 4: New Features for Application Development

Android Native Development Kit Cookbook
Android Native Development Kit Cookbook

Android User Interface Development: Beginner's Guide
Android User Interface Development: Beginner's Guide

Android Application Programming with OpenCV
Android Application Programming with OpenCV

 Android NDK Beginner’s Guide
Android NDK Beginner’s Guide

AndEngine for Android Game Development Cookbook
AndEngine for Android Game Development Cookbook


No votes yet

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
M
X
T
v
b
i
Enter the code without spaces and pay attention to upper/lower case.
Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software