Create a Local Ubuntu Repository using Apt-Mirror and Apt-Cacher

Exclusive offer: get 50% off this eBook here
Asterisk Gateway Interface 1.4 and 1.6 Programming

Asterisk Gateway Interface 1.4 and 1.6 Programming — Save 50%

Design and develop Asterisk-based VoIP telephony platforms and services using PHP and PHPAGI

$23.99    $12.00
by Christer Edwards | August 2009 | Linux Servers

How can a company or organization minimize bandwidth costs when maintaining multiple Ubuntu installations? With bandwidth becoming the currency of the new millennium, being responsible with the bandwidth you have can be a real concern. In this article by Christer Edwards, we will learn how to create, maintain and make available a local Ubuntu repository mirror, allowing you to save bandwidth and improve network efficiency with each machine you add to your network.

(For more resources on Ubuntu, see here.)

Background

Before I begin, let me give you some background into what prompted me to initially come up with these solutions.

I used to volunteer at local university user groups whenever they held "installfests". I always enjoyed offering my support and expertise with new Linux users. I have to say, the distribution that we helped deploy the most was Ubuntu. The fact that Ubuntu's parent company, Canonical, provided us with pressed CDs didn't hurt! Despite having more CDs than we could handle we still regularly ran into the same problem. Bandwidth. Fetching updates and security errata was always our bottleneck, consistently slowing down our deployments.

Imagine you are in a conference room with dozens of other Linux users, each of them trying to use the internet to fetch updates. Imagine how slow and bogged down that internet connection would be. If you've ever been to a large tech conference you've probably experienced this first hand!

After a few of these "installfests" I started looking for a solution to our bandwidth issue. I knew that all of these machines were downloading the same updates and security errata, therefore duplicating work that another user had already done. It just seemed so inefficient. There had to be a better way. Eventually I came across two solutions, which I will outline below.

Apt-Mirror

The first method makes use of a tool called “apt-mirror”. It is a perl-based utility for downloading and mirroring the entire contents of a public repository. This may likely include packages that you don't use and will not use, but anything stored in a public repository will also be stored in your mirror.

To configure apt-mirror you will need the following:

  • apt-mirror package (sudo aptitude install apt-mirror)
  • apache2 package (sudo aptitude install apache2)
  • roughly 15G of storage per release, per architecture

Once these requirements are met you can begin configuring the apt-mirror tool. The main items that you'll need to define are:

  • storage location (base_path)
  • number of download threads (nthreads)
  • release(s) and architecture(s) you want

The configuration is done in the /etc/apt/mirror.list file. A default config was put in place when you installed the package, but it is generic. You'll need to update the values as mentioned above.

Below I have included a complete configuration which will mirror Ubuntu 8.04 LTS for both 32 and 64 bit installations. It will require nearly 30G of storage space, which I will be putting on a removable drive. The drive will be mounted at /media/STORAGE/.

# apt-mirror configuration file



##

## The default configuration options (uncomment and change to override)

##

#

set base_path /media/STORAGE/

# set mirror_path $base_path/mirror

# set skel_path $base_path/skel

# set var_path $base_path/var

#

# set defaultarch <running host architecture>

set nthreads 20

#



# 8.04 "hardy" i386 mirror

deb-i386 http://us.archive.ubuntu.com/ubuntu hardy main restricted universe
multiverse

deb-i386 http://us.archive.ubuntu.com/ubuntu hardy-updates main restricted
universe multiverse

deb-i386 http://us.archive.ubuntu.com/ubuntu hardy-security main restricted
universe multiverse

deb-i386 http://us.archive.ubuntu.com/ubuntu hardy-backports main restricted
universe multiverse

deb-i386 http://us.archive.ubuntu.com/ubuntu hardy-proposed main restricted
universe multiverse



deb-i386 http://us.archive.ubuntu.com/ubuntu hardy main/debian-installer
restricted/debian-installer universe/debian-installer multiverse/debian-installer

deb-i386 http://packages.medibuntu.org/ hardy free non-free



# 8.04 "hardy" amd64 mirror

deb-amd64 http://us.archive.ubuntu.com/ubuntu hardy main restricted universe
multiverse

deb-amd64 http://us.archive.ubuntu.com/ubuntu hardy-updates main restricted
universe multiverse

deb-amd64 http://us.archive.ubuntu.com/ubuntu hardy-security main restricted
universe multiverse

deb-amd64 http://us.archive.ubuntu.com/ubuntu hardy-backports main restricted
universe multiverse

deb-amd64 http://us.archive.ubuntu.com/ubuntu hardy-proposed main restricted
universe multiverse



deb-amd64 http://us.archive.ubuntu.com/ubuntu hardy main/debian-installer
restricted/debian-installer universe/debian-installer multiverse/debian-installer

deb-amd64 http://packages.medibuntu.org/ hardy free non-free



# Cleaning section

clean http://us.archive.ubuntu.com/

clean http://packages.medibuntu.org/

It should be noted that each of the repository lines within the file should begin with deb-i386 or deb-amd64. Formatting may have changed based on the web formatting.

Once your configuration is saved you can begin to populate your mirror by running the command:

apt-mirror

Be warned that, based on your internet connection speeds, this could take quite a long time. Hours, if not more than a day for the initial 30G to download. Each time you run the command after the initial download will be much faster, as only incremental updates are downloaded.

You may be wondering if it is possible to cancel the transfer before it is finished and begin again where it left off. Yes, I have done this countless times and I have not run into any issues.

You should now have a copy of the repository stored locally on your machine. In order for this to be available to other clients you'll need to share the contents over http. This is why we listed Apache as a requirement above. In my example configuration above I mirrored the repository on a removable drive, and mounted it at /media/STORAGE. What I need to do now is make that address available over the web. This can be done by way of a symbolic link.

cd /var/www/
sudo ln -s /media/STORAGE/mirror/us.archive.ubuntu.com/ubuntu/ ubuntu

The commands above will tell the filesystem to follow any requests for “ubuntu” back up to the mounted external drive, where it will find your mirrored contents. If you have any problems with this linking double-check your paths (as compared to those suggested here) and make sure your link points to the ubuntu directory, which is a subdirectory of the mirror you pulled from. If you point anywhere below this point your clients will not properly be able to find the contents.

The additional task of keeping this newly downloaded mirror updated with the latest updates can be automated by way of a cron job. By activating the cron job your machine will automatically run the apt-mirror command on a regular daily basis, keeping your mirror up to date without any additional effort on your part.

To activate the automated cron job, edit the file /etc/cron.d/apt-mirror. There will be a sample entry in the file. Simply uncomment the line and (optionally) change the “4” to an hour more convenient for you. If left with its defaults it will run the apt-mirror command, updating and synchronizing your mirror, every morning at 4:00am.

The final step needed, now that you have your repository created, shared over http and configured to automatically update, is to configure your clients to use it. Ubuntu clients define where they should grab errata and security updates in the /etc/apt/sources.list file. This will usually point to the default or regional mirror. To update your clients to use your local mirror instead you'll need to edit the /etc/apt/sources.list and comment the existing entries (you may want to revert to them later!)

Once you've commented the entries you can create new entries, this time pointing to your mirror. A sample entry pointing to a local mirror might look something like this:

deb http://192.168.0.10/ubuntu hardy main restricted universe multiverse
deb http://192.168.0.10/ubuntu hardy-updates main restricted universe multiverse
deb http://192.168.0.10/ubuntu hardy-security main restricted universe multiverse

Basically what you are doing is recreating your existing entries but replacing the archive.ubuntu.com with the IP of your local mirror. As long as the mirrored contents are made available over http on the mirror-server itself you should have no problems. If you do run into problems check your apache logs for details.

By following these simple steps you'll have your own private (even portable!) Ubuntu repository available within your LAN, saving you future bandwidth and avoiding redundant downloads.

Asterisk Gateway Interface 1.4 and 1.6 Programming Design and develop Asterisk-based VoIP telephony platforms and services using PHP and PHPAGI
Published: February 2009
eBook Price: $23.99
Book Price: $39.99
See more
Select your format and quantity:

(For more resources on Ubuntu, see here.)

Apt-Cacher

The second method of mirroring Ubuntu packages is through the use of a tool called Apt-Cacher.

Apt-Cacher differs from apt-mirror in that it does not mirror the entire repository contents, but simply caches (stores) any packages requested by clients within the network. In other words, it works as a middle man between the public repository and the LAN clients, sharing any common downloads internally.

Each time any client requests a package, whether it be a security update or a new application to install, the apt-cache server will request the package from the public repository, store it and pass it back to the original requesting client. The package is then made available to any future requests from the internal network.

This method is best suited for those with minimal storage availability, or those that may simply want to make updates more efficient.

To configure this method you will need the following:

  • apt-cacher package (sudo aptitude install apt-cacher)
  • apache2 package (sudo aptitude install apache2)
  • available storage (size varies based on the number of requested packages stored)

Once you've installed the required packages you'll need to activate the service. This is done by editing the /etc/default/apt-cacher file and changing the value of AUTOSTART:

"AUTOSTART=0" to "AUTOSTART=1"

You can also limit access to the cache within the /etc/apt-cacher/apt-cacher.conf file. Update the value for allowed_hosts to match the individual hosts or subnets that you want to allow. Once these access rules are set you can start the caching system:

sudo /etc/init.d/apt-cacher restart

The final step in configuring apt-cacher is the configuration of the clients. Each machine within your network will need to be configured to work with this caching system, otherwise it will be ineffective.

The way that I prefer to do this is by defining an apt proxy setting within the /etc/apt/apt.conf.d/ directory. The way you can do this is by creating a new file called 90-apt-proxy.conf and entering the line:

Acquire::http::Proxy "http://repository-cache:3142";

Note, make sure you replace "repository-cache" with the IP address or URL of the machine you installed apt-cacher on.

Now, any time that client is able to access the URL (repository-cache:3142) it will use the cache. If the URL is not available it should default back to the entries in the /etc/apt/sources.list file. If it does not, and you see 404 errors or similar, simply comment out the "Acquire::http::Proxy" line that you populated in the file above.

This step should be repeated for any and all machines that you want to access the cache. With each additional system you configure you're saving potential bandwidth and adding to the pool of cached packages. There is no limit on the number of machines you can point to an apt-cache.

Summary

In conclusion, creating a mirror or caching updates can have a dramatic impact on the use of bandwidth within your organization. Whether it is a full blown mirror, holding every last package available, or simply a cache of requested updates, both of these tools are simple to configure and a breeze to maintain.

I highly recommend using one of these solutions if you have multiple Ubuntu installations.


Further resources on this subject:

 

Asterisk Gateway Interface 1.4 and 1.6 Programming Design and develop Asterisk-based VoIP telephony platforms and services using PHP and PHPAGI
Published: February 2009
eBook Price: $23.99
Book Price: $39.99
See more
Select your format and quantity:

About the Author :


Christer Edwards currently works for a US Department of Defense Contractor maintaining Linux and FreeBSD Systems. In a previous life he was a Technical Instructor developing and teaching courseware for Red Hat Enterprise. While it is the Enterprise  that pays the bills, his real passion is in the Linux community trenches. He has been a long-time contributor to Ubuntu where he has maintained a technical blog on the Ubuntu Planet for over three years. He was also key in organizing the Ubuntu LoCo project throughout the US. An ideal day for him is one spent quietly writing documentation and articles.

Books From Packt

Lighttpd
Lighttpd

Cacti 0.8 Network Monitoring
Cacti 0.8 Network Monitoring

trixbox CE 2.6
trixbox CE 2.6

Learning Nagios 3.0
Learning Nagios 3.0

FreePBX 2.5 Powerful Telephony Solutions
FreePBX 2.5 Powerful Telephony Solutions

Grails 1.1 Web Application Development
Grails 1.1 Web Application Development

Asterisk 1.4 – the Professional’s Guide
Asterisk 1.4 – the Professional’s Guide

Scalix: Linux Administrator’s Guide
Scalix: Linux Administrator’s Guide

Your rating: None Average: 4.5 (4 votes)
cache the updated package list by
Thank you for the great guide! Is there any way I could also cache the updated list of packages that I get using apt-get update?
Thanks. Very good and useful. by
Thanks. Very good and useful. As long as i spread ubuntu a bit more I will make use of it!
semicolon by
The semicolon after Acquire::http::Proxy "http://repository-cache:3142" has been added, Thanks.
Missing semicolon by
The line "Acquire::http::Proxy "http://repository-cache:3142"" should end with a semicolon, otherwise apt will be complaining about "Extra junk at end of file". Hope this saves someone 10 minutes to solve this ;)
Small issue with the mirror.list config above by
With the config above, pxe imaging will still fail. In order to succeed, you need to add 2 more lines (4 if you're mirroring both 64bit and 32bit repos). This line was already posted above... hardy main/debian-installer restricted/debian-installer universe/debian-installer multiverse/debian-installer These two lines also need to mirror... hardy-updates main/debian-installer restricted/debian-installer universe/debian-installer hardy-security main/debian-installer restricted/debian-installer universe/debian-installer
Help by
Hi Edwards, I read your post and find it very interesting and need your help about configuring my local repository. My problem is that: I use debarchiver to manage my repository. My dupload also work and put my package on the repository. But I cannot install this package using: aptitude update aptitude upgrade aptitude install name_of_paquet.deb My configuration is: Linux Ubuntu Hardy In my source.list I add this: http://ip.local stable main contrib non-free I notice that the folder http://192.168.0.201/debian/incoming/stable/, is empty and not updated. My question is: How can I do to get my package installed using aptitude install package_name I'm newbee in repository; Thx, Ghislain.

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
d
y
U
i
G
2
Enter the code without spaces and pay attention to upper/lower case.
Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software