Using memcached with Python

Exclusive offer: get 50% off this eBook here
Getting Started with Memcached

Getting Started with Memcached — Save 50%

Speed up and scale out your web applications with Memcached with this book and ebook

$14.99    $7.50
by Ahmed Soliman | November 2013 | Open Source Web Development

This article by Ahmed Soliman, the author of Getting started with memcached, is about how to connect to your memcached server(s) from your Python application.

(For more resources related to this topic, see here.)

If you want to make such a connection, there are several clients available for you. The most popular ones are:

  • python-memcached: This is a pure-python implementation of the memcached client (implemented 100 percent in Python). It offers good performance and is extremely simple to install and use.
  • pylibmc: This is a Python wrapper on the libmemcached C/C++ library, it offers excellent performance, thread safety, and light memory usage, yet it's not as simple as python-memcached to install, since you will need to have the libmemcached library compiled and installed on your system.
  • Twisted memcache: This client is part of the Python twisted event-driven networking engine for Python. It offers a reactive code structure and excellent performance as well, but it is not as simple to use as pylibmc or python-memcached but it fits perfectly if your entire application is built on twisted.

In this recipe, we will be using python-memcached for the sake of simplicity and since other clients have almost the same API, it does not make much difference from a developer's perspective.

Getting ready

It's always a good idea to create virtualenv for your experiments to keep your experiments contained and not to pollute the global system with the packages you install.

You can create virtualenv easily:

virtualenv memcache_experiments source memcache_experiments/bin/activate

We will need to install python-memcached first, using the pip package manager on our system:

sudo pip install python-memcached

How to do it...

  1. Let's start with a simple set and get script:

    import memcache client = memcache.Client([('127.0.0.1', 11211)]) sample_obj = {"name": "Soliman", "lang": "Python"} client.set("sample_user", sample_obj, time=15) print "Stored to memcached, will auto-expire after 15 seconds" print client.get("sample_user")

  2. Save the script into a file called memcache_test1.py and run it using python memcache_test1.py.
  3. On running the script you should see something like the following:

    Stored to memcached, will auto-expire after 15 seconds {'lang': 'Python', 'name': 'Soliman'}

  4. Let's now try other memcached features:

    import memcache client = memcache.Client([('127.0.0.1', 11211)]) client.set("counter", "10") client.incr("counter") print "Counter was incremented on the server by 1, now it's %s" % client.get("counter") client.incr("counter", 9) print "Counter was incremented on the server by 9, now it's %s" % client.get("counter") client.decr("counter") print "Counter was decremented on the server by 1, now it's %s" % client.get("counter")

The output of the script looks like the following:

Counter was incremented on the server by 1, now it's 11 Counter was incremented on the server by 9, now it's 20 Counter was decremented on the server by 1, now it's 19

The incr and decr methods allow you to specify a delta value or to by default increment/decrement by 1.

Alright, now let's sync a Python dict to memcached with a certain prefix:

import memcache client = memcache.Client([('127.0.0.1', 11211)]) data = {"some_key1": "value1", "some_key2": "value2"} client.set_multi(data, time=15, key_prefix="pfx_") print "saved the dict with prefix pfx_" print "getting one key: %s" % client.get("pfx_some_key1") print "Getting all values: %s" % client.get_multi(["some_key1", "some_ key2"], key_prefix="pfx_")

How it works...

In this script, we are connecting to the memcached server(s) using the Client constructor, and then we are using the set method to store a standard Python dict as the value of the "sample_user" key. After that we use the get method to retrieve the value.

The client automatically serialized the python dict to memcached and deserialized the object after getting it from memcached server.

In the second script, we are playing with some of the features we never tried in the memcached server. The incr and decr are methods that allow you to increment and decrement integer values directly on the server automatically.

Then, we are using an awesome feature that we also didn't play with before, that is get/set_multi that allows us to set or get multiple key/values at a single request. Also it allows us to add a certain prefix to all the keys during the set or get operations.

The output of the last script should look like the following:

saved the dict with prefix pfx_ getting one key: value1 Getting all values: {'some_key1': 'value1', 'some_key2': 'value2'}

There's more...

In the Client constructor, we specified the server hostname and port in a tuple (host, port) and passed that in a list of servers. This allows you to connect to a cluster of memcached servers by adding more servers to this list. For example:

client = memcache.Client([('host1', 1121), ('host2', 1121), ('host3', 1122)])

Also, you can also specify custom picklers/unpicklers to tell the memcached client how to serialize or de-serialize the Python types using your custom algorithm.

Summary

Thus we learned how to connect to memcached servers from your python application.

Resources for Article:


Further resources on this subject:


Getting Started with Memcached Speed up and scale out your web applications with Memcached with this book and ebook
Published: November 2013
eBook Price: $14.99
Book Price: $24.99
See more
Select your format and quantity:

About the Author :


Ahmed Soliman

Ahmed Soliman is an entrepreneur and software and systems engineer coming from a diverse background of highly scalable applications design, mission-critical systems, asynchronous data analytics, social networks design, reactive distributed systems, and systems administration and engineering. He has also published a technology patent in distributed computer-based virtual laboratories and designed numerous large-scale distributed systems for massive-scale enterprise customers.

A software engineer at heart, he is experienced in over 10 programming languages but most recently he was busy designing and writing applications in Python, Ruby, and Scala for several customers. He is also an opensource evangelist and activist. He contributed and maintained several open source projects on the Web.

Ahmed is a co-founder in Cloud Niners Ltd., a software and services company focusing on highly scalable cloud-based applications that have been delivering private and public cloud computing services to customers in the MEA region on different platforms and technologies.

Books From Packt


Lift Application Development Cookbook
Lift Application Development Cookbook

 Instant Lift Web Applications How-to [Instant]
Instant Lift Web Applications How-to [Instant]

 Instant Effective Caching with Ehcache [Instant]
Instant Effective Caching with Ehcache [Instant]

 Instant Redis Optimization How-to [Instant]
Instant Redis Optimization How-to [Instant]

Firebug 1.5: Editing, Debugging, and Monitoring Web Pages
Firebug 1.5: Editing, Debugging, and Monitoring Web Pages

Django 1.1 Testing and Debugging
Django 1.1 Testing and Debugging

Practical Maya Programming with Python
Practical Maya Programming with Python

Spring Python 1.1
Spring Python 1.1


No votes yet

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
m
N
z
p
V
Y
Enter the code without spaces and pay attention to upper/lower case.
Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software