Evan Weaver showed a benchmark of his memcached ruby gem outperforming the memcache-client gem by 20 times. Mike Perham then made the claim that memcache-client, being only 30-50% slower, is not really very slow.

I decided to do a real-world comparison of these two ruby libraries, mimicking how our back-end thrift services have many threads each handling service requests and contacting our memcache servers.

Setup

For this test, I really only cared about wallclock time, since I’m interested in the time a user has to wait. I performed these on a macbook pro x86, running ruby 1.8.7, using memcache server 1.2.8, memcache-client gem 1.7.1, and memcached gem version 0.12 with libmemcached 0.25.

I started the memcache server with a high max connection count so that this wouldn’t be an issue:

% memcached -c 8192


I quickly hacked together some simple testing code:

#!/usr/bin/env ruby
## compare_memcache_clients.rb

require 'rubygems'
require 'sequel'

class CompareMemcacheClients
 def pool_memcache_client(max_conns = 5)
   require 'memcache'
   Sequel::ConnectionPool.new(:max_connections => max_conns) { MemCache.new('127.0.0.1:11211')}
 end

 def pool_memcached(max_conns = 5)
   require 'memcached'
   Sequel::ConnectionPool.new(:max_connections => max_conns) {Memcached.new('127.0.0.1:11211')}
 end

 class MockPool
   def initialize(&proc)
     @proc = proc
   end

   def hold
     server = @proc.call
     yield server
   end
 end

 def not_pool_memcache_client()
   require 'memcache'
   MockPool.new() {MemCache.new('127.0.0.1:11211')}
 end

 def not_pool_memcached()
   require 'memcached'
   MockPool.new() {Memcached.new('127.0.0.1:11211')}
 end

 def time_test(pool, nthreads = 0)
   if nthreads == 0
     t0 = Time.now
     pool.hold do |server|
       server.get('foo')
     end

     puts "Took: #{Time.now - t0}"
   else
     @threads = []
     @timed_out = 0
     @times = []
     nthreads.times do |i|
       @threads << Thread.new do
         begin
           t0 = Time.now

           pool.hold do |server|
             server.set(’foo’, “bar#{i}”)
           end
           pool.hold do |server|
             server.get(’foo’)
           end
           pool.hold do |server|
             server.set(’foo’, “bar#{i}”)
           end
           pool.hold do |server|
             server.get(’foo’)
           end

           total_time = Time.now - t0
           #puts “Thread #{i} Took #{total_time}”
           @times << total_time
         rescue Sequel::Error::PoolTimeoutError
           @timed_out += 1
         end
       end
     end

     @threads.map{|t| t.join}

     puts “#{@timed_out} threads timed out”

     avg = std_dev = 0
     if @times.length > 0
       tot = 0.0
       @times.map{|t| tot += t}
       avg = tot / @times.length

       @times.map{|t| std_dev += (t-avg)*(t-avg)}
       std_dev /= @times.length
       std_dev = Math::sqrt(std_dev)
     end

     puts “Avg. time for threads that didn’t time out: #{avg} +/- #{std_dev}s”
   end
 end
end

test = CompareMemcacheClients.new
max_conns = 64

puts “Using memcache-client”
test.time_test(test.pool_memcache_client(max_conns), ARGV[0].to_i)

puts “”

puts “Using memcached”
test.time_test(test.pool_memcached(max_conns), ARGV[0].to_i)

puts “”

#puts “Using memcache-client, no pool”
#test.time_test(test.not_pool_memcache_client(), ARGV[0].to_i)

#puts “”

puts “Using memcached, no pool”
test.time_test(test.not_pool_memcached(), ARGV[0].to_i)

This code is centered around the idea that we use connection pools for reusing client connections in a thread safe manner, although as you can see, I made some no_pool mocks that just generate a new connection every time. The memcache-client no_pool test is commented out because this was causing segfaults with a large number of connections.

Results and Analysis

I ran the test 3 times each for each number of threads (from 1 to 64), using max connections set to 64 for the pooled clients.

What strikes me the most is the large standard deviations (not explicitly show here, as error bars made the image too messy) between runs of the memcache-client, while the results with memcached are much more uniform. This graph also shows the benefit of pooling connections.

With 63 threads running, memcache-client took about 0.02s with a std deviation of about 0.015s. memcached, on the other hand, took about 0.0005s with a std deviation of about 0.0002s. That’s nearly 2 orders of magnitude better!

When we go up to 256 threads (leaving the max connections in the pool still set to 64, so we see the effects of thread contention), the differences become even more apparent:

Here I’ve shown the standard deviations for the memcached tests as light orange error bars. Showing similar bars for the memcache-client tests was too messy.

At this point, we have to ask the question of why the pure ruby library is so much slower than the C client wrapper? Is it just that the C code is that much more optimized, or does ruby threading greatly improve when we let native code handle networking? I’d be curious to see how these results differ when run in JRuby.

Conclusion

We initially chose memcache-client over memcached because of its multithreaded support, but it turns out that wrapping memcached access inside a Sequel ConnectionPool is the way to go when you have lots of threads and want to share connections.

Notes
I used Plot (osx) to create these graphs. While it takes a few minutes to get used to its interface, it’s quite a nice (free) package.