Introducing leveldb-server

We just released leveldb-server. Start forking: Github leveldb-server

leveldb-server

  • Async leveldb server and client based on zeromq
  • Storage engine leveldb
  • Networking library zeromq
  • We use leveldb-server at Safebox

License

New BSD license. Please see license.txt for more details.

Features

  • Very simple key-value storage
  • Data is sorted by key – allows range queries
  • Data is automatically compressed
  • Can act as persistent cache
  • For our use at Safebox it replaced memcached+mysql
  • Simple backups cp -rf level.db backup.db
  • Networking/wiring from zeromq messaging library – allows many topologies
  • Async server for scalability and capacity
  • Sync client for easy coding
  • Easy polyglot client bindings. See zmq bindings
>>> db.put("k3", "v3")
'True'
>>> db.get("k3")
'v3'
>>> db.range()
'[{"k1": "v1"}, {"k2": "v2"}, {"k3": "v3"}]'
>>> db.range("k1", "k2")
'[{"k1": "v1"}, {"k2": "v2"}]'
>>> db.delete('k1')
>>>
Will be adding high availability, replication and autosharding using the same zeromq framework.

Dependencies

python 2.6+ (older versions with simplejson)
zmq
pyzmq
leveldb
pyleveldb 

Getting Started

Instructions for an EC2 Ubuntu box.

Installing zeromq

wget http://download.zeromq.org/zeromq-2.1.10.tar.gz
tar xvfz zeromq-2.1.10.tar.gz
cd zeromq-2.1.10
sudo ./configure
sudo make
sudo make install

Installing pyzmq

wget https://github.com/zeromq/pyzmq/downloads/pyzmq-2.1.10.tar.gz
tar xvfz pyzmq-2.1.10.tar.gz
cd pyzmq-2.1.10/
sudo python setup.py configure --zmq=/usr/local/lib/
sudo python setup.py install

Installing leveldb and pyleveldb

svn checkout http://py-leveldb.googlecode.com/svn/trunk/ py-leveldb-read-only
cd py-leveldb-read-only
sudo compile_leveldb.sh
sudo python setup.py install

Starting the leveldb-server

> python leveldb-server.py -h
Usage: leveldb-server.py 
 -p [port and host settings] Default: tcp://127.0.0.1:5147
 -f [database file name] Default: level.db

leveldb-server

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -p HOST, --host=HOST  
  -d DBFILE, --dbfile=DBFILE
> python leveldb-server.py

Using the leveldb-client-py

> cd clients/py/
> sudo python setup.py install
> python 
>>> from leveldbClient import database
>>> db = database.leveldb()
>>> db.get("Key")
>>> db.put("K", "V")
>>> db.range()
>>> db.range(start, end)
>>> db.delete("K")

Backups
cp -rpf level.db backup.db

Known issues and work in progress

Would love your pull requests on
  • Benchmarking and performance analysis
  • client libraries for other languages
  • [issue] zeromq performance issues with 1M+ inserts at a time
  • [feature] timeouts in client library
  • [feature] support for counters
  • [feature] limit support in range queries
  • Serializing and seperate threads for get/put/range in leveldb-server
  • HA/replication/autosharding and possibly pub-sub for replication

Thanks

Thanks to all the folks who have contributed to all the dependencies. Special thanks to pyzmq/examples/mongo* author for inspiration.

No comments:

Post a Comment