Getting Started With The Source Code

For programmers interested in getting to know the source code, here are a few pointers to get started. One good starting place is the Amazon Dynamo paper, it is a bit dense in places but will give an idea of why certain things are done a given way.

Most packages have a javadoc.html file that describes what the code in that package does, this can help to get a broad overview of the areas of the code.

Internally much of Voldemort implements a single very simple interface that provides basically put/get/delete given in Store.java.

Each Store implementation is in a subdirectory of voldemort.store. This is what allows layering of functionality for things like networking, conflict resolution, etc. A specialized instance of this is StorageEngine.java which provides the ability to iterate over all the keys. Implementations of this interface are responsible for the actual persistence of the data. The important StorageEngines are InMemoryStorageEngine.java, BdbStorageEngine.java, and ReadOnlyStorageEngine.java, each of which has a different persistence strategy.

The server code is found under voldemort.server, and the primary class is VoldemortServer.java. Internally the server is comprised of multiple smaller services that handle particular functionality such as storage, various kinds of networking, etc. These services implement a very simple VoldemortService lifecycle interface to allow them to be started and stopped with the server.

The client code is found under voldemort.client. StoreClient.java is the interface the user interacts with. This is little more than a richer wrapper around a Store that provides a friendlier API, but it also serves to isolate client code from internal changes to avoid API breakage. To see how the various layers are wired up in the client look at AbstractStoreClientFactory.java, which does the bootstrapping and assembly necessary to create a functioning Java client.

Code related to versioning is under voldemort.versioning; in particular the class used for conflict resolution is VectorClock.java.

Contributions

Code contributions are very welcome. There is a project idea wiki page that has interesting (but sometimes non-trivial) projects if you want to see some ideas. Feel free to email the mailing list if you want to discuss any idea you are considering or have questions about how to proceed. There is also the IRC channel #voldemort on irc.oftc.net.

The best approach for actually submitting a contribution is to fork the project in git and make your changes available on github (or another public git repository). If you aren't familiar with git you can just download a tarball of the source code from the build server, create a patch against that, and email it to the mailing list. The project comes with eclipse settings that will enforce some basic stylistic guidelines, but if you don't use eclipse you can just copy the style you see elsewhere in the code base. Non-trivial code should have unit tests to keep it working.

Here is how to submit a change using git:

• Create a github account, if you don't already have one.
• Go to the master voldemort repository and click the fork button to create a copy in your own repo.
• Checkout the code:

  git clone git://github.com/your_user_name/voldemort.git

• Eclipse and Idea IDEs are supported out of the box.
  
  • For Eclipse, .project and .classpath can be generated by the command "./gradlew eclipse" . Use the File -> Import and open the voldemort directory.
  • For Idea, .iml , .ipr and .iws files can be generated by running the command "./gradlew idea". 
• Gradle is the supported build system for voldemort. ( ant is deprecated and will be eventually removed).
  
• Gradle wrapper (./gradlew on *nix and gradlew.bat on windows ) takes care of installing the correct version of gradle on your machine
  
• Use the following gradle commands for the regular tasks
  • ./gradlew build -x test -> builds the jar
  • ./gradlew junitAll -> builds and runs the unit tests ( mandatory before committing)
  • ./gradlew junitLong -> builds and run the long running tests like rebalance and others. Required for big changes.
  • ./gradlew clean -> cleans the build outputs.
  • ./gradlew build zip tar -x test -> creates a tar ball and zip. We found it is convenient to copy the tar balls to remote machine, launch a client/admin shell for devops related tasks.
    
• Commit your changes:

  git commit -a

• Push your changes back to github:

  git push git://github.com/your_user_name/voldemort.git

• Tell us what you have done so it can be integrated.