Class StreamingClient

  extended by voldemort.client.protocol.admin.BaseStreamingClient
      extended by voldemort.client.protocol.admin.StreamingClient

public class StreamingClient
extends BaseStreamingClient

This class is a wrapper on the StreamingClient It serves the purpose of making streaming resilient to single node failures and making it highly available. The StreamingClient stops streaming when a voldemort node is down The streaming client interface would expect the data producer(eg: faust) to provide the failed list of nodes upfront. On a node failure the streaming client also provides the list of faulty nodes which can be used by the data producer to reinitialize the streaming session. Why did we choose this design: ==> The streaming session does not buffer the entries, it just keeps them on a network buffer, if a node fails during this time, the following flush() call will yield and generate an exception Only the data producer can resend those events, thereby re-initializing the session

Field Summary
Fields inherited from class voldemort.client.protocol.admin.BaseStreamingClient
adminClient, blackListedNodes, CHECKPOINT_COMMIT_SIZE, entriesProcessed, faultyNodes, MARKED_BAD, nodeIdStoreInitialized, nodeIdStoreToOutputStreamRequest, nodesToStream, routingStrategy, storeToRoutingStrategy, streamingSocketPool, throttler
Constructor Summary
StreamingClient(StreamingClientConfig config)
Method Summary
protected  void initializeWithFailedNodes()
 void initStreamingSessions(java.util.List<java.lang.String> stores, java.util.concurrent.Callable checkpointCallback, java.util.concurrent.Callable recoveryCallback, boolean allowMerge, java.util.List<java.lang.Integer> blackListedNodes)
 void streamingPut(ByteArray key, Versioned<byte[]> value, java.lang.String storeName)
protected  void streamingSlopPut(ByteArray key, Versioned<byte[]> value, java.lang.String storeName, int failedNodeId)
          This is a method to stream slops to "slop" store when a node is detected faulty in a streaming session
Methods inherited from class voldemort.client.protocol.admin.BaseStreamingClient
addStoreToSession, blacklistNode, close, closeStreamingSession, closeStreamingSession, closeStreamingSessions, closeStreamingSessions, commitToVoldemort, finalize, getAdminClient, getFaultyNodes, initStreamingSession, initStreamingSessions, removeStoreFromSession, streamingPut, unmarkBad, updateThrottleLimit
Methods inherited from class java.lang.Object
clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail


public StreamingClient(StreamingClientConfig config)
Method Detail


public void initStreamingSessions(java.util.List<java.lang.String> stores,
                                  java.util.concurrent.Callable checkpointCallback,
                                  java.util.concurrent.Callable recoveryCallback,
                                  boolean allowMerge,
                                  java.util.List<java.lang.Integer> blackListedNodes)
initStreamingSessions in class BaseStreamingClient
stores - - the list of name of the stores to be streamed to
checkpointCallback - - the callback that allows for the user to record the progress, up to the last event delivered. This callable would be invoked every so often internally.
recoveryCallback - - the callback that allows the user to rewind the upstream to the position recorded by the last complete call on checkpointCallback whenever an exception occurs during the streaming session.
allowMerge - - whether to allow for the streaming event to be merged with online writes. If not, all online writes since the completion of the last streaming session will be lost at the end of the current streaming session.
blackListedNodes - - the list of Nodes not to stream to; we can probably recover them later from the replicas


public void streamingPut(ByteArray key,
                         Versioned<byte[]> value,
                         java.lang.String storeName)
streamingPut in class BaseStreamingClient
key - - The key
value - - The value
storeName - takes an additional store name as a parameter If a store is added mid way through a streaming session we do not play catchup and entries that were processed earlier during the session will not be applied for the store.


protected void streamingSlopPut(ByteArray key,
                                Versioned<byte[]> value,
                                java.lang.String storeName,
                                int failedNodeId)
                         throws java.io.IOException
This is a method to stream slops to "slop" store when a node is detected faulty in a streaming session

key - -- original key
value - -- original value
storeName - -- the store for which we are registering the slop
failedNodeId - -- the faulty node ID for which we register a slop


protected void initializeWithFailedNodes()

Jay Kreps, Roshan Sumbaly, Alex Feinberg, Bhupesh Bansal, Lei Gao, Chinmay Soman, Vinoth Chandar, Zhongjie Wu