voldemort.client.protocol.admin
Class StreamingClient

java.lang.Object
  extended by voldemort.client.protocol.admin.BaseStreamingClient
      extended by voldemort.client.protocol.admin.StreamingClient

public class StreamingClient
extends BaseStreamingClient

This class is a wrapper on the StreamingClient It serves the purpose of making streaming resilient to single node failures and making it highly available. The StreamingClient stops streaming when a voldemort node is down The streaming client interface would expect the data producer(eg: faust) to provide the failed list of nodes upfront. On a node failure the streaming client also provides the list of faulty nodes which can be used by the data producer to reinitialize the streaming session. Why did we choose this design: ==> The streaming session does not buffer the entries, it just keeps them on a network buffer, if a node fails during this time, the following flush() call will yield and generate an exception Only the data producer can resend those events, thereby re-initializing the session


Field Summary
 
Fields inherited from class voldemort.client.protocol.admin.BaseStreamingClient
adminClient, blackListedNodes, CHECKPOINT_COMMIT_SIZE, entriesProcessed, faultyNodes, MARKED_BAD, nodeIdStoreInitialized, nodeIdStoreToOutputStreamRequest, nodesToStream, routingStrategy, storeToRoutingStrategy, streamingSocketPool, throttler
 
Constructor Summary
StreamingClient(StreamingClientConfig config)
           
 
Method Summary
protected  void initializeWithFailedNodes()
           
 void initStreamingSessions(java.util.List<java.lang.String> stores, java.util.concurrent.Callable checkpointCallback, java.util.concurrent.Callable recoveryCallback, boolean allowMerge, java.util.List<java.lang.Integer> blackListedNodes)
           
 void streamingPut(ByteArray key, Versioned<byte[]> value, java.lang.String storeName)
           
protected  void streamingSlopPut(ByteArray key, Versioned<byte[]> value, java.lang.String storeName, int failedNodeId)
          This is a method to stream slops to "slop" store when a node is detected faulty in a streaming session
 
Methods inherited from class voldemort.client.protocol.admin.BaseStreamingClient
addStoreToSession, blacklistNode, close, closeStreamingSession, closeStreamingSession, closeStreamingSessions, closeStreamingSessions, commitToVoldemort, finalize, getAdminClient, getFaultyNodes, initStreamingSession, initStreamingSessions, removeStoreFromSession, streamingPut, unmarkBad, updateThrottleLimit
 
Methods inherited from class java.lang.Object
clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

StreamingClient

public StreamingClient(StreamingClientConfig config)
Method Detail

initStreamingSessions

public void initStreamingSessions(java.util.List<java.lang.String> stores,
                                  java.util.concurrent.Callable checkpointCallback,
                                  java.util.concurrent.Callable recoveryCallback,
                                  boolean allowMerge,
                                  java.util.List<java.lang.Integer> blackListedNodes)
Overrides:
initStreamingSessions in class BaseStreamingClient
Parameters:
stores - - the list of name of the stores to be streamed to
checkpointCallback - - the callback that allows for the user to record the progress, up to the last event delivered. This callable would be invoked every so often internally.
recoveryCallback - - the callback that allows the user to rewind the upstream to the position recorded by the last complete call on checkpointCallback whenever an exception occurs during the streaming session.
allowMerge - - whether to allow for the streaming event to be merged with online writes. If not, all online writes since the completion of the last streaming session will be lost at the end of the current streaming session.
blackListedNodes - - the list of Nodes not to stream to; we can probably recover them later from the replicas

streamingPut

public void streamingPut(ByteArray key,
                         Versioned<byte[]> value,
                         java.lang.String storeName)
Overrides:
streamingPut in class BaseStreamingClient
Parameters:
key - - The key
value - - The value
storeName - takes an additional store name as a parameter If a store is added mid way through a streaming session we do not play catchup and entries that were processed earlier during the session will not be applied for the store.

streamingSlopPut

protected void streamingSlopPut(ByteArray key,
                                Versioned<byte[]> value,
                                java.lang.String storeName,
                                int failedNodeId)
                         throws java.io.IOException
This is a method to stream slops to "slop" store when a node is detected faulty in a streaming session

Parameters:
key - -- original key
value - -- original value
storeName - -- the store for which we are registering the slop
failedNodeId - -- the faulty node ID for which we register a slop
Throws:
java.io.IOException

initializeWithFailedNodes

protected void initializeWithFailedNodes()


Jay Kreps, Roshan Sumbaly, Alex Feinberg, Bhupesh Bansal, Lei Gao, Chinmay Soman, Vinoth Chandar, Zhongjie Wu