voldemort.client.protocol.admin
Class StreamingClient
java.lang.Object
voldemort.client.protocol.admin.BaseStreamingClient
voldemort.client.protocol.admin.StreamingClient
public class StreamingClient
- extends BaseStreamingClient
This class is a wrapper on the StreamingClient It serves the purpose of
making streaming resilient to single node failures and making it highly
available.
The StreamingClient stops streaming when a voldemort node is down
The streaming client interface would expect the data producer(eg: faust) to
provide the failed list of nodes upfront. On a node failure the streaming
client also provides the list of faulty nodes which can be used by the data
producer to reinitialize the streaming session.
Why did we choose this design: ==> The streaming session does not buffer the
entries, it just keeps them on a network buffer, if a node fails during this
time, the following flush() call will yield and generate an exception Only
the data producer can resend those events, thereby re-initializing the
session
Fields inherited from class voldemort.client.protocol.admin.BaseStreamingClient |
adminClient, blackListedNodes, CHECKPOINT_COMMIT_SIZE, entriesProcessed, faultyNodes, MARKED_BAD, nodeIdStoreInitialized, nodeIdStoreToOutputStreamRequest, nodesToStream, routingStrategy, storeToRoutingStrategy, streamingSocketPool, throttler |
Method Summary |
protected void |
initializeWithFailedNodes()
|
void |
initStreamingSessions(java.util.List<java.lang.String> stores,
java.util.concurrent.Callable checkpointCallback,
java.util.concurrent.Callable recoveryCallback,
boolean allowMerge,
java.util.List<java.lang.Integer> blackListedNodes)
|
void |
streamingPut(ByteArray key,
Versioned<byte[]> value,
java.lang.String storeName)
|
protected void |
streamingSlopPut(ByteArray key,
Versioned<byte[]> value,
java.lang.String storeName,
int failedNodeId)
This is a method to stream slops to "slop" store when a node is detected
faulty in a streaming session |
Methods inherited from class voldemort.client.protocol.admin.BaseStreamingClient |
addStoreToSession, blacklistNode, close, closeStreamingSession, closeStreamingSession, closeStreamingSessions, closeStreamingSessions, commitToVoldemort, finalize, getAdminClient, getFaultyNodes, initStreamingSession, initStreamingSessions, removeStoreFromSession, streamingPut, unmarkBad, updateThrottleLimit |
Methods inherited from class java.lang.Object |
clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
StreamingClient
public StreamingClient(StreamingClientConfig config)
initStreamingSessions
public void initStreamingSessions(java.util.List<java.lang.String> stores,
java.util.concurrent.Callable checkpointCallback,
java.util.concurrent.Callable recoveryCallback,
boolean allowMerge,
java.util.List<java.lang.Integer> blackListedNodes)
- Overrides:
initStreamingSessions
in class BaseStreamingClient
- Parameters:
stores
- - the list of name of the stores to be streamed tocheckpointCallback
- - the callback that allows for the user to
record the progress, up to the last event delivered. This callable
would be invoked every so often internally.recoveryCallback
- - the callback that allows the user to rewind the
upstream to the position recorded by the last complete call on
checkpointCallback whenever an exception occurs during the
streaming session.allowMerge
- - whether to allow for the streaming event to be merged
with online writes. If not, all online writes since the completion
of the last streaming session will be lost at the end of the
current streaming session.blackListedNodes
- - the list of Nodes not to stream to; we can
probably recover them later from the replicas
streamingPut
public void streamingPut(ByteArray key,
Versioned<byte[]> value,
java.lang.String storeName)
- Overrides:
streamingPut
in class BaseStreamingClient
- Parameters:
key
- - The keyvalue
- - The valuestoreName
- takes an additional store name as a parameter
If a store is added mid way through a streaming session we do not
play catchup and entries that were processed earlier during the
session will not be applied for the store.
streamingSlopPut
protected void streamingSlopPut(ByteArray key,
Versioned<byte[]> value,
java.lang.String storeName,
int failedNodeId)
throws java.io.IOException
- This is a method to stream slops to "slop" store when a node is detected
faulty in a streaming session
- Parameters:
key
- -- original keyvalue
- -- original valuestoreName
- -- the store for which we are registering the slopfailedNodeId
- -- the faulty node ID for which we register a slop
- Throws:
java.io.IOException
initializeWithFailedNodes
protected void initializeWithFailedNodes()
Jay Kreps, Roshan Sumbaly, Alex Feinberg, Bhupesh Bansal, Lei Gao, Chinmay Soman, Vinoth Chandar, Zhongjie Wu