voldemort.client.protocol.admin
Class StreamingClient

java.lang.Object
  voldemort.client.protocol.admin.StreamingClient

public class StreamingClient
extends java.lang.Object

Author:

anagpal The streaming API allows for send events into voldemort stores in the async fashion. All the partition and replication logic will be taken care of internally. The users is expected to provide two callbacks, one for performing period checkpoints and one for recovering the streaming process from the last checkpoint. NOTE: The API is not thread safe, since if multiple threads use this API we cannot make any guarantees about correctness of the checkpointing mechanism. Right now we expect this to used by a single thread per data source

Field Summary

protected RoutingStrategy	routingStrategy
protected EventThrottler	throttler

Constructor Summary

StreamingClient(StreamingClientConfig config)

Method Summary

void	closeStreamingSession() Close the streaming session Flush all n/w buffers and call the commit callback
void	closeStreamingSession(java.util.concurrent.Callable resetCheckpointCallback)
void	closeStreamingSessions() Close the streaming session Flush all n/w buffers and call the commit callback
void	closeStreamingSessions(java.util.concurrent.Callable resetCheckpointCallback)
void	commitToVoldemort() Flush the network buffer and write all entries to the server Wait for an ack from the server This is a blocking call.
protected void	finalize()
void	initStreamingSession(java.lang.String store, java.util.concurrent.Callable checkpointCallback, java.util.concurrent.Callable recoveryCallback, boolean allowMerge)
void	initStreamingSessions(java.util.List<java.lang.String> stores, java.util.concurrent.Callable checkpointCallback, java.util.concurrent.Callable recoveryCallback, boolean allowMerge)
void	initStreamingSessions(java.util.List<java.lang.String> stores, java.util.concurrent.Callable checkpointCallback, java.util.concurrent.Callable recoveryCallback, boolean allowMerge, java.util.List<java.lang.Integer> blackListedNodes)
void	removeStoreFromSession(java.util.List<java.lang.String> storeNameToRemove) Remove a list of stores from the session First commit all entries for these stores and then cleanup resources
void	streamingPut(ByteArray key, Versioned<byte[]> value) A Streaming Put call
void	streamingPut(ByteArray key, Versioned<byte[]> value, java.lang.String storeName)
void	updateThrottleLimit(int throttleQPS)

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

routingStrategy

protected RoutingStrategy routingStrategy

throttler

protected EventThrottler throttler

Constructor Detail

StreamingClient

public StreamingClient(StreamingClientConfig config)

Method Detail

updateThrottleLimit

public void updateThrottleLimit(int throttleQPS)

initStreamingSession

public void initStreamingSession(java.lang.String store,
                                 java.util.concurrent.Callable checkpointCallback,
                                 java.util.concurrent.Callable recoveryCallback,
                                 boolean allowMerge)

Parameters:

store - - the name of the store to be streamed to

checkpointCallback - - the callback that allows for the user to record the progress, up to the last event delivered. This callable would be invoked every so often internally.

recoveryCallback - - the callback that allows the user to rewind the upstream to the position recorded by the last complete call on checkpointCallback whenever an exception occurs during the streaming session.

allowMerge - - whether to allow for the streaming event to be merged with online writes. If not, all online writes since the completion of the last streaming session will be lost at the end of the current streaming session.

streamingPut

public void streamingPut(ByteArray key,
                         Versioned<byte[]> value)

A Streaming Put call

Parameters:

key - - The key

value - - The value

closeStreamingSession

public void closeStreamingSession(java.util.concurrent.Callable resetCheckpointCallback)

Parameters:

resetCheckpointCallback - - the callback that allows for the user to clean up the checkpoint at the end of the streaming session so a new session could, if necessary, start from 0 position.

closeStreamingSession

public void closeStreamingSession()

Close the streaming session Flush all n/w buffers and call the commit callback

finalize

protected void finalize()

Overrides:

finalize in class java.lang.Object

initStreamingSessions

public void initStreamingSessions(java.util.List<java.lang.String> stores,
                                  java.util.concurrent.Callable checkpointCallback,
                                  java.util.concurrent.Callable recoveryCallback,
                                  boolean allowMerge)

Parameters:

stores - - the list of name of the stores to be streamed to

checkpointCallback - - the callback that allows for the user to record the progress, up to the last event delivered. This callable would be invoked every so often internally.

recoveryCallback - - the callback that allows the user to rewind the upstream to the position recorded by the last complete call on checkpointCallback whenever an exception occurs during the streaming session.

allowMerge - - whether to allow for the streaming event to be merged with online writes. If not, all online writes since the completion of the last streaming session will be lost at the end of the current streaming session.

initStreamingSessions

public void initStreamingSessions(java.util.List<java.lang.String> stores,
                                  java.util.concurrent.Callable checkpointCallback,
                                  java.util.concurrent.Callable recoveryCallback,
                                  boolean allowMerge,
                                  java.util.List<java.lang.Integer> blackListedNodes)

Parameters:

stores - - the list of name of the stores to be streamed to

checkpointCallback - - the callback that allows for the user to record the progress, up to the last event delivered. This callable would be invoked every so often internally.

recoveryCallback - - the callback that allows the user to rewind the upstream to the position recorded by the last complete call on checkpointCallback whenever an exception occurs during the streaming session.

allowMerge - - whether to allow for the streaming event to be merged with online writes. If not, all online writes since the completion of the last streaming session will be lost at the end of the current streaming session.

blackListedNodes - - the list of Nodes not to stream to; we can probably recover them later from the replicas

removeStoreFromSession

public void removeStoreFromSession(java.util.List<java.lang.String> storeNameToRemove)

Remove a list of stores from the session First commit all entries for these stores and then cleanup resources

Parameters:

storeNameToRemove - List of stores to be removed from the current streaming session

streamingPut

public void streamingPut(ByteArray key,
                         Versioned<byte[]> value,
                         java.lang.String storeName)

Parameters:

key - - The key

value - - The value

storeName - takes an additional store name as a parameter If a store is added mid way through a streaming session we do not play catchup and entries that were processed earlier during the session will not be applied for the store.

commitToVoldemort

public void commitToVoldemort()

Flush the network buffer and write all entries to the server Wait for an ack from the server This is a blocking call. It is invoked on every Commit batch size of entries It is also called on the close session call

closeStreamingSessions

public void closeStreamingSessions(java.util.concurrent.Callable resetCheckpointCallback)

Parameters:

resetCheckpointCallback - - the callback that allows for the user to clean up the checkpoint at the end of the streaming session so a new session could, if necessary, start from 0 position.

closeStreamingSessions

public void closeStreamingSessions()

Close the streaming session Flush all n/w buffers and call the commit callback