voldemort.utils
Class RebalanceUtils

java.lang.Object
  extended by voldemort.utils.RebalanceUtils

public class RebalanceUtils
extends java.lang.Object

RebalanceUtils provide basic functionality for rebalancing.


Field Summary
static java.util.List<java.lang.String> canRebalanceList
           
static java.lang.String currentClusterFileName
           
static java.lang.String finalClusterFileName
           
 
Constructor Summary
RebalanceUtils()
           
 
Method Summary
static int countTaskStores(java.util.List<RebalanceTaskInfo> infos)
           
static java.util.concurrent.ExecutorService createExecutors(int numThreads)
           
static void dumpAnalysisToFile(java.lang.String outputDirName, java.lang.String baseFileName, PartitionBalance partitionBalance)
          Prints a balance analysis to a file.
static void dumpClusters(Cluster currentCluster, Cluster finalCluster, java.lang.String outputDirName)
          Given the current and final cluster dumps it into the output directory
static void dumpClusters(Cluster currentCluster, Cluster finalCluster, java.lang.String outputDirName, java.lang.String filePrefix)
          Given the initial and final cluster dumps it into the output directory
static void dumpClusterToFile(java.lang.String outputDirName, java.lang.String fileName, Cluster cluster)
          Prints a cluster xml to a file.
static void dumpPlanToFile(java.lang.String outputDirName, RebalancePlan plan)
          Prints the plan to a file.
static void executorShutDown(java.util.concurrent.ExecutorService executorService, long timeOutSec)
          Wait to shutdown service
static java.util.List<RebalanceTaskInfo> filterTaskPlanWithStores(java.util.List<RebalanceTaskInfo> existingPlanList, java.util.List<StoreDefinition> storeDefs)
          Given a list of partition plans and a set of stores, copies the store names to every individual plan and creates a new list
static Cluster getInterimCluster(Cluster currentCluster, Cluster finalCluster)
          Given the current cluster and final cluster, generates an interim cluster with empty new nodes (and zones).
static java.util.List<java.lang.Integer> getStolenPrimaryPartitions(Cluster currentCluster, Cluster finalCluster, int stealNodeId)
          For a particular stealer node find all the "primary" tuples it will steal.
static java.util.HashMap<java.lang.Integer,java.util.List<RebalanceTaskInfo>> groupPartitionsTaskByNode(java.util.List<RebalanceTaskInfo> rebalanceTaskPlanList, boolean groupByStealerNode)
          Given a list of partition infos, generates a map of stealer / donor node to list of partition infos
static void printBatchLog(int batchId, org.apache.log4j.Logger logger, java.lang.String message)
          Print log to the following logger ( Info level )
static void printBatchTaskLog(int batchId, int taskId, org.apache.log4j.Logger logger, java.lang.String message)
          Print log to the following logger ( Info level )
static void printErrorLog(int taskId, org.apache.log4j.Logger logger, java.lang.String message, java.lang.Exception e)
          Print log to the following logger ( Error level )
static void validateClusterNodeCounts(Cluster lhs, Cluster rhs)
          Confirms that both clusters have the same number of nodes by comparing set of node Ids between clusters.
static void validateClusterNodeState(Cluster subsetCluster, Cluster supersetCluster)
          Confirms that any nodes from supersetCluster that are in subsetCluster have the same state (i.e., node id, host name, and ports).
static void validateClusterPartitionCounts(Cluster lhs, Cluster rhs)
          Confirms that both clusters have the same number of total partitions.
static void validateClusterPartitionState(Cluster subsetCluster, Cluster supersetCluster)
          Confirm that all nodes shared between clusters host exact same partition IDs and that nodes only in the super set cluster have no partition IDs.
static void validateClusterStores(Cluster cluster, java.util.List<StoreDefinition> storeDefs)
          Verify store definitions are congruent with cluster definition.
static void validateClusterZonesSame(Cluster lhs, Cluster rhs)
          Confirms that both clusters have the same set of zones defined.
static void validateCurrentFinalCluster(Cluster currentCluster, Cluster finalCluster)
          A final cluster ought to be a super set of current cluster.
static void validateCurrentInterimCluster(Cluster currentCluster, Cluster interimCluster)
          An interim cluster ought to be a super set of current cluster.
static void validateInterimFinalCluster(Cluster interimCluster, Cluster finalCluster)
          Interim and final clusters ought to have same partition counts, same zones, and same node state.
static java.util.List<StoreDefinition> validateRebalanceStore(java.util.List<StoreDefinition> storeDefList)
          Given a list of store definitions, makes sure that rebalance supports all of them.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

canRebalanceList

public static final java.util.List<java.lang.String> canRebalanceList

currentClusterFileName

public static final java.lang.String currentClusterFileName
See Also:
Constant Field Values

finalClusterFileName

public static final java.lang.String finalClusterFileName
See Also:
Constant Field Values
Constructor Detail

RebalanceUtils

public RebalanceUtils()
Method Detail

validateClusterStores

public static void validateClusterStores(Cluster cluster,
                                         java.util.List<StoreDefinition> storeDefs)
Verify store definitions are congruent with cluster definition.

Parameters:
cluster -
storeDefs -

validateCurrentFinalCluster

public static void validateCurrentFinalCluster(Cluster currentCluster,
                                               Cluster finalCluster)
A final cluster ought to be a super set of current cluster. I.e., existing node IDs ought to map to same server, but partition layout can have changed and there may exist new nodes.

Parameters:
currentCluster -
finalCluster -

validateCurrentInterimCluster

public static void validateCurrentInterimCluster(Cluster currentCluster,
                                                 Cluster interimCluster)
An interim cluster ought to be a super set of current cluster. I.e., it ought to either be the same as current cluster (every partition is mapped to the same node of current & interim), or it ought to have more nodes (possibly in new zones) without partitions.

Parameters:
currentCluster -
interimCluster -

validateInterimFinalCluster

public static void validateInterimFinalCluster(Cluster interimCluster,
                                               Cluster finalCluster)
Interim and final clusters ought to have same partition counts, same zones, and same node state. Partitions per node may of course differ.

Parameters:
interimCluster -
finalCluster -

validateClusterPartitionCounts

public static void validateClusterPartitionCounts(Cluster lhs,
                                                  Cluster rhs)
Confirms that both clusters have the same number of total partitions.

Parameters:
lhs -
rhs -

validateClusterPartitionState

public static void validateClusterPartitionState(Cluster subsetCluster,
                                                 Cluster supersetCluster)
Confirm that all nodes shared between clusters host exact same partition IDs and that nodes only in the super set cluster have no partition IDs.

Parameters:
subsetCluster -
supersetCluster -

validateClusterZonesSame

public static void validateClusterZonesSame(Cluster lhs,
                                            Cluster rhs)
Confirms that both clusters have the same set of zones defined.

Parameters:
lhs -
rhs -

validateClusterNodeCounts

public static void validateClusterNodeCounts(Cluster lhs,
                                             Cluster rhs)
Confirms that both clusters have the same number of nodes by comparing set of node Ids between clusters.

Parameters:
lhs -
rhs -

validateClusterNodeState

public static void validateClusterNodeState(Cluster subsetCluster,
                                            Cluster supersetCluster)
Confirms that any nodes from supersetCluster that are in subsetCluster have the same state (i.e., node id, host name, and ports). Specific partitions hosted are not compared.

Parameters:
subsetCluster -
supersetCluster -

getInterimCluster

public static Cluster getInterimCluster(Cluster currentCluster,
                                        Cluster finalCluster)
Given the current cluster and final cluster, generates an interim cluster with empty new nodes (and zones).

Parameters:
currentCluster - Current cluster metadata
finalCluster - Final cluster metadata
Returns:
Returns a new interim cluster which contains nodes and zones of final cluster, but with empty partition lists if they were not present in current cluster.

getStolenPrimaryPartitions

public static java.util.List<java.lang.Integer> getStolenPrimaryPartitions(Cluster currentCluster,
                                                                           Cluster finalCluster,
                                                                           int stealNodeId)
For a particular stealer node find all the "primary" tuples it will steal. In other words, expect the "replica" part to be 0 always.

Parameters:
currentCluster - The cluster definition of the existing cluster
finalCluster - The final cluster definition
stealNodeId - Node id of the stealer node
Returns:
Returns a list of primary partitions which this stealer node will get

printBatchLog

public static void printBatchLog(int batchId,
                                 org.apache.log4j.Logger logger,
                                 java.lang.String message)
Print log to the following logger ( Info level )

Parameters:
batchId - Task id
logger - Logger class
message - The message to print

printBatchTaskLog

public static void printBatchTaskLog(int batchId,
                                     int taskId,
                                     org.apache.log4j.Logger logger,
                                     java.lang.String message)
Print log to the following logger ( Info level )

Parameters:
batchId -
taskId -
logger -
message -

printErrorLog

public static void printErrorLog(int taskId,
                                 org.apache.log4j.Logger logger,
                                 java.lang.String message,
                                 java.lang.Exception e)
Print log to the following logger ( Error level )

Parameters:
taskId - Stealer node id
logger - Logger class
message - The message to print

validateRebalanceStore

public static java.util.List<StoreDefinition> validateRebalanceStore(java.util.List<StoreDefinition> storeDefList)
Given a list of store definitions, makes sure that rebalance supports all of them. If not it throws an error.

Parameters:
storeDefList - List of store definitions
Returns:
Filtered list of store definitions which rebalancing supports

dumpClusters

public static void dumpClusters(Cluster currentCluster,
                                Cluster finalCluster,
                                java.lang.String outputDirName,
                                java.lang.String filePrefix)
Given the initial and final cluster dumps it into the output directory

Parameters:
currentCluster - Initial cluster metadata
finalCluster - Final cluster metadata
outputDirName - Output directory where to dump this file
filePrefix - String to prepend to the initial & final cluster metadata files
Throws:
java.io.IOException

dumpClusters

public static void dumpClusters(Cluster currentCluster,
                                Cluster finalCluster,
                                java.lang.String outputDirName)
Given the current and final cluster dumps it into the output directory

Parameters:
currentCluster - Initial cluster metadata
finalCluster - Final cluster metadata
outputDirName - Output directory where to dump this file
Throws:
java.io.IOException

dumpClusterToFile

public static void dumpClusterToFile(java.lang.String outputDirName,
                                     java.lang.String fileName,
                                     Cluster cluster)
Prints a cluster xml to a file.

Parameters:
outputDirName -
fileName -
cluster -

dumpAnalysisToFile

public static void dumpAnalysisToFile(java.lang.String outputDirName,
                                      java.lang.String baseFileName,
                                      PartitionBalance partitionBalance)
Prints a balance analysis to a file.

Parameters:
outputDirName -
baseFileName - suffix '.analysis' is appended to baseFileName.
partitionBalance -

dumpPlanToFile

public static void dumpPlanToFile(java.lang.String outputDirName,
                                  RebalancePlan plan)
Prints the plan to a file.

Parameters:
outputDirName -
plan -

countTaskStores

public static int countTaskStores(java.util.List<RebalanceTaskInfo> infos)

filterTaskPlanWithStores

public static java.util.List<RebalanceTaskInfo> filterTaskPlanWithStores(java.util.List<RebalanceTaskInfo> existingPlanList,
                                                                         java.util.List<StoreDefinition> storeDefs)
Given a list of partition plans and a set of stores, copies the store names to every individual plan and creates a new list

Parameters:
existingPlanList - Existing partition plan list
storeDefs - List of store names we are rebalancing
Returns:
List of updated partition plan

groupPartitionsTaskByNode

public static java.util.HashMap<java.lang.Integer,java.util.List<RebalanceTaskInfo>> groupPartitionsTaskByNode(java.util.List<RebalanceTaskInfo> rebalanceTaskPlanList,
                                                                                                               boolean groupByStealerNode)
Given a list of partition infos, generates a map of stealer / donor node to list of partition infos

Parameters:
rebalanceTaskPlanList - Complete list of partition plans
groupByStealerNode - Boolean indicating if we want to group by stealer node ( or donor node )
Returns:
Flattens it into a map on a per node basis

executorShutDown

public static void executorShutDown(java.util.concurrent.ExecutorService executorService,
                                    long timeOutSec)
Wait to shutdown service

Parameters:
executorService - Executor service to shutdown
timeOutSec - Time we wait for

createExecutors

public static java.util.concurrent.ExecutorService createExecutors(int numThreads)


Jay Kreps, Roshan Sumbaly, Alex Feinberg, Bhupesh Bansal, Lei Gao, Chinmay Soman, Vinoth Chandar, Zhongjie Wu