voldemort.utils
Class RebalanceUtils

java.lang.Object
  voldemort.utils.RebalanceUtils

public class RebalanceUtils
extends java.lang.Object

RebalanceUtils provide basic functionality for rebalancing.

Field Summary

static java.util.List<java.lang.String>	canRebalanceList
static java.lang.String	currentClusterFileName
static java.lang.String	finalClusterFileName
static java.lang.String	finalStoresFileName

Constructor Summary

RebalanceUtils()

Method Summary

static int	countTaskStores(java.util.List<RebalanceTaskInfo> infos)
static java.util.concurrent.ExecutorService	createExecutors(int numThreads)
static Cluster	dropZone(Cluster intermediateCluster, int dropZoneId) Given a interim cluster with a previously vacated zone, constructs a new cluster object with the drop zone completely removed
static java.util.List<StoreDefinition>	dropZone(java.util.List<StoreDefinition> currentStoreDefs, int dropZoneId) Similar to vacateZone(Cluster, int), takes the current store definitions in the cluster and creates store definitions with the specified zone effectively dropped.
static void	dumpAnalysisToFile(java.lang.String outputDirName, java.lang.String baseFileName, PartitionBalance partitionBalance) Prints a balance analysis to a file.
static void	dumpClusters(Cluster currentCluster, Cluster finalCluster, java.lang.String outputDirName) Given the current and final cluster dumps it into the output directory
static void	dumpClusters(Cluster currentCluster, Cluster finalCluster, java.lang.String outputDirName, java.lang.String filePrefix) Given the initial and final cluster dumps it into the output directory
static void	dumpClusterToFile(java.lang.String outputDirName, java.lang.String fileName, Cluster cluster) Prints a cluster xml to a file.
static void	dumpPlanToFile(java.lang.String outputDirName, RebalancePlan plan) Prints the plan to a file.
static void	dumpStoreDefsToFile(java.lang.String outputDirName, java.lang.String fileName, java.util.List<StoreDefinition> storeDefs) Prints a stores xml to a file.
static void	executorShutDown(java.util.concurrent.ExecutorService executorService, long timeOutSec) Wait to shutdown service
static java.util.List<RebalanceTaskInfo>	filterTaskPlanWithStores(java.util.List<RebalanceTaskInfo> existingPlanList, java.util.List<StoreDefinition> storeDefs) Given a list of partition plans and a set of stores, copies the store names to every individual plan and creates a new list
static Cluster	getInterimCluster(Cluster currentCluster, Cluster finalCluster) Given the current cluster and final cluster, generates an interim cluster with empty new nodes (and zones).
static java.util.List<java.lang.Integer>	getStolenPrimaryPartitions(Cluster currentCluster, Cluster finalCluster, int stealNodeId) For a particular stealer node find all the primary partitions tuples it will steal.
static java.util.HashMap<java.lang.Integer,java.util.List<RebalanceTaskInfo>>	groupPartitionsTaskByNode(java.util.List<RebalanceTaskInfo> rebalanceTaskPlanList, boolean groupByStealerNode) Given a list of partition infos, generates a map of stealer / donor node to list of partition infos
static void	printBatchLog(int batchId, org.apache.log4j.Logger logger, java.lang.String message) Print log to the following logger ( Info level )
static void	printBatchTaskLog(int batchId, int taskId, org.apache.log4j.Logger logger, java.lang.String message) Print log to the following logger ( Info level )
static void	printErrorLog(int taskId, org.apache.log4j.Logger logger, java.lang.String message, java.lang.Exception e) Print log to the following logger ( Error level )
static Cluster	vacateZone(Cluster currentCluster, int dropZoneId) Given the current cluster and a zone id that needs to be dropped, this method will remove all partitions from the zone that is being dropped and move it to the existing zones.
static void	validateClusterNodeCounts(Cluster lhs, Cluster rhs) Confirms that both clusters have the same number of nodes by comparing set of node Ids between clusters.
static void	validateClusterNodeState(Cluster subsetCluster, Cluster supersetCluster) Confirms that any nodes from supersetCluster that are in subsetCluster have the same state (i.e., node id, host name, and ports).
static void	validateClusterPartitionCounts(Cluster lhs, Cluster rhs) Confirms that both clusters have the same number of total partitions.
static void	validateClusterPartitionState(Cluster subsetCluster, Cluster supersetCluster) Confirm that all nodes shared between clusters host exact same partition IDs and that nodes only in the super set cluster have no partition IDs.
static void	validateClusterStores(Cluster cluster, java.util.List<StoreDefinition> storeDefs) Verify store definitions are congruent with cluster definition.
static void	validateClusterZonesSame(Cluster lhs, Cluster rhs) Confirms that both clusters have the same set of zones defined.
static void	validateCurrentFinalCluster(Cluster currentCluster, Cluster finalCluster) A final cluster ought to be a super set of current cluster.
static void	validateCurrentInterimCluster(Cluster currentCluster, Cluster interimCluster) An interim cluster ought to be a super set of current cluster.
static void	validateInterimFinalCluster(Cluster interimCluster, Cluster finalCluster) Interim and final clusters ought to have same partition counts, same zones, and same node state.
static java.util.List<StoreDefinition>	validateRebalanceStore(java.util.List<StoreDefinition> storeDefList) Given a list of store definitions, makes sure that rebalance supports all of them.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

canRebalanceList

public static final java.util.List<java.lang.String> canRebalanceList

currentClusterFileName

public static final java.lang.String currentClusterFileName

See Also:

Constant Field Values

finalClusterFileName

public static final java.lang.String finalClusterFileName

See Also:

Constant Field Values

finalStoresFileName

public static final java.lang.String finalStoresFileName

See Also:

Constant Field Values

Constructor Detail

RebalanceUtils

public RebalanceUtils()

Method Detail

validateClusterStores

public static void validateClusterStores(Cluster cluster,
                                         java.util.List<StoreDefinition> storeDefs)

Verify store definitions are congruent with cluster definition.

Parameters:

cluster -

storeDefs -

validateCurrentFinalCluster

public static void validateCurrentFinalCluster(Cluster currentCluster,
                                               Cluster finalCluster)

A final cluster ought to be a super set of current cluster. I.e., existing node IDs ought to map to same server, but partition layout can have changed and there may exist new nodes.

Parameters:

currentCluster -

finalCluster -

validateCurrentInterimCluster

public static void validateCurrentInterimCluster(Cluster currentCluster,
                                                 Cluster interimCluster)

An interim cluster ought to be a super set of current cluster. I.e., it ought to either be the same as current cluster (every partition is mapped to the same node of current & interim), or it ought to have more nodes (possibly in new zones) without partitions.

Parameters:

currentCluster -

interimCluster -

validateInterimFinalCluster

public static void validateInterimFinalCluster(Cluster interimCluster,
                                               Cluster finalCluster)

Interim and final clusters ought to have same partition counts, same zones, and same node state. Partitions per node may of course differ.

Parameters:

interimCluster -

finalCluster -

validateClusterPartitionCounts

public static void validateClusterPartitionCounts(Cluster lhs,
                                                  Cluster rhs)

Confirms that both clusters have the same number of total partitions.

Parameters:

lhs -

rhs -

validateClusterPartitionState

public static void validateClusterPartitionState(Cluster subsetCluster,
                                                 Cluster supersetCluster)

Confirm that all nodes shared between clusters host exact same partition IDs and that nodes only in the super set cluster have no partition IDs.

Parameters:

subsetCluster -

supersetCluster -

validateClusterZonesSame

public static void validateClusterZonesSame(Cluster lhs,
                                            Cluster rhs)

Confirms that both clusters have the same set of zones defined.

Parameters:

lhs -

rhs -

validateClusterNodeCounts

public static void validateClusterNodeCounts(Cluster lhs,
                                             Cluster rhs)

Confirms that both clusters have the same number of nodes by comparing set of node Ids between clusters.

Parameters:

lhs -

rhs -

validateClusterNodeState

public static void validateClusterNodeState(Cluster subsetCluster,
                                            Cluster supersetCluster)

Confirms that any nodes from supersetCluster that are in subsetCluster have the same state (i.e., node id, host name, and ports). Specific partitions hosted are not compared.

Parameters:

subsetCluster -

supersetCluster -

getInterimCluster

public static Cluster getInterimCluster(Cluster currentCluster,
                                        Cluster finalCluster)

Given the current cluster and final cluster, generates an interim cluster with empty new nodes (and zones).

Parameters:

currentCluster - Current cluster metadata

finalCluster - Final cluster metadata

Returns:

Returns a new interim cluster which contains nodes and zones of final cluster, but with empty partition lists if they were not present in current cluster.

vacateZone

public static Cluster vacateZone(Cluster currentCluster,
                                 int dropZoneId)

Given the current cluster and a zone id that needs to be dropped, this method will remove all partitions from the zone that is being dropped and move it to the existing zones. The partitions are moved intelligently so as not to avoid any data movement in the existing zones. This is achieved by moving the partitions to nodes in the surviving zones that is zone-nry to that partition in the surviving zone.

Parameters:

currentCluster - Current cluster metadata

Returns:

Returns an interim cluster with empty partition lists on the nodes from the zone being dropped

dropZone

public static Cluster dropZone(Cluster intermediateCluster,
                               int dropZoneId)

Given a interim cluster with a previously vacated zone, constructs a new cluster object with the drop zone completely removed

Parameters:

intermediateCluster -

dropZoneId -

Returns:

adjusted cluster with the zone dropped

dropZone

public static java.util.List<StoreDefinition> dropZone(java.util.List<StoreDefinition> currentStoreDefs,
                                                       int dropZoneId)

Similar to vacateZone(Cluster, int), takes the current store definitions in the cluster and creates store definitions with the specified zone effectively dropped. In order to drop a zone, we adjust the total replication factor and remove zone replication factor for the dropped zone

Parameters:

currentStoreDefs -

dropZoneId -

Returns:

the adjusted list of store definitions

getStolenPrimaryPartitions

public static java.util.List<java.lang.Integer> getStolenPrimaryPartitions(Cluster currentCluster,
                                                                           Cluster finalCluster,
                                                                           int stealNodeId)

For a particular stealer node find all the primary partitions tuples it will steal.

Parameters:

currentCluster - The cluster definition of the existing cluster

finalCluster - The final cluster definition

stealNodeId - Node id of the stealer node

Returns:

Returns a list of primary partitions which this stealer node will get

printBatchLog

public static void printBatchLog(int batchId,
                                 org.apache.log4j.Logger logger,
                                 java.lang.String message)

Print log to the following logger ( Info level )

Parameters:

batchId - Task id

logger - Logger class

message - The message to print

printBatchTaskLog

public static void printBatchTaskLog(int batchId,
                                     int taskId,
                                     org.apache.log4j.Logger logger,
                                     java.lang.String message)

Print log to the following logger ( Info level )

Parameters:

batchId -

taskId -

logger -

message -

printErrorLog

public static void printErrorLog(int taskId,
                                 org.apache.log4j.Logger logger,
                                 java.lang.String message,
                                 java.lang.Exception e)

Print log to the following logger ( Error level )

Parameters:

taskId - Stealer node id

logger - Logger class

message - The message to print

validateRebalanceStore

public static java.util.List<StoreDefinition> validateRebalanceStore(java.util.List<StoreDefinition> storeDefList)

Given a list of store definitions, makes sure that rebalance supports all of them. If not it throws an error.

Parameters:

storeDefList - List of store definitions

Returns:

Filtered list of store definitions which rebalancing supports

dumpClusters

public static void dumpClusters(Cluster currentCluster,
                                Cluster finalCluster,
                                java.lang.String outputDirName,
                                java.lang.String filePrefix)

Given the initial and final cluster dumps it into the output directory

Parameters:

currentCluster - Initial cluster metadata

finalCluster - Final cluster metadata

outputDirName - Output directory where to dump this file

filePrefix - String to prepend to the initial & final cluster metadata files

Throws:

java.io.IOException

dumpClusters

public static void dumpClusters(Cluster currentCluster,
                                Cluster finalCluster,
                                java.lang.String outputDirName)

Given the current and final cluster dumps it into the output directory

Parameters:

currentCluster - Initial cluster metadata

finalCluster - Final cluster metadata

outputDirName - Output directory where to dump this file

Throws:

java.io.IOException

dumpClusterToFile

public static void dumpClusterToFile(java.lang.String outputDirName,
                                     java.lang.String fileName,
                                     Cluster cluster)

Prints a cluster xml to a file.

Parameters:

outputDirName -

fileName -

cluster -

dumpStoreDefsToFile

public static void dumpStoreDefsToFile(java.lang.String outputDirName,
                                       java.lang.String fileName,
                                       java.util.List<StoreDefinition> storeDefs)

Prints a stores xml to a file.

Parameters:

outputDirName -

fileName -

list - of storeDefs

dumpAnalysisToFile

public static void dumpAnalysisToFile(java.lang.String outputDirName,
                                      java.lang.String baseFileName,
                                      PartitionBalance partitionBalance)

Prints a balance analysis to a file.

Parameters:

outputDirName -

baseFileName - suffix '.analysis' is appended to baseFileName.

partitionBalance -

dumpPlanToFile

public static void dumpPlanToFile(java.lang.String outputDirName,
                                  RebalancePlan plan)

Prints the plan to a file.

Parameters:

outputDirName -

plan -

countTaskStores

public static int countTaskStores(java.util.List<RebalanceTaskInfo> infos)

filterTaskPlanWithStores

public static java.util.List<RebalanceTaskInfo> filterTaskPlanWithStores(java.util.List<RebalanceTaskInfo> existingPlanList,
                                                                         java.util.List<StoreDefinition> storeDefs)

Given a list of partition plans and a set of stores, copies the store names to every individual plan and creates a new list

Parameters:

existingPlanList - Existing partition plan list

storeDefs - List of store names we are rebalancing

Returns:

List of updated partition plan

groupPartitionsTaskByNode

public static java.util.HashMap<java.lang.Integer,java.util.List<RebalanceTaskInfo>> groupPartitionsTaskByNode(java.util.List<RebalanceTaskInfo> rebalanceTaskPlanList,
                                                                                                               boolean groupByStealerNode)

Given a list of partition infos, generates a map of stealer / donor node to list of partition infos

Parameters:

rebalanceTaskPlanList - Complete list of partition plans

groupByStealerNode - Boolean indicating if we want to group by stealer node ( or donor node )

Returns:

Flattens it into a map on a per node basis

executorShutDown

public static void executorShutDown(java.util.concurrent.ExecutorService executorService,
                                    long timeOutSec)

Wait to shutdown service

Parameters:

executorService - Executor service to shutdown

timeOutSec - Time we wait for

createExecutors

public static java.util.concurrent.ExecutorService createExecutors(int numThreads)