开发者

HDFS Balancer负载均衡器及语法详解

目录
  • 1、背景
  • 2、什么是平衡
    • 2.1 每个DataNode的利用率计算
    • 2.2 集群的利用率
    • 2.3 平衡
  • 3、hdfs balancer语法
    • 4、运行一个简单的balance案例
      • 4.1 设置平衡数据传输带宽
      • 4.2 执行banalce
    • 5、参考文档

      1、背景

      当我们的hadoop集群运行了一段时间之后,各个DataNode上的数据分布不一定均匀分布的。比如说: 我们向现有集群中添加了一个新的DataNode。

      HDFS Balancer负载均衡器及语法详解

      2、什么是平衡

      此处是我自己的一个简单的理解

      所谓的平衡指的是 每个DataNode的利用率集群的利用率 之间相差不超过给定的阈值百分比。此处的平衡 指的是各个DataNode之间的平衡,同一个DataNode之间的各个磁盘yBdSb是不会平衡的。

      2.1 每个DataNode的利用率计算

      HDFS Balancer负载均衡器及语法详解

      DataNode的利用率= dfs已用的空间 / 分配给dfs的空间。

      注意: 分配给dfs的空间 不是磁盘的总空间。

      2.2 集群的利用率

      HDFS Balancer负载均衡器及语法详解

      集群的利用率= 各datanode dfs已使用的空间 / 各datanode总空间

      2.3 平衡

      假设平衡的阈值是 5%,集群的利用率是 37.5,那么每个节点的利用率在32.5%42.5%之间都认为是均衡的。也就是说,极端情况下,DataNode的利用率最大相差10%

      3、hdfs balancer语法

      [hadoopdeploy@hadoop01 ~]$ hdfs balancer --help
      Usage: hdfs balancer
      	[-policy <policy>]	the balancing policy: datanode or blockpool
      	[-threshold <threshold>]	Percentage of disk capacity
      	[-exclude [-f <hosts-file> | <comma-separated list of hosts>]]	Excludes the specified datanodes.
      	[-include [-f <hosts-file> | <comma-separated list of hosts>]]	Includes only the specified datanodes.
      	[-source [-f <hosts-file> | <comma-separated list of hosts>]]	Pick only the specified datanodes as source nodes.
      	[-blockpools <comma-separated list of blockpool ids>]	The balancer will only run on blockpools included in this list.
      	[-idleiterations <idleiterations>]	Number of consecutive idle iterations (-1 for Infinite) before exit.
      	[-runDuringUpgrade]	Whether to run the balancer during an ongoing HDFS upgrade.This is usually not desired since it will not affect used space on over-utilized MAChines.
      	[-asService]	Run as a long running service.
      
      Generic options supported are:
      -conf <configuration file>        specify an application configuratiohttp://www.devze.comn file
      -D <property=value>               define a value for a given property
      -fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'androidfs.defaultFS' property from configurations.
      -jt <local|resourcemanager:port>  specify a ResourceManager
      -files <file1,...>                specify a comma-separated list of files to be copied to the map reduce cluster
      -libjars <jar1,...>               specify a comma-separated list of jar files to be included in the classpath
      -archives <archive1,...>          specify a comma-separated list of archives to be unarchived on the compute machines
      
      The general command line syntax is:
      command [genericOptions] [commandOptions]
      
      参数描述
      threshold磁盘容量的百分比。默认值为10%,表示上下浮动10%。
      policy平衡策略。

      datanode(默认):当每一个DataNode是平衡的时候,集群就是平衡的。

      blockpool:当每一个DataNode中的blockpool是平衡的,集群就是平衡的。

      exclude不参与平衡的DataNode节点
      include参与平衡的DataNode节点
      source仅选取指定的数据节点作为源节点
      blockpoolsBalancer仅在指定的blockpools中运行
      idleiterations退出前的连续空闲迭代次数(-1表示无限)
      -runDuringUpgrade是否在正在进行的HDFS升级过程中运行平衡器。通常不需要这样做,因为这不会影响过度使用的计算机上的已用空间。
      -asService作为长期运行的服务运行

      4、运行一个简单的balance案例

      4.1 设置平衡数据传输带宽

      [hadoopdeploy@hadoop01 ~]$ hdfs dfsadmin  -setBalancerBandwidth 10485760
      Bala编程客栈ncer bandwidth is set to 10485760
      [hadoopdeploy@hadoop01 ~]$
      

      当我们的集群负载需要调低这个值,当我们的集群负载较低时,可以适当调高这个值。

      4.2 执行banalce

      [hadoopdeploy@hadoop01 ~]$ hdfs balancer -policy d开发者_Go开发atanode -threshold 5
      2023-03-26 14:10:09,785 INFO balancer.Balancer: Using a threshold of 5.0
      2023-03-26 14:10:09,786 INFO balancer.Balancer: namenodes  = [hdfs://hadoop01:8020]
      2023-03-26 14:10:09,786 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]
      2023-03-26 14:10:09,786 INFO balancer.Balancer: included nodes = []
      2023-03-26 14:10:09,786 INFO balancer.Balancer: excluded nodes = []
      2023-03-26 14:10:09,786 INFO balancer.Balancer: source nodes = []
      Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved  NameNode
      2023-03-26 14:10:09,787 IpythonNFO balancer.NameNodeConnector: getBlocks calls for hdfs://hadoop01:8020 will be rate-limited to 20 per second
      2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.namenode.get-blocks.max-qps = 20 (default=20)
      2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)
      2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)
      2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)
      2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)
      2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)
      2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 100 (default=100)
      2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.datanode.balance.bandwidthPerSec = 104857600 (default=104857600)
      2023-03-26 14:10:10,395 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)
      2023-03-26 14:10:10,395 INFO balancer.Balancer: dfs.blocksize = 134217728 (default=134217728)
      2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.141:9866
      2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.140:9866
      2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.142:9866
      2023-03-26 14:10:10,402 INFO balancer.Balancer: 0 over-utilized: []
      2023-03-26 14:10:10,402 INFO balancer.Balancer: 0 underutilized: []
      2023-3-26 14:10:10                0                  0 B                 0 B                0 B                  0  hdfs://hadoop01:8020
      The cluster is balanced. Exiting...
      2023-3-26 14:10:10       Balancing took 810.0 milliseconds
      [hadoopdeploy@hadoop01 ~]$
      

      5、参考文档

      1、hadoop.apache.org/docs/curren…

      2、https://www.jb51.net/article/279671.htm

      以上就是HDFS Balancer负载均衡器及语法详解的详细内容,更多关于HDFS Balancer负载均衡器的资料请关注我们其它相关文章!

      0

      上一篇:

      下一篇:

      精彩评论

      暂无评论...
      验证码 换一张
      取 消

      最新开发

      开发排行榜