Hadoop配置机架感知(python脚
- 2020 年 1 月 6 日
- 笔记
昨天QQ群里提了一个hadoop运行效率分配的问题，总结一下，写个文章。集群使用hadoop-1.0.3
有些hadoop集群在运行的时候，不完全是绝对平均的分配，不过需要尽可能平均的分配任务，避免某一台或者某几台服务器任务过重，其他服务器无事可做。这个，一方面是需要用到balancer，一个就是机架感知了。
通常，balancer是自动启动的。而机架感知则需要单独配置和编写脚本。不过，机架感知，不是说是感知哪个服务器坏了，是根据机架位置的拓扑结构来选取服务器进行任务的权重分配。
机架感知需要自己写一个脚本，然后放到hadoop的core-site.xml配置文件中，用namenode和jobtracker进行调用。
python代码摘自竹叶青的博客
#!/usr/bin/python #-*-coding:UTF-8 -*- import sys rack = {"hadoop-node-31":"rack1", "hadoop-node-32":"rack1", "hadoop-node-33":"rack1", "hadoop-node-34":"rack1", "hadoop-node-49":"rack2", "hadoop-node-50":"rack2", "hadoop-node-51":"rack2", "hadoop-node-52":"rack2", "hadoop-node-53":"rack2", "hadoop-node-54":"rack2", "192.168.1.31":"rack1", "192.168.1.32":"rack1", "192.168.1.33":"rack1", "192.168.1.34":"rack1", "192.168.1.49":"rack2", "192.168.1.50":"rack2", "192.168.1.51":"rack2", "192.168.1.52":"rack2", "192.168.1.53":"rack2", "192.168.1.54":"rack2", } if __name__=="__main__": print "/" + rack.get(sys.argv[1],"rack0")
按照老赵的博客，由于hadoop没有明确的说明机架感知是感知IP还是感知主机名，所以就都写上了。然后将脚本赋予可执行权限chmod +x RackAware.py，并放到bin/目录下。
然后打开conf/core-site.html
放入一下代码
<property> <name>topology.script.file.name</name> <value>/opt/modules/hadoop/hadoop-1.0.3/bin/RackAware.py</value> <!–机架感知脚本路径–> </property> <property> <name>topology.script.number.args</name> <value>20</value> <!–机架服务器数量，由于我写了20个，所以这里写20–> </property>
然后重启hadoop的namenode和jobtracker，可以在logs里看下namenode和jobtracker的日志，看到机架感知功能已经启用了。
namenode日志
2012-06-08 14:42:19,174 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.49:50010 storage DS-1155827498-192.168.1.49-50010-1338289368956 2012-06-08 14:42:19,204 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/192.168.1.49:50010 2012-06-08 14:42:19,205 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.53:50010 storage DS-1773813988-192.168.1.53-50010-1338289405131 2012-06-08 14:42:19,226 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/192.168.1.53:50010 2012-06-08 14:42:19,226 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.34:50010 storage DS-2024494948-127.0.0.1-50010-1338289438983 2012-06-08 14:42:19,242 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack1/192.168.1.34:50010 2012-06-08 14:42:19,242 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.54:50010 storage DS-767528606-192.168.1.54-50010-1338289412267 …… 2012-06-08 14:42:49,492 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 2 racks and 10 datanodes 2012-06-08 14:42:49,492 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks 2012-06-08 14:42:49,642 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 0 msec 2012-06-08 14:42:49,642 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 0 msec processing time, 0 msec clock time, 1 cycles
jobtracker 日志
2012-06-08 14:50:42,728 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/hadoop-node-52 2012-06-08 14:50:42,731 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_hadoop-node-52:localhost.localdomain/127.0.0.1:33999 to host hadoop-node-52 2012-06-08 14:50:42,753 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack1/hadoop-node-34 2012-06-08 14:50:42,754 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_hadoop-node-34:localhost.localdomain/127.0.0.1:37025 to host hadoop-node-34 2012-06-08 14:50:42,773 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/hadoop-node-51
