Hadoop配置機架感知(python腳
- 2020 年 1 月 6 日
- 筆記
昨天QQ群里提了一個hadoop運行效率分配的問題，總結一下，寫個文章。集群使用hadoop-1.0.3
有些hadoop集群在運行的時候，不完全是絕對平均的分配，不過需要儘可能平均的分配任務，避免某一台或者某幾台伺服器任務過重，其他伺服器無事可做。這個，一方面是需要用到balancer，一個就是機架感知了。
通常，balancer是自動啟動的。而機架感知則需要單獨配置和編寫腳本。不過，機架感知，不是說是感知哪個伺服器壞了，是根據機架位置的拓撲結構來選取伺服器進行任務的權重分配。
機架感知需要自己寫一個腳本，然後放到hadoop的core-site.xml配置文件中，用namenode和jobtracker進行調用。
python程式碼摘自竹葉青的部落格
#!/usr/bin/python #-*-coding:UTF-8 -*- import sys rack = {"hadoop-node-31":"rack1", "hadoop-node-32":"rack1", "hadoop-node-33":"rack1", "hadoop-node-34":"rack1", "hadoop-node-49":"rack2", "hadoop-node-50":"rack2", "hadoop-node-51":"rack2", "hadoop-node-52":"rack2", "hadoop-node-53":"rack2", "hadoop-node-54":"rack2", "192.168.1.31":"rack1", "192.168.1.32":"rack1", "192.168.1.33":"rack1", "192.168.1.34":"rack1", "192.168.1.49":"rack2", "192.168.1.50":"rack2", "192.168.1.51":"rack2", "192.168.1.52":"rack2", "192.168.1.53":"rack2", "192.168.1.54":"rack2", } if __name__=="__main__": print "/" + rack.get(sys.argv[1],"rack0")
按照老趙的部落格，由於hadoop沒有明確的說明機架感知是感知IP還是感知主機名，所以就都寫上了。然後將腳本賦予可執行許可權chmod +x RackAware.py，並放到bin/目錄下。
然後打開conf/core-site.html
放入一下程式碼
<property> <name>topology.script.file.name</name> <value>/opt/modules/hadoop/hadoop-1.0.3/bin/RackAware.py</value> <!--機架感知腳本路徑--> </property> <property> <name>topology.script.number.args</name> <value>20</value> <!--機架伺服器數量，由於我寫了20個，所以這裡寫20--> </property>
然後重啟hadoop的namenode和jobtracker，可以在logs里看下namenode和jobtracker的日誌，看到機架感知功能已經啟用了。
namenode日誌
2012-06-08 14:42:19,174 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.49:50010 storage DS-1155827498-192.168.1.49-50010-1338289368956 2012-06-08 14:42:19,204 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/192.168.1.49:50010 2012-06-08 14:42:19,205 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.53:50010 storage DS-1773813988-192.168.1.53-50010-1338289405131 2012-06-08 14:42:19,226 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/192.168.1.53:50010 2012-06-08 14:42:19,226 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.34:50010 storage DS-2024494948-127.0.0.1-50010-1338289438983 2012-06-08 14:42:19,242 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack1/192.168.1.34:50010 2012-06-08 14:42:19,242 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.54:50010 storage DS-767528606-192.168.1.54-50010-1338289412267 ...... 2012-06-08 14:42:49,492 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 2 racks and 10 datanodes 2012-06-08 14:42:49,492 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks 2012-06-08 14:42:49,642 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 0 msec 2012-06-08 14:42:49,642 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 0 msec processing time, 0 msec clock time, 1 cycles
jobtracker 日誌
2012-06-08 14:50:42,728 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/hadoop-node-52 2012-06-08 14:50:42,731 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_hadoop-node-52:localhost.localdomain/127.0.0.1:33999 to host hadoop-node-52 2012-06-08 14:50:42,753 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack1/hadoop-node-34 2012-06-08 14:50:42,754 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_hadoop-node-34:localhost.localdomain/127.0.0.1:37025 to host hadoop-node-34 2012-06-08 14:50:42,773 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/hadoop-node-51
