大數據平台Hadoop集群搭建

  一、概念

  Hadoop是由java語言編寫的,在分散式伺服器集群上存儲海量數據並運行分散式分析應用的開源框架,其核心部件是HDFS與MapReduce。HDFS是一個分散式文件系統,類似mogilefs,但又不同於mogilefs,hdfs由存放文件元數據信息的namenode和存放數據的伺服器datanode組成;hdfs它不同於mogilefs,hdfs把元數據信息放在記憶體中,而mogilefs把元數據放在資料庫中;而對於hdfs的元數據信息持久化是依靠secondary name node(第二名稱節點),第二名稱節點並不是真正扮演名稱節點角色,它的主要任務是周期性地將編輯日誌合併至名稱空間鏡像文件中以免編輯日誌變得過大;它可以獨立運行在一個物理主機上,並需要同名稱節點同樣大小的記憶體資源來完成文件合併;另外它還保持一份名稱空間鏡像的副本,以防名稱節點掛了,丟失數據;然而根據其工作機制,第二名稱節點要滯後主節點,所以當主名稱節點掛掉以後,丟失數據是在所難免的;所以snn(secondary name node)保存鏡像副本的主要作用是儘可能的減少數據的丟失;MapReduce是一個計算框架,這種計算框架主要有兩個階段,第一階段是map計算;第二階段是Reduce計算;map計算的作用是把相同key的數據始終發送給同一個mapper進行計算;reduce就是把mapper計算的結果進行摺疊計算(我們可以理解為合併),最終得到一個結果;在hadoop v1版本是這樣的架構,v2就不是了,v2版本中把mapreduce框架拆分yarn框架和mapreduce,其計算任務可以跑在yarn框架上;所以hadoop v1核心就是hdfs+mapreduce兩個集群;v2的架構就是hdfs+yarn+mapreduce;

  HDFS架構

  提示:從上圖架構可以看到,客戶端訪問hdfs上的某一文件,首先要向namenode請求文件的元數據信息,然後nn就會告訴客戶端,訪問的文件在datanode上的位置,然後客戶端再依次向datanode請求對應的數據,最後拼接成一個完整的文件;這裡需要注意一個概念,datanode存放文件數據是按照文件大小和塊大小來切分存放的,什麼意思呢?比如一個文件100M大小,假設dn(datanode)上的塊大小為10M一塊,那麼它存放在dn上是把100M切分為10M一塊,共10塊,然後把這10塊數據分別存放在不同的dn上;同時這些塊分別存放在不同的dn上,還會分別在不同的dn上存在副本,這樣一來使得一個文件的數據塊被多個dn分散冗餘的存放;對於nn節點,它主要維護了那個文件的數據存放在那些節點,和那些dn存放了那些文件的數據塊(這個數據是通過dn周期性的向nn發送);我們可以理解為nn內部有兩張表分別記錄了那些文件的數據塊分別存放在那些dn上(以文件為中心),和那些dn存放了那些文件的數據塊(以節點為中心);從上面的描述不難想像,當nn掛掉以後,整個存放在hdfs上的文件都將找不到,所以在生產中我們會使用zk(zookeeper)來對nn節點做高可用;對於hdfs來講,它本質上不是內核文件系統,所以它依賴本地Linux文件系統;

  mapreduce計算過程

  提示:如上圖所示,首先mapreduce會把給定的數據切分為多個(切分之前通過程式員寫程式實現把給定的數據切分為多分,並抽取成kv鍵值對),然後啟動多個mapper對其進行map計算,多個mapper計算後的結果在通過combiner進行合併(combiner是有程式員編寫程式實現,主要實現合併規則),把相同key的值根據某種計算規則合併在一起,然後把結果在通過partitoner(分區器,這個分區器是通過程式員寫程式實現,主要實現對map後的結果和對應reducer進行關聯)分別發送給不同的reducer進行計算,最終每個reducer會產生一個最終的唯一結果;簡單講mapper的作用是讀入kv鍵值對,輸出新的kv鍵值對,會有新的kv產生;combiner的作用是把當前mapper生成的新kv鍵值對進行相同key的鍵值對進行合併,至於怎麼合併,合併規則是什麼是由程式員定義,所以combiner就是程式員寫的程式實現,本質上combiner是讀入kv鍵值對,輸出kv鍵值對,不會產生新的kv;partitioner的作用就是把combiner合併後的鍵值對進行調度至reducer,至於怎麼調度,該發往那個reducer,以及由幾個reducer進行處理,由程式員定義;最終reducer摺疊計算以後生成新的kv鍵值對;

  hadoop v1與v2架構

  提示:在hadoop v1的架構中,所有計算任務都跑在mapreduce之上,mapreduce就主要擔任了兩個角色,第一個是集群資源管理器和數據處理;到了hadoop v2 其架構就為hdfs+yarn+一堆任務,其實我們可以把一堆任務理解為v1中的mapreduce,不同於v1中的mapreduce,v2中mapreduce只負責數據計算,不在負責集群資源管理,集群資源管理由yarn實現;對於v2來講其計算任務都跑在了執yarn之上;對於hdfs來講,v1和v2中的作用都是一樣的,都是起存儲文件作用;

  hadoop v2 計算任務資源調度過程

  提示:rm(resource manager)收到客戶端的任務請求,此時rm會根據各dn上運行的nm(node manager)周期性報告的狀態信息來決定把客戶端的任務調度給那個nm來執行;當rm選定好nm後,就把任務發送給對應nm,對應nm內部會起一個appmaster(am)的容器,負責本次任務的主控端,而appmaster需要啟動container來運行任務,它會向rm請求,然後rm會根據am的請求在對應的nm上啟動一個或多個container;最後各container運行後的結果會發送給am,然後再由am返回給rm,rm再返回給客戶端;在這其中rm主要用來接收個nm發送的各節點狀態信息和資源調度以及接收各am計算任務後的結果並回饋給各客戶端;nm主要用來管理各node上的資源和上報狀態信息給rm;am主要用來管理各任務的資源申請和各任務執行後端結果返回給rm;

  hadoop生態圈

  提示:上圖是hadoop v2生態圈架構圖,其中hdfs和yarn是hadoop的核心組件,對於運行在其上的各種任務都必須依賴hadoop,也必須支援調用mapreduce介面;

  二、hadoop集群部署

  環境說明

名稱 角色 ip
node01 nn,snn,rm 192.168.0.41
node02 dn,nm 192.168.0.42
node03 dn,nm 192.168.0.43
node04 dn,nm 192.168.0.44

 

 

 

 

 

 

 

  各節點同步時間

  配置/etc/hosts解析個節點主機名

  各節點安裝jdk

yum install -y java-1.8.0-openjdk-devel

  提示:安裝devel包才會有jps命令

  驗證jdk是否安裝完成,版本是否正確,確定java命令所在位置

  添加JAVA_HOME環境變數

  驗證JAVA_HOME變數配置是否正確

  創建目錄,用於存放hadoop安裝包

mkdir /bigdata

  到此基礎環境就準備OK,接下來下載hadoop二進位包

[[email protected] ~]# wget //mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.9.2/hadoop-2.9.2.tar.gz
--2020-09-27 22:50:16--  //mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.9.2/hadoop-2.9.2.tar.gz
Resolving mirror.bit.edu.cn (mirror.bit.edu.cn)... 202.204.80.77, 219.143.204.117, 2001:da8:204:1205::22
Connecting to mirror.bit.edu.cn (mirror.bit.edu.cn)|202.204.80.77|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 366447449 (349M) [application/octet-stream]
Saving to: 『hadoop-2.9.2.tar.gz』

100%[============================================================================>] 366,447,449 1.44MB/s   in 2m 19s 

2020-09-27 22:52:35 (2.51 MB/s) - 『hadoop-2.9.2.tar.gz』 saved [366447449/366447449]

[[email protected] ~]# ls
hadoop-2.9.2.tar.gz
[[email protected] ~]#

  解壓hadoop-2.9.3.tar.gz到/bigdata/目錄,並將解壓到目錄鏈接至hadoop

  導出hadoop環境變數配置

[[email protected] ~]# cat /etc/profile.d/hadoop.sh
export HADOOP_HOME=/bigdata/hadoop
export PATH=$PATH:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin
export HADOOP_YARN_HOME=${HADOOP_HOME}
export HADOOP_MAPPERD_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
[[email protected] ~]# 

  創建hadoop用戶,並設置其密碼為admin

[[email protected] ~]# useradd hadoop
[[email protected] ~]# echo "admin" |passwd --stdin hadoop
Changing password for user hadoop.
passwd: all authentication tokens updated successfully.
[[email protected] ~]# 

  各節點間hadoop用戶做免密登錄

[[email protected] ~]$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:6CNhqdagySJXc4iRBVSoLENddO7JLZMCsdjQzqSFnmw [email protected]
The key's randomart image is:
+---[RSA 2048]----+
| o*==o .         |
| o=Bo o          |
|=oX+   .         |
|+E =.oo.+        |
|o.o B.oBS.       |
|.o * =. o        |
|=.+ o o          |
|oo   . .         |
|                 |
+----[SHA256]-----+
[[email protected] ~]$ ssh-copy-id node01
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub"
The authenticity of host 'node01 (192.168.0.41)' can't be established.
ECDSA key fingerprint is SHA256:lE8/Vyni4z8hsXaa8OMMlDpu3yOIRh6dLcIr+oE57oE.
ECDSA key fingerprint is MD5:14:59:02:30:c0:16:b8:6c:1a:84:c3:0f:a7:ac:67:b3.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'node01'"
and check to make sure that only the key(s) you wanted were added.

[[email protected] ~]$ scp -r ./.ssh node02:/home/hadoop/
The authenticity of host 'node02 (192.168.0.42)' can't be established.
ECDSA key fingerprint is SHA256:lE8/Vyni4z8hsXaa8OMMlDpu3yOIRh6dLcIr+oE57oE.
ECDSA key fingerprint is MD5:14:59:02:30:c0:16:b8:6c:1a:84:c3:0f:a7:ac:67:b3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node02,192.168.0.42' (ECDSA) to the list of known hosts.
[email protected]'s password: 
id_rsa                                                                                  100% 1679   636.9KB/s   00:00    
id_rsa.pub                                                                              100%  404   186.3KB/s   00:00    
known_hosts                                                                             100%  362   153.4KB/s   00:00    
authorized_keys                                                                         100%  404   203.9KB/s   00:00    
[[email protected] ~]$ scp -r ./.ssh node03:/home/hadoop/
The authenticity of host 'node03 (192.168.0.43)' can't be established.
ECDSA key fingerprint is SHA256:lE8/Vyni4z8hsXaa8OMMlDpu3yOIRh6dLcIr+oE57oE.
ECDSA key fingerprint is MD5:14:59:02:30:c0:16:b8:6c:1a:84:c3:0f:a7:ac:67:b3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node03,192.168.0.43' (ECDSA) to the list of known hosts.
[email protected]'s password:  
id_rsa                                                                                  100% 1679   755.1KB/s   00:00    
id_rsa.pub                                                                              100%  404   165.7KB/s   00:00    
known_hosts                                                                             100%  543   350.9KB/s   00:00    
authorized_keys                                                                         100%  404   330.0KB/s   00:00    
[[email protected] ~]$ scp -r ./.ssh node04:/home/hadoop/
The authenticity of host 'node04 (192.168.0.44)' can't be established.
ECDSA key fingerprint is SHA256:lE8/Vyni4z8hsXaa8OMMlDpu3yOIRh6dLcIr+oE57oE.
ECDSA key fingerprint is MD5:14:59:02:30:c0:16:b8:6c:1a:84:c3:0f:a7:ac:67:b3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node04,192.168.0.44' (ECDSA) to the list of known hosts.
[email protected]'s password: 
id_rsa                                                                                  100% 1679   707.0KB/s   00:00    
id_rsa.pub                                                                              100%  404   172.8KB/s   00:00    
known_hosts                                                                             100%  724   437.7KB/s   00:00    
authorized_keys                                                                         100%  404   165.2KB/s   00:00    
[[email protected] ~]$ 

  驗證:用node01去連接node02,node03,node04看看是否是免密登錄了

  創建數據目錄/data/hadoop/hdfs/{nn,snn,dn},並將其屬主屬組更改為hadoop

  進入到hadoop安裝目錄,創建其logs目錄,並將其安裝目錄的屬主和屬組更改為hadoop

  提示:以上所有步驟都需要在各節點挨著做一遍;

  配置hadoop的core-site.xml

  提示:hadoop的配置文件語法都是xml格式的配置文件,其中<property>和</property>是一對標籤,裡面用name標籤來引用配置的選項的key的名稱,其value標籤用來配置對應key的值;上面配置表示配置默認的文件系統地址;hdfs://node01:8020是hdfs文件系統訪問的地址;

  完整的配置

[[email protected] hadoop]# cat core-site.xml 
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://node01:8020</value>
        <final>true</final>
    </property>
</configuration>
[[email protected] hadoop]# 

View Code

  配置hdfs-site.xml

  提示:以上配置主要指定hdfs相關目錄以及訪問web埠信息,副本數量;

  完整的配置

[[email protected] hadoop]# cat hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
                    <property>
                        <name>dfs.replication</name>
                        <value>3</value>
                    </property>
                    <property>
                        <name>dfs.namenode.name.dir</name>
                        <value>file:///data/hadoop/hdfs/nn</value>
                    </property>
                    <property>
                         <name>dfs.namenode.secondary.http-address</name>
                         <value>node01:50090</value>
                    </property>
                    <property>
                        <name>dfs.namenode.http-address</name>
                        <value>node01:50070</value>
                    </property>
                    <property>
                        <name>dfs.datanode.data.dir</name>
                        <value>file:///data/hadoop/hdfs/dn</value>
                    </property>
                    <property>
                        <name>fs.checkpoint.dir</name>
                        <value>file:///data/hadoop/hdfs/snn</value>
                    </property>
                    <property>
                        <name>fs.checkpoint.edits.dir</name>
                        <value>file:///data/hadoop/hdfs/snn</value>
                    </property>

</configuration>
[[email protected] hadoop]# 

View Code

  配置mapred-site.xml

  提示:以上配置主要指定了mapreduce的框架為yarn;默認沒有mapred-site.xml,我們需要將mapred-site.xml.template修改成mapred.site.xml;這裡需要注意我上面是通過複製修改文件名,當然屬主信息都會變成root,不要忘記把屬組信息修改成hadoop;

  完整的配置

[[email protected] hadoop]# cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
                    <property>
                        <name>mapreduce.framework.name</name>
                        <value>yarn</value>
                    </property>

</configuration>
[[email protected] hadoop]# 

View Code

  配置yarn-site.xml

  提示:以上配置主要配置了yarn框架rm和nm相關地址和指定相關類;

  完整的配置

[[email protected] hadoop]# cat yarn-site.xml
<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

                    <property>
                        <name>yarn.resourcemanager.address</name>
                        <value>node01:8032</value>
                    </property>
                    <property>
                        <name>yarn.resourcemanager.scheduler.address</name>
                        <value>node01:8030</value>
                    </property>
                    <property>
                        <name>yarn.resourcemanager.resource-tracker.address</name>
                        <value>node01:8031</value>
                    </property>
                    <property>
                        <name>yarn.resourcemanager.admin.address</name>
                        <value>node01:8033</value>
                    </property>
                    <property>
                        <name>yarn.resourcemanager.webapp.address</name>
                        <value>node01:8088</value>
                    </property>
                    <property>
                        <name>yarn.nodemanager.aux-services</name>
                        <value>mapreduce_shuffle</value>
                    </property>
                    <property>
                        <name>yarn.nodemanager.auxservices.mapreduce_shuffle.class</name>
                        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
                    </property>
                    <property>
                        <name>yarn.resourcemanager.scheduler.class</name>
                        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
                    </property>

</configuration>
[[email protected] hadoop]# 

View Code

  配置slave.xml

[[email protected] hadoop]# cat slaves 
node02
node03
node04
[[email protected] hadoop]# 

  複製各配置文件到其他節點

  到此hadoop配置就完成了;

  接下來切換到hadoop用戶下,初始化hdfs

hdfs namenode -format

  提示:如果執行hdfs namenode -format 出現紅框中的提示,說明hdfs格式化就成功了;

  啟動hdfs集群

  提示:hdfs主要由namenode、secondarynamenode和datanode組成,只要看到對應節點上的進程啟動起來,就沒有多大問題;

  到此hdfs集群就正常啟動了

  驗證:把/etc/passwd上傳到hdfs的/test目錄下,看看是否可以正常上傳?

  提示:可以看到/etc/passwd文件已經上傳至hdfs的/test目錄下了;

  驗證:查看hdfs /test目錄下passwd文件,看看是否同/etc/passwd文件內容相同?

  提示:可以看到hdfs上的/test/passwd文件內容同/etc/passwd文件內容相同;

  驗證:在dn節點查看對應目錄下的文件內容,看看是否同/etc/passwd文件內容相同?

[[email protected] ~]# tree /data
/data
└── hadoop
    └── hdfs
        ├── dn
        │   ├── current
        │   │   ├── BP-157891879-192.168.0.41-1601224158145
        │   │   │   ├── current
        │   │   │   │   ├── finalized
        │   │   │   │   │   └── subdir0
        │   │   │   │   │       └── subdir0
        │   │   │   │   │           ├── blk_1073741825
        │   │   │   │   │           └── blk_1073741825_1001.meta
        │   │   │   │   ├── rbw
        │   │   │   │   └── VERSION
        │   │   │   ├── scanner.cursor
        │   │   │   └── tmp
        │   │   └── VERSION
        │   └── in_use.lock
        ├── nn
        └── snn

13 directories, 6 files
[[email protected] ~]# cat /data/hadoop/hdfs/dn/current/BP-157891879-192.168.0.41-1601224158145/
current/        scanner.cursor  tmp/            
[[email protected] ~]# cat /data/hadoop/hdfs/dn/current/BP-157891879-192.168.0.41-1601224158145/current/finalized/subdir0/subdir0/blk_1073741825
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:997:User for polkitd:/:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
ntp:x:38:38::/etc/ntp:/sbin/nologin
tcpdump:x:72:72::/:/sbin/nologin
chrony:x:998:996::/var/lib/chrony:/sbin/nologin
hadoop:x:1000:1000::/home/hadoop:/bin/bash
[[email protected] ~]# 

  提示:可以看到在dn節點上的dn目錄下能夠找到我們上傳的passwd文件;

  驗證:查看其它節點是否有相同的文件?是否有我們指定數量的副本?

  提示:在node03和node04上也有相同的目錄和文件;說明我們設置的副本數量為3生效了;

  啟動yarn集群

  提示:可以看到對應節點上的nm啟動了;主節點上的rm也正常啟動了;

  訪問nn的50070和8088,看看對應的web地址是否能夠訪問到頁面?

  提示:這個地址是hdfs的web地址,在這個界面可以看到hdfs的存儲狀況,以及對hdfs上的文件做操作;

  提示:8088是yarn集群的管理地址;在這個界面上能夠看到運行的計算任務的狀態信息,集群配置信息,日誌等等;

  驗證:在yarn上跑一個計算任務,統計/test/passwd文件的單詞數量,看看對應的計算任務是否能夠跑起來?

[[email protected] hadoop]$ yarn jar /bigdata/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar   
An example program must be given as the first argument.
Valid program names are:
  aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
  aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
  bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
  dbcount: An example job that count the pageview counts from a database.
  distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
  grep: A map/reduce program that counts the matches of a regex in the input.
  join: A job that effects a join over sorted, equally partitioned datasets
  multifilewc: A job that counts words from several files.
  pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
  pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
  randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
  randomwriter: A map/reduce program that writes 10GB of random data per node.
  secondarysort: An example defining a secondary sort to the reduce.
  sort: A map/reduce program that sorts the data written by the random writer.
  sudoku: A sudoku solver.
  teragen: Generate data for the terasort
  terasort: Run the terasort
  teravalidate: Checking results of terasort
  wordcount: A map/reduce program that counts the words in the input files.
  wordmean: A map/reduce program that counts the average length of the words in the input files.
  wordmedian: A map/reduce program that counts the median length of the words in the input files.
  wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.
[[email protected] hadoop]$ yarn jar /bigdata/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar wordcount
Usage: wordcount <in> [<in>...] <out>
[[email protected] hadoop]$ yarn jar /bigdata/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar wordcount /test/passwd /test/passwd-word-count20/09/28 00:58:01 INFO client.RMProxy: Connecting to ResourceManager at node01/192.168.0.41:8032
20/09/28 00:58:01 INFO input.FileInputFormat: Total input files to process : 1
20/09/28 00:58:01 INFO mapreduce.JobSubmitter: number of splits:1
20/09/28 00:58:01 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
20/09/28 00:58:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1601224871685_0001
20/09/28 00:58:02 INFO impl.YarnClientImpl: Submitted application application_1601224871685_0001
20/09/28 00:58:02 INFO mapreduce.Job: The url to track the job: //node01:8088/proxy/application_1601224871685_0001/
20/09/28 00:58:02 INFO mapreduce.Job: Running job: job_1601224871685_0001
20/09/28 00:58:08 INFO mapreduce.Job: Job job_1601224871685_0001 running in uber mode : false
20/09/28 00:58:08 INFO mapreduce.Job:  map 0% reduce 0%
20/09/28 00:58:14 INFO mapreduce.Job:  map 100% reduce 0%
20/09/28 00:58:20 INFO mapreduce.Job:  map 100% reduce 100%
20/09/28 00:58:20 INFO mapreduce.Job: Job job_1601224871685_0001 completed successfully
20/09/28 00:58:20 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=1144
                FILE: Number of bytes written=399079
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=1053
                HDFS: Number of bytes written=1018
                HDFS: Number of read operations=6
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=1
                Launched reduce tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=2753
                Total time spent by all reduces in occupied slots (ms)=2779
                Total time spent by all map tasks (ms)=2753
                Total time spent by all reduce tasks (ms)=2779
                Total vcore-milliseconds taken by all map tasks=2753
                Total vcore-milliseconds taken by all reduce tasks=2779
                Total megabyte-milliseconds taken by all map tasks=2819072
                Total megabyte-milliseconds taken by all reduce tasks=2845696
        Map-Reduce Framework
                Map input records=22
                Map output records=30
                Map output bytes=1078
                Map output materialized bytes=1144
                Input split bytes=95
                Combine input records=30
                Combine output records=30
                Reduce input groups=30
                Reduce shuffle bytes=1144
                Reduce input records=30
                Reduce output records=30
                Spilled Records=60
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=87
                CPU time spent (ms)=620
                Physical memory (bytes) snapshot=444997632
                Virtual memory (bytes) snapshot=4242403328
                Total committed heap usage (bytes)=285212672
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=958
        File Output Format Counters 
                Bytes Written=1018
[[email protected] hadoop]$ 

  查看計算後生成的報告

[[email protected] hadoop]$ hdfs dfs -ls -R /test
-rw-r--r--   3 hadoop supergroup        958 2020-09-28 00:32 /test/passwd
drwxr-xr-x   - hadoop supergroup          0 2020-09-28 00:58 /test/passwd-word-count
-rw-r--r--   3 hadoop supergroup          0 2020-09-28 00:58 /test/passwd-word-count/_SUCCESS
-rw-r--r--   3 hadoop supergroup       1018 2020-09-28 00:58 /test/passwd-word-count/part-r-00000
[[email protected] hadoop]$ hdfs dfs -cat /test/passwd-word-count/part-r-00000
Management:/:/sbin/nologin      1
Network 1
SSH:/var/empty/sshd:/sbin/nologin       1
User:/var/ftp:/sbin/nologin     1
adm:x:3:4:adm:/var/adm:/sbin/nologin    1
bin:x:1:1:bin:/bin:/sbin/nologin        1
bus:/:/sbin/nologin     1
chrony:x:998:996::/var/lib/chrony:/sbin/nologin 1
daemon:x:2:2:daemon:/sbin:/sbin/nologin 1
dbus:x:81:81:System     1
for     1
ftp:x:14:50:FTP 1
games:x:12:100:games:/usr/games:/sbin/nologin   1
hadoop:x:1000:1000::/home/hadoop:/bin/bash      1
halt:x:7:0:halt:/sbin:/sbin/halt        1
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin        1
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin  1
message 1
nobody:x:99:99:Nobody:/:/sbin/nologin   1
ntp:x:38:38::/etc/ntp:/sbin/nologin     1
operator:x:11:0:operator:/root:/sbin/nologin    1
polkitd:/:/sbin/nologin 1
polkitd:x:999:997:User  1
postfix:x:89:89::/var/spool/postfix:/sbin/nologin       1
root:x:0:0:root:/root:/bin/bash 1
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown    1
sshd:x:74:74:Privilege-separated        1
sync:x:5:0:sync:/sbin:/bin/sync 1
systemd-network:x:192:192:systemd       1
tcpdump:x:72:72::/:/sbin/nologin        1
[[email protected] hadoop]$ 

  在8088頁面上查看任務的狀態信息

  到此hadoop v2集群就搭建完畢了;