趣搭-大集群
- 2019 年 12 月 30 日
- 筆記
大數據集群安裝詳解（Hadoop2.x）
搭建環境：RadHat5.5 HaDoop:hadoop-2.6.0-cdh5.5.2 JDK:jdk-7u25-linux-i586 vmware 虛擬機 三台 主節點：hdp-01 ip 192.168.6.2 從節點：hdp-02 ip 192.168.6.4 從節點：hdp-03 ip 192.168.6.6
將jdk-7u25-linux-i586.tar.gz 分別傳到每台機器的/TMP下 將hadoop-2.6.0-cdh5.5.2.tar.gz 傳入到主節點的/TMP下 注意要關閉防火牆 1./etc/init.d/iptables stop或者service iptables stop 2.chkconfig iptables off 3.setenforce 0 4.vim /etc/sysconfig/selinux ，修改為:SELINUX=disabled並且SELINUXTYPE=disabled.
開始安裝JDK（三台都需要）
HDP-01 主節點
vim 1.sh #!/bin/bash cat <<EOF >/etc/sysconfig/network NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=hdp-01 EOF hostname hdp-01 cat <<EOF > /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 192.168.6.2 hdp-01 192.168.6.4 hdp-02 192.168.6.6 hdp-03 EOF useradd hadoop echo "123456" | passwd --stdin hadoop cd /tmp ls tar -zxvf jdk-7u25-linux-i586.tar.gz -C /usr/ cd /usr/ ls rpm -e --nodeps java-1.4.2-gcj-compat-1.4.2.0-40jpp.115 cd cat <<EOF >> /etc/profile export JAVA_HOME=/usr/jdk1.7.0_25 export JAVA_BIN=/usr/jdk1.7.0_25/bin export PATH=$PATH:$JAVA_HOME/bin export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export JAVA_HOME JAVA_BIN PATH CLASSPATH EOF source /etc/profile java -version tail -n 6 /etc/profile su - hadoop
HDP-02 從節點
vim 1.sh cat <<EOF >/etc/sysconfig/network NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=hdp-02 EOF hostname hdp-02 cat <<EOF > /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 192.168.6.2 hdp-01 192.168.6.4 hdp-02 192.168.6.6 hdp-03 EOF useradd hadoop echo "123456" | passwd --stdin hadoop cd /tmp ls tar -zxvf jdk-7u25-linux-i586.tar.gz -C /usr/ cd /usr/ cd cat <<EOF >> /etc/profile export JAVA_HOME=/usr/jdk1.7.0_25 export JAVA_BIN=/usr/jdk1.7.0_25/bin export PATH=$PATH:$JAVA_HOME/bin export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export JAVA_HOME JAVA_BIN PATH CLASSPATH EOF source /etc/profile /usr/jdk1.7.0_25/bin/java -version tail -n 6 /etc/profile su - hadoop
HDP-03 從節點
vim 1.sh cat <<EOF >/etc/sysconfig/network NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=hdp-03 EOF hostname hdp-03 cat <<EOF > /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 192.168.6.2 hdp-01 192.168.6.4 hdp-02 192.168.6.6 hdp-03 EOF useradd hadoop echo "123456" | passwd --stdin hadoop cd /tmp ls tar -zxvf jdk-7u25-linux-i586.tar.gz -C /usr/ cd /usr/ ls cd cat <<EOF >> /etc/profile export JAVA_HOME=/usr/jdk1.7.0_25 export JAVA_BIN=/usr/jdk1.7.0_25/bin export PATH=$PATH:$JAVA_HOME/bin export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export JAVA_HOME JAVA_BIN PATH CLASSPATH EOF source /etc/profile /usr/jdk1.7.0_25/bin/java -version echo $? tail -n 6 /etc/profile su - hadoop
分別安裝完之後 都會有如下圖所示(主節點)： 創建hadoop用戶 密碼為123456 自動su到hadoop用戶下 java 版本以及環境變量 hadoop用戶名
安裝主節點 CDH (切換到root用戶)
core-site.xml
vim core-site.xml <property> <name>fs.defaultFS</name> <value>hdfs://hdp-01:9000</value> <!--主機名--> <description>NameNode URI.</description> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> <description>Size of read/write buffer used inSequenceFiles.</description> </property>
cd /usr/local/hadoop-2.6.0/ mkdir -p dfs/name mkdir -p dfs/data mkdir -p dfs/namesecondary cd etc/hadoop
hdfs-site.xml
mapred-site.xml
