趣搭-大集群
- 2019 年 12 月 30 日
- 筆記
大数据集群安装详解（Hadoop2.x）
搭建环境：RadHat5.5 HaDoop:hadoop-2.6.0-cdh5.5.2 JDK:jdk-7u25-linux-i586 vmware 虚拟机 三台 主节点：hdp-01 ip 192.168.6.2 从节点：hdp-02 ip 192.168.6.4 从节点：hdp-03 ip 192.168.6.6
将jdk-7u25-linux-i586.tar.gz 分别传到每台机器的/TMP下 将hadoop-2.6.0-cdh5.5.2.tar.gz 传入到主节点的/TMP下 注意要关闭防火墙 1./etc/init.d/iptables stop或者service iptables stop 2.chkconfig iptables off 3.setenforce 0 4.vim /etc/sysconfig/selinux ，修改为:SELINUX=disabled并且SELINUXTYPE=disabled.
开始安装JDK（三台都需要）
HDP-01 主节点
vim 1.sh #!/bin/bash cat <<EOF >/etc/sysconfig/network NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=hdp-01 EOF hostname hdp-01 cat <<EOF > /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 192.168.6.2 hdp-01 192.168.6.4 hdp-02 192.168.6.6 hdp-03 EOF useradd hadoop echo "123456" | passwd --stdin hadoop cd /tmp ls tar -zxvf jdk-7u25-linux-i586.tar.gz -C /usr/ cd /usr/ ls rpm -e --nodeps java-1.4.2-gcj-compat-1.4.2.0-40jpp.115 cd cat <<EOF >> /etc/profile export JAVA_HOME=/usr/jdk1.7.0_25 export JAVA_BIN=/usr/jdk1.7.0_25/bin export PATH=$PATH:$JAVA_HOME/bin export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export JAVA_HOME JAVA_BIN PATH CLASSPATH EOF source /etc/profile java -version tail -n 6 /etc/profile su - hadoop
HDP-02 从节点
vim 1.sh cat <<EOF >/etc/sysconfig/network NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=hdp-02 EOF hostname hdp-02 cat <<EOF > /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 192.168.6.2 hdp-01 192.168.6.4 hdp-02 192.168.6.6 hdp-03 EOF useradd hadoop echo "123456" | passwd --stdin hadoop cd /tmp ls tar -zxvf jdk-7u25-linux-i586.tar.gz -C /usr/ cd /usr/ cd cat <<EOF >> /etc/profile export JAVA_HOME=/usr/jdk1.7.0_25 export JAVA_BIN=/usr/jdk1.7.0_25/bin export PATH=$PATH:$JAVA_HOME/bin export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export JAVA_HOME JAVA_BIN PATH CLASSPATH EOF source /etc/profile /usr/jdk1.7.0_25/bin/java -version tail -n 6 /etc/profile su - hadoop
HDP-03 从节点
vim 1.sh cat <<EOF >/etc/sysconfig/network NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=hdp-03 EOF hostname hdp-03 cat <<EOF > /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 192.168.6.2 hdp-01 192.168.6.4 hdp-02 192.168.6.6 hdp-03 EOF useradd hadoop echo "123456" | passwd --stdin hadoop cd /tmp ls tar -zxvf jdk-7u25-linux-i586.tar.gz -C /usr/ cd /usr/ ls cd cat <<EOF >> /etc/profile export JAVA_HOME=/usr/jdk1.7.0_25 export JAVA_BIN=/usr/jdk1.7.0_25/bin export PATH=$PATH:$JAVA_HOME/bin export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export JAVA_HOME JAVA_BIN PATH CLASSPATH EOF source /etc/profile /usr/jdk1.7.0_25/bin/java -version echo $? tail -n 6 /etc/profile su - hadoop
分别安装完之后 都会有如下图所示(主节点)： 创建hadoop用户 密码为123456 自动su到hadoop用户下 java 版本以及环境变量 hadoop用户名
安装主节点 CDH (切换到root用户)
core-site.xml
vim core-site.xml <property> <name>fs.defaultFS</name> <value>hdfs://hdp-01:9000</value> <!--主机名--> <description>NameNode URI.</description> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> <description>Size of read/write buffer used inSequenceFiles.</description> </property>
cd /usr/local/hadoop-2.6.0/ mkdir -p dfs/name mkdir -p dfs/data mkdir -p dfs/namesecondary cd etc/hadoop
hdfs-site.xml
mapred-site.xml
