hadoop集群安装（先安装jdk，zookeeper，再安装hadoop）
①创建hadoop用户
groupadd hadoop #创建hadoop组
useradd -g hadoop hadoop #创建hadoop用户
passwd hadoop #创建hadoop密码
②配置免密码登录
#ssh-keygen -f .ssh/id_rsa -N ""
#生成秘钥和公钥
ssh-keygen -t rsa
#将公钥写入authorized_keys
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
#将每一个节点的公钥写入主节点的authorized_keys
cat ~/.ssh/id_rsa.pub | ssh 用户名@ip 'cat >> ~/.ssh/authorized_keys'
[root@hadoop ~]# cd /home/hadoop/.ssh
[root@hadoop .ssh]# chmod 710 authorized_keys #使用默认的权限时，普通用户的免密码认证无效，可试试600
#将写入所有节点公钥的authorized_keys发放到每个节点上
scp authorized_keys hadoop@192.8.8.12:~/.ssh/
③下载并拷贝hadoop-2.7.3.tar.gz到soft目录
mkdir /soft;cd /soft;tar -zxvf hadoop-2.7.3.tar.gz #解压
ln -s hadoop-2.7.3 hadoop #创建软链接
#更改目录所属用户（用root进行更改）为hadoop
chown -R hadoop:hadoop soft*
④修改各配置文件（6个文件）,使用hadoop用户
hadoop/etc/hadoop/hadoop-env.sh #修改JAVA_HOME
hadoop/etc/hadoop/yarn-env.sh #修改JAVA_HOME
hadoop/etc/hadoop/core-site.xml #增加属性，参考官方配置文档
hadoop/etc/hadoop/hdfs-site.xml #增加属性，参考官方配置文档
#cp mapred-site.xml.template mapred-site.xml
hadoop/etc/hadoop/mapred-site.xml
hadoop/etc/hadoop/yarn-site.xml #增加属性，参考官方配置文档
#具体内容请观察文件末尾
⑤增加hadoop环境变量
vi ~/.bashrc
export HADOOP_HOME=/soft/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
#使环境变量生效
source ~/.bashrc
#scp -qr ./hadoop-2.7.3 hostname@ip:/soft/hadoop-2.7.3
⑥启动：
a.先启动所有节点上的journalnode进程,每个节点都启动
cd /soft/hadoop
./sbin/hadoop-daemon.sh start journalnode
b.再格式化hdfs，只在主节点操作
./bin/hdfs namenode -format #格式化namenode
./bin/hdfs zkfc -formatZK #格式化高可用
./bin/hdfs namenode #启动namenode
c.同步主节点和备节点之间的元数据，在namenode存活的情况下在备用节点上执行
./bin/hdfs namenode -bootstrapStandby
#同步完数据之后，在主节点按下ctrl+c 结束namenode进程。
d.关闭所有节点上的journalnode进程
./sbin/hadoop-daemon.sh stop journalnode
e.单独启动一个zkfc进程
./sbin/hadoop-daemon.sh start zkfc
f.以上无误后，启动hdfs所有相关进程
./sbin/start-dfs.sh #./sbin/stop-dfs.sh
g.启动yarn
./sbin/start-yarn.sh #./sbin/stop-yarn.sh
h.在备用节点上执行
./sbin/yarn-daemon.sh start resourcemanager
i.检查ResourceManager状态
./bin/yarn rmadmin -getServiceState rm1
./bin/yarn rmadmin -getServiceState rm2
j.Wordcount示例测试
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /test/test.txt /test/out/
⑦停止：
./sbin/stop-all.sh
⑧强制Active/Standby切换命令
#NN切换
hdfs haadmin -transitionToActive/transitionToStandby -forcemanual nn1
#RM切换
yarn rmadmin -transitionToActive/transitionToStandby -forcemanual rm1
#这样做的后果是ZKFC将停止工作，不会再有自动故障切换的保障。
⑨其他命令
hdfs dfsadmin -refreshNodes #重新读取hosts和exclude文件
hdfs dfsadmin -safemode #安全模式维护命令
hdfs dfsadmin -report #报告文件系统的基本信息和统计信息
hdfs操作：
vi test.txt
hadoop apache
hadoop ywendeng
hadoop tomcat
hdfs dfs -mkdir /test #在hdfs上创建一个文件目录
hdfs dfs -put test.txt /test #向hdfs上传一个文件
hdfs dfs -ls/test #查看test.txt是否上传成功
ntp:
service ntpd start
service ntpd status
service ntpd stop
-------------------------------------------------------------------------------------------
故障：
Exception:原因-hadoop位数和操作系统不一致导致的
Java HotSpot(TM) Client VM warning:
You have loaded library /soft/hadoop-2.7.3/lib/native/libhadoop.so.1.0.0
which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>',
or link it with '-z noexecstack'.
17/11/03 01:20:27 WARN util.NativeCodeLoader:
Unable to load native-hadoop library for your platform...
using builtin-java classes where applicable
17/11/03 10:52:46 INFO ipc.Client: Retrying connect to server:
hadoop1/192.8.8.12:8033. Already tried 0 time(s);
retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From hadoop1/192.8.8.12 to hadoop1:8033 failed on connection exception:
java.net.ConnectException: 拒绝连接; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
#hadoop fs -ls hdfs://192.8.8.11:8033
#有一个resourcemanager没有开启
#datanode没启动起来可能是datanode与namenode#./current/VERSION中的clusterID不同