본문 바로가기

2012

Hive 설치 가이드

OSS 2013-09-30 18:52:29 1387

Hive 설치 가이드


본 설치 가이드는 공개SW 역량프라자에서 클라우드 컴퓨팅 기반 기술 중 HIVE에 대한 테스트 결과 보고서 외에 테스트 환경에 대한 이해를 돕고자 작성되었습니다.
모든 테스트 환경 구성에 대한 내용은 포함되어 있지 않으며, 주의가 필요하거나 참고해야 할 내용을 기반으로 작성되었습니다.


1. 설치환경

□ KVM 환경

모듈 Version
Hive 0.9.0

□ Stack 환경

구성

OS

JDK

Hadoop
Master Node CentOS 6.3(64bit) 1.6.0_35 1.0.3
Slave Node 1 CentOS 6.3(64bit) 1.6.0_35 1.0.3
Slave Node 2 CentOS 6.3(64bit) 1.6.0_35 1.0.3
Slave Node 3 CentOS 6.3(64bit) 1.6.0_35 1.0.3

□ HW 환경

제조사

모델명

CPU

MEM Disk NIC
IBM X3850M2 Intel Xeon(R)CPU 2.40GHz * 4 8GB 320G Gigabit 1Port

※ 동일 사양의 HW Stack 구성



2. Server 설정

□ CentOS 설정


o SELinux 파일 수정

vi /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
# SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
reboot;

o iptables 중단

chkconfig iptables off
chkconfig ip6tables off

o iptables 중단

vi /etc/hosts
xxx.xxx.xxx.xxx master
xxx.xxx.xxx.xx1 slave1
xxx.xxx.xxx.xx2 slave2
xxx.xxx.xxx.xx3 slave3

o profile 파일 추가

vi /etc/profile
HADOOP_HOME="/usr/local/hadoop"
JAVA_HOME="/usr/local/java/j2se"
CLASSPATH=".:$JAVA_HOME/lib:$JAVA_HOME/jre/lib/ext"
PATH="$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin"
export HIVE_HOME="/usr/local/hadoop/hive"
export PATH="$HIVE_HOME/bin:$PATH"

o sshd_config 파일 수정

vi /etc/ssh/sshd_config
port 22
.....
PubkeyAuthentication yes
AuthorizeKeyFile .ssh/authorized_keys
.....
PermitEmptyPasswords yes
PermitRootLogin yes


□ Master Server

o master 공개키 생성

ssh-keygen
cd .ssh
scp ~/.ssh/id_rsa.pub xxx.xxx.xxx.xx1:~/.ssh
scp ~/.ssh/id_rsa.pub xxx.xxx.xxx.xx2:~/.ssh
scp ~/.ssh/id_rsa.pub xxx.xxx.xxx.xx3:~/.ssh

o slave1 공개키 생성

cd .ssh
cp id_rsa.pub authorized_keys
rm id_rsa.pub

o slave2 공개키 생성

cd .ssh
cp id_rsa.pub authorized_keys
rm id_rsa.pub

o slave3 공개키 생성

cd .ssh
cp id_rsa.pub authorized_keys
rm id_rsa.pub

o 디렉토리 설정

mkdir -p /usr/local/hadoop/work/mapred/system
mkdir -p /usr/local/hadoop/work/namenode

o hadoop-env.sh

vi /usr/local/hadoop/conf/hadoop-env.sh
export JAVA_HOME=/usr/local/java/jdk1.6.0_35
export HADOOP_CLASSPATH=/usr/local/hadoop/hadoop-examples-1.0.3.jar
for f in ${HIVE_HOME}/lib/*.jar; do
  HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:$f;
done
.....
export HADOOP_HEAPSIZE=2000
.....
export HADOOP_SSH_OPTS 주석풀기

o core-site.xml

vi /usr/local/hadoop/conf/core-site.xml
<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://xxx.xxx.xxx.xxx:9000</value>
</property>
</configuraton>

o hdfs-site.xml

vi /usr/local/hadoop/conf/hdfs-site.xml
<configuration>
<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>
<property>
  <name>dfs.name.dir</name>
  <value>/usr/local/hadoop/work/name</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/usr/local/hadoop/work/data</value>
</property>
<property>
  <name>dfs.block.size</name>
  <value>67108864</value>
</property>
</configuraton>

o mapred-site.xml

vi /usr/local/hadoop/conf/hdfs-site.xml
<configuration>
<property>
  <name>mapred.map.child.java.opts</name>
  <value>-Xmx1024m</value>
</property>
<property>
  <name>mapred.reduce.child.java.opts</name>
  <value>-Xmx1024m</value>
</property>
<property>
  <name>mapred.job.tracker</name>
  <value>hdfs://xxx.xxx.xxx.xxx:9001</value>
</property>
<property>
  <name>mapred.system.dir</name>
  <value>/usr/local/hadoop/work/mapred/system</value>
</property>
</configuraton>

o master

vi /usr/local/hadoop/conf/master
xxx.xxx.xxx.xxx

o slave

vi /usr/local/hadoop/conf/slave
xxx.xxx.xxx.xx1
xxx.xxx.xxx.xx2
xxx.xxx.xxx.xx3


□ slave2 Server

o 디렉토리 설정

mkdir -p /usr/local/hadoop/work/datanode
chmod 755 /usr/local/hadoop/work/datanode

o hadoop-env.sh

vi /usr/local/hadoop/conf/hadoop-env.sh
export JAVA_HOME=/usr/local/java/jdk1.6.0_35
export HADOOP_CLASSPATH=/usr/local/hadoop/hadoop-examples-1.0.3.jar
.....
export HADOOP_HEAPSIZE=2000
.....
export HADOOP_SSH_OPTS 주석풀기

o core-site.xml

vi /usr/local/hadoop/conf/core-site.xml
<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://xxx.xxx.xxx.xxx:9000</value>
</property>
</configuraton>

o hdfs-site.xml

vi /usr/local/hadoop/conf/hdfs-site.xml

<configuration>
<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>
<property>
  <name>dfs.name.dir</name>
  <value>/usr/local/hadoop/work/name</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/usr/local/hadoop/work/data</value>
</property>
<property>
  <name>dfs.block.size</name>
  <value>67108864</value>
</property>
</configuraton>


o mapred-site.xml

vi /usr/local/hadoop/conf/mapred-site.xml
<configuration>
<property>
  <name>mapred.map.child.java.opts</name>
  <value>-Xmx1024m</value>
</property>
<property>
  <name>mapred.reduce.child.java.opts</name>
  <value>-Xmx1024m</value>
</property>
<property>
  <name>mapred.job.tracker</name>
  <value>hdfs://xxx.xxx.xxx.xxx:9001</value>
</property>
<property>
  <name>mapred.system.dir</name>
  <value>/usr/local/hadoop/work/mapred/system</value>
</property>
</configuraton>

o master

vi /usr/local/hadoop/conf/master
xxx.xxx.xxx.xxx

o master

vi /usr/local/hadoop/conf/slave
xxx.xxx.xxx.xx1
xxx.xxx.xxx.xx2
xxx.xxx.xxx.xx3


□ slave3 Server

o 디렉토리 설정

mkdir -p /usr/local/hadoop/work/datanode
chmod 755 /usr/local/hadoop/work/datanode

o hadoop-env.sh

vi /usr/local/hadoop/conf/hadoop-env.sh
export JAVA_HOME=/usr/local/java/jdk1.6.0_35
export HADOOP_CLASSPATH=/usr/local/hadoop/hadoop-examples-1.0.3.jar
.....
export HADOOP_HEAPSIZE=2000
.....
export HADOOP_SSH_OPTS 주석풀기

o core-site.xml

vi /usr/local/hadoop/conf/core-site.xml
<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://xxx.xxx.xxx.xxx:9000</value>
</property>
</configuraton>

o hdfs-site.xml

vi /usr/local/hadoop/conf/hdfs-site.xml
<configuration>
<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>
<property>
  <name>dfs.name.dir</name>
  <value>/usr/local/hadoop/work/name</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/usr/local/hadoop/work/data</value>
</property>
<property>
  <name>dfs.block.size</name>
  <value>67108864</value>
</property>
</configuraton>

o mapred-site.xml

vi /usr/local/hadoop/conf/mapred-site.xml
<configuration>
<property>
  <name>mapred.map.child.java.opts</name>
  <value>-Xmx1024m</value>
</property>
<property>
  <name>mapred.reduce.child.java.opts</name>
  <value>-Xmx1024m</value>
</property>
<property>
  <name>mapred.job.tracker</name>
  <value>hdfs://xxx.xxx.xxx.xxx:9001</value>
</property>
<property>
  <name>mapred.system.dir</name>
  <value>/usr/local/hadoop/work/mapred/system</value>
</property>
</configuraton>

o master

vi /usr/local/hadoop/conf/master
xxx.xxx.xxx.xxx

o slave

vi /usr/local/hadoop/conf/slave
xxx.xxx.xxx.xx1
xxx.xxx.xxx.xx2
xxx.xxx.xxx.xx3


□ hadoop 시작

o Master Server

cd /usr/local/hadoop/bin
hadoop namenode -format
start-all.sh


□ HIVE 시작

o 디렉토리 생성

hadoop fs -mkdir /tmp
hadoop fs -mkdir /user/hive/warehouse
hadoop fs -chmod g+w /tmp
hadoop fs -chmod g+w /user/hive/warehouse
cd /usr/local/hadoop/hive/bin
hive
공개SW 설치/활용 - 번호, 제목, 작성자
번호 제목 작성자 조회수 작성
39 Alfresco 유저 가이드 OSS 4831 2016-08-16
38 Alfresco 설치 가이드 file OSS 2343 2013-10-28
37 KVM 설치 가이드 file OSS 3015 2013-10-11
36 Hive 설치 가이드 file OSS 1387 2013-09-30
35 Cassandra 테스트 환경 구축 가이드 file OSS 1514 2013-06-18
34 활용도가 높은 메모장 '톰보이' 생생한 사용기. file 소재성 1345 2011-10-28
33 Audacity를 이용한 소리 편집 DK 2168 2011-10-28
32 공개 소프트웨어 실 업무 적응기 큰사람 3121 2011-10-28
31 OSS를 활용하여 OSS 저작권 문제를 관리하자!! - Fossology file primobeat 1576 2011-10-28
30 Stellarium (스텔라리움) - 오픈소스 플레타리움 file 태권브이 2497 2011-10-28