공개SW 설치/활용

자료년도 : 2012

Hive 설치 가이드


본 설치 가이드는 공개SW 역량프라자에서 클라우드 컴퓨팅 기반 기술 중 HIVE에 대한 테스트 결과 보고서 외에 테스트 환경에 대한 이해를 돕고자 작성되었습니다.
모든 테스트 환경 구성에 대한 내용은 포함되어 있지 않으며, 주의가 필요하거나 참고해야 할 내용을 기반으로 작성되었습니다.


1. 설치환경

□ KVM 환경

모듈 Version
Hive 0.9.0

□ Stack 환경

구성

OS

JDK

Hadoop
Master Node CentOS 6.3(64bit) 1.6.0_35 1.0.3
Slave Node 1 CentOS 6.3(64bit) 1.6.0_35 1.0.3
Slave Node 2 CentOS 6.3(64bit) 1.6.0_35 1.0.3
Slave Node 3 CentOS 6.3(64bit) 1.6.0_35 1.0.3

□ HW 환경

제조사

모델명

CPU

MEM Disk NIC
IBM X3850M2 Intel Xeon(R)CPU 2.40GHz * 4 8GB 320G Gigabit 1Port

※ 동일 사양의 HW Stack 구성



2. Server 설정

□ CentOS 설정


o SELinux 파일 수정

vi /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
# SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
reboot;

o iptables 중단

chkconfig iptables off
chkconfig ip6tables off

o iptables 중단

vi /etc/hosts
xxx.xxx.xxx.xxx master
xxx.xxx.xxx.xx1 slave1
xxx.xxx.xxx.xx2 slave2
xxx.xxx.xxx.xx3 slave3

o profile 파일 추가

vi /etc/profile
HADOOP_HOME="/usr/local/hadoop"
JAVA_HOME="/usr/local/java/j2se"
CLASSPATH=".:$JAVA_HOME/lib:$JAVA_HOME/jre/lib/ext"
PATH="$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin"
export HIVE_HOME="/usr/local/hadoop/hive"
export PATH="$HIVE_HOME/bin:$PATH"

o sshd_config 파일 수정

vi /etc/ssh/sshd_config
port 22
.....
PubkeyAuthentication yes
AuthorizeKeyFile .ssh/authorized_keys
.....
PermitEmptyPasswords yes
PermitRootLogin yes


□ Master Server

o master 공개키 생성

ssh-keygen
cd .ssh
scp ~/.ssh/id_rsa.pub xxx.xxx.xxx.xx1:~/.ssh
scp ~/.ssh/id_rsa.pub xxx.xxx.xxx.xx2:~/.ssh
scp ~/.ssh/id_rsa.pub xxx.xxx.xxx.xx3:~/.ssh

o slave1 공개키 생성

cd .ssh
cp id_rsa.pub authorized_keys
rm id_rsa.pub

o slave2 공개키 생성

cd .ssh
cp id_rsa.pub authorized_keys
rm id_rsa.pub

o slave3 공개키 생성

cd .ssh
cp id_rsa.pub authorized_keys
rm id_rsa.pub

o 디렉토리 설정

mkdir -p /usr/local/hadoop/work/mapred/system
mkdir -p /usr/local/hadoop/work/namenode

o hadoop-env.sh

vi /usr/local/hadoop/conf/hadoop-env.sh
export JAVA_HOME=/usr/local/java/jdk1.6.0_35
export HADOOP_CLASSPATH=/usr/local/hadoop/hadoop-examples-1.0.3.jar
for f in ${HIVE_HOME}/lib/*.jar; do
  HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:$f;
done
.....
export HADOOP_HEAPSIZE=2000
.....
export HADOOP_SSH_OPTS 주석풀기

o core-site.xml

vi /usr/local/hadoop/conf/core-site.xml
<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://xxx.xxx.xxx.xxx:9000</value>
</property>
</configuraton>

o hdfs-site.xml

vi /usr/local/hadoop/conf/hdfs-site.xml
<configuration>
<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>
<property>
  <name>dfs.name.dir</name>
  <value>/usr/local/hadoop/work/name</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/usr/local/hadoop/work/data</value>
</property>
<property>
  <name>dfs.block.size</name>
  <value>67108864</value>
</property>
</configuraton>

o mapred-site.xml

vi /usr/local/hadoop/conf/hdfs-site.xml
<configuration>
<property>
  <name>mapred.map.child.java.opts</name>
  <value>-Xmx1024m</value>
</property>
<property>
  <name>mapred.reduce.child.java.opts</name>
  <value>-Xmx1024m</value>
</property>
<property>
  <name>mapred.job.tracker</name>
  <value>hdfs://xxx.xxx.xxx.xxx:9001</value>
</property>
<property>
  <name>mapred.system.dir</name>
  <value>/usr/local/hadoop/work/mapred/system</value>
</property>
</configuraton>

o master

vi /usr/local/hadoop/conf/master
xxx.xxx.xxx.xxx

o slave

vi /usr/local/hadoop/conf/slave
xxx.xxx.xxx.xx1
xxx.xxx.xxx.xx2
xxx.xxx.xxx.xx3


□ slave2 Server

o 디렉토리 설정

mkdir -p /usr/local/hadoop/work/datanode
chmod 755 /usr/local/hadoop/work/datanode

o hadoop-env.sh

vi /usr/local/hadoop/conf/hadoop-env.sh
export JAVA_HOME=/usr/local/java/jdk1.6.0_35
export HADOOP_CLASSPATH=/usr/local/hadoop/hadoop-examples-1.0.3.jar
.....
export HADOOP_HEAPSIZE=2000
.....
export HADOOP_SSH_OPTS 주석풀기

o core-site.xml

vi /usr/local/hadoop/conf/core-site.xml
<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://xxx.xxx.xxx.xxx:9000</value>
</property>
</configuraton>

o hdfs-site.xml

vi /usr/local/hadoop/conf/hdfs-site.xml

<configuration>
<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>
<property>
  <name>dfs.name.dir</name>
  <value>/usr/local/hadoop/work/name</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/usr/local/hadoop/work/data</value>
</property>
<property>
  <name>dfs.block.size</name>
  <value>67108864</value>
</property>
</configuraton>


o mapred-site.xml

vi /usr/local/hadoop/conf/mapred-site.xml
<configuration>
<property>
  <name>mapred.map.child.java.opts</name>
  <value>-Xmx1024m</value>
</property>
<property>
  <name>mapred.reduce.child.java.opts</name>
  <value>-Xmx1024m</value>
</property>
<property>
  <name>mapred.job.tracker</name>
  <value>hdfs://xxx.xxx.xxx.xxx:9001</value>
</property>
<property>
  <name>mapred.system.dir</name>
  <value>/usr/local/hadoop/work/mapred/system</value>
</property>
</configuraton>

o master

vi /usr/local/hadoop/conf/master
xxx.xxx.xxx.xxx

o master

vi /usr/local/hadoop/conf/slave
xxx.xxx.xxx.xx1
xxx.xxx.xxx.xx2
xxx.xxx.xxx.xx3


□ slave3 Server

o 디렉토리 설정

mkdir -p /usr/local/hadoop/work/datanode
chmod 755 /usr/local/hadoop/work/datanode

o hadoop-env.sh

vi /usr/local/hadoop/conf/hadoop-env.sh
export JAVA_HOME=/usr/local/java/jdk1.6.0_35
export HADOOP_CLASSPATH=/usr/local/hadoop/hadoop-examples-1.0.3.jar
.....
export HADOOP_HEAPSIZE=2000
.....
export HADOOP_SSH_OPTS 주석풀기

o core-site.xml

vi /usr/local/hadoop/conf/core-site.xml
<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://xxx.xxx.xxx.xxx:9000</value>
</property>
</configuraton>

o hdfs-site.xml

vi /usr/local/hadoop/conf/hdfs-site.xml
<configuration>
<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>
<property>
  <name>dfs.name.dir</name>
  <value>/usr/local/hadoop/work/name</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/usr/local/hadoop/work/data</value>
</property>
<property>
  <name>dfs.block.size</name>
  <value>67108864</value>
</property>
</configuraton>

o mapred-site.xml

vi /usr/local/hadoop/conf/mapred-site.xml
<configuration>
<property>
  <name>mapred.map.child.java.opts</name>
  <value>-Xmx1024m</value>
</property>
<property>
  <name>mapred.reduce.child.java.opts</name>
  <value>-Xmx1024m</value>
</property>
<property>
  <name>mapred.job.tracker</name>
  <value>hdfs://xxx.xxx.xxx.xxx:9001</value>
</property>
<property>
  <name>mapred.system.dir</name>
  <value>/usr/local/hadoop/work/mapred/system</value>
</property>
</configuraton>

o master

vi /usr/local/hadoop/conf/master
xxx.xxx.xxx.xxx

o slave

vi /usr/local/hadoop/conf/slave
xxx.xxx.xxx.xx1
xxx.xxx.xxx.xx2
xxx.xxx.xxx.xx3


□ hadoop 시작

o Master Server

cd /usr/local/hadoop/bin
hadoop namenode -format
start-all.sh


□ HIVE 시작

o 디렉토리 생성

hadoop fs -mkdir /tmp
hadoop fs -mkdir /user/hive/warehouse
hadoop fs -chmod g+w /tmp
hadoop fs -chmod g+w /user/hive/warehouse
cd /usr/local/hadoop/hive/bin
hive
조회 수 :
18650
등록일 :
2013.10.01
03:52:29 (*.162.249.76)
List of Articles
번호 제목 날짜 조회 수
44 마이크로소프트 애저 IoT 개발 키트 시작하기 2017-10-27 412
43 How-To : 크롬 원격 데스크톱 앱 설치와 활용 2017-10-25 391
42 오픈소스 개발을 위한 Git 사용법 실습 2017-02-24 2816
41 워드 파일을 PDF로 변환하기 2016-12-06 3130
40 Alfresco 유저 가이드 2016-08-17 9200
39 [슬라이드] 젠킨스의 설치와 설정 2015-02-12 12738
38 Alfresco 설치 가이드 file 2013-10-29 24634
37 KVM 설치 가이드 file 2013-10-11 31715
» Hive 설치 가이드 file 2013-10-01 18650
35 Cassandra 테스트 환경 구축 가이드 file 2013-06-19 26315


사이트하단 로고, 하단메뉴, 트위터 바로가기

퀵메뉴모음
퀵메뉴열기
퀵메뉴닫기