系统环境
CentOS 6.5
组件版本
name | version |
---|---|
Sqoop | 1.4.6 |
Sqoop安装
解压
懒的敲了
- 配置环境变量
# SQOOP
SQOOP_HOME=/home/Hadoop/development/src/sqoop-1.4.6-cdh5.6.0
PATH=$PATH:$SQOOP_HOME/bin
export PATH
cd 到Sqoop根目录
配置Sqoop核心文件
主要配置${SQOOP_HOME}/conf/sqoop-env-template.sh 文件
cp ${SQOOP_HOME}/conf/sqoop-env-template.sh ${SQOOP_HOME}/conf/sqoop-env.sh
主要修改参数
# 指定 Hadoop 安装目录
export HADOOP_COMMON_HOME=
# 指定 Hadoop MapReduce 技术jar存放目录
export HADOOP_MAPRED_HOME=
# Hbase 安装目录
export HBASE_HOME=
# Zookeeper 安装目录
export ZOOCFGDIR=
如果没有安装的可以不用填写
例如
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/home/hadoop/development/src/hadoop-2.6.0-cdh5.6.0
#Set path to where hadoop-*-core.jar is available
"这里之所以和Hadoop安装目录一样,是由于上次我配置成mapreduce jar地址时,sqoop报错"
export HADOOP_MAPRED_HOME=/home/hadoop/development/src/hadoop-2.6.0-cdh5.6.0
#set the path to where bin/hbase is available
"我没有安装Hbase所以这里直接 注释掉就可以"
#export HBASE_HOME=
#Set the path to where bin/hive is available
export HIVE_HOME=/home/hadoop/development/src/hive-1.1.0-cdh5.6.0
#Set the path for where zookeper config dir is
export ZOOCFGDIR=/home/hadoop/development/src/zookeeper-3.4.5-cdh5.6.0
测试Sqoop是否配置成功
sqoop list-databases --connect jdbc:mysql://192.168.2.251:3306/canbot --username root --password root
Warning: /home/hadoop/development/src/sqoop-1.4.6-cdh5.6.0/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/hadoop/development/src/sqoop-1.4.6-cdh5.6.0/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/hadoop/development/src/sqoop-1.4.6-cdh5.6.0/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/hadoop/development/src/sqoop-1.4.6-cdh5.6.0/bin/../../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
16/05/23 10:34:46 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.6.0
16/05/23 10:34:46 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/05/23 10:34:47 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
"information_schema
canbot
from_66internet
from_fhmysql
hive
mysql
Sqoop 数据导入到Hive中报错 HIVE_CONF_DIR
解决方法:export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*
相关阅读:
通过Sqoop实现Mysql / Oracle 与HDFS / Hbase互导数据 http://www.linuxidc.com/Linux/2013-06/85817.htm
[Hadoop] Sqoop安装过程详解 http://www.linuxidc.com/Linux/2013-05/84082.htm
用Sqoop进行MySQL和HDFS系统间的数据互导 http://www.linuxidc.com/Linux/2013-04/83447.htm
Hadoop Oozie学习笔记 Oozie不支持Sqoop问题解决 http://www.linuxidc.com/Linux/2012-08/67027.htm
Hadoop生态系统搭建(hadoop hive hbase zookeeper oozie Sqoop) http://www.linuxidc.com/Linux/2012-03/55721.htm
Hadoop学习全程记录——使用Sqoop将MySQL中数据导入到Hive中 http://www.linuxidc.com/Linux/2012-01/51993.htm
Sqoop 的详细介绍:请点这里
Sqoop 的下载地址:请点这里
更多Hadoop相关信息见Hadoop 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=13
本文永久更新链接地址:http://www.linuxidc.com/Linux/2016-08/134188.htm