【甘道夫】Sqoop1.99.3基础操作--导入Oracle的数据到HDFS
【甘道夫】Sqoop1.99.3基础操作--导入Oracle的数据到HDFS
第一步：进入客户端Shell fulong@FBI008:~$ sqoop.sh client Sqoop home directory: /home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200 Sqoop Shell: Type 'help' or '\h' for help.sqoop:000> set server --host FBI003 --port 12000 --webapp sqoop Server is set successfully sqoop:000> show version --all client version: Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013 server version: Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013 Protocol version: [1]
第二步：创建连接
sqoop:000> create connection --cid 001 Creating connection for connector with id 1 Please fill following values to create new connection object Name: 147patents
Connection configuration
JDBC Driver Class: oracle.jdbc.driver.OracleDriver JDBC Connection String: jdbc:oracle:thin:@192.168.0.147:1521:ORCLGBK Username: zhaobiao Password: ****** JDBC Connection Properties: There are currently 0 values in the map: entry#
Security related configuration options
Max connections: 100 New connection was successfully created with validation status FINE and persistent id 1
第三步：创建Job
我这里顺便尝试了一下update命令，所以第一遍创建job时输入了错误的tablename：
sqoop:000> create job Required argument --xid is missing. sqoop:000> create job --xid 1 --type import Creating job for connection with id 1 Please fill following values to create new job object Name: ImportPatents
Database configuration
Schema name: zhaobiao Table name: zhaobiao Table SQL statement: Table column names: Partition column name: Nulls in partition column: Boundary query:
Output configuration
Storage type: 0 : HDFS Choose: 0 Output format: 0 : TEXT_FILE 1 : SEQUENCE_FILE Choose: 0 Compression format: 0 : NONE 1 : DEFAULT 2 : DEFLATE 3 : GZIP 4 : BZIP2 5 : LZO 6 : LZ4 7 : SNAPPY Choose: 0 Output directory: /Data/zhaobiao
Throttling resources
Extractors: Loaders: New job was successfully created with validation status FINE and persistent id 1
sqoop:000> update job Required argument --jid is missing. sqoop:000> update job --jid 1 Updating job with id 1 Please update job metadata: Name: ImportPatents
Database configuration
Schema name: zhaobiao Table name: CMS_NEWS_0625 Table SQL statement: Table column names: Partition column name: Nulls in partition column: Boundary query:
Output configuration
Storage type: 0 : HDFS Choose: 0 Output format: 0 : TEXT_FILE 1 : SEQUENCE_FILE Choose: 0 Compression format: 0 : NONE 1 : DEFAULT 2 : DEFLATE 3 : GZIP 4 : BZIP2 5 : LZO 6 : LZ4 7 : SNAPPY Choose: 0 Output directory: /Data/zhaobiao
Throttling resources
Extractors: Loaders: Job was successfully updated with status FINE
总结： 1.create job时必须指定一个连接id 2.create job完成后，自动分配了一个job id 3.update job时，需要指定job id 4.创建Job过程中指定的Output directory指定的是HDFS中的路径
第四步：执行job
sqoop:000> start job --jid 1 2014-06-27 12:44:17 CST: FAILURE_ON_SUBMIT Exception: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://fulonghadoop/Data/zhaobiao already exists sqoop:000> start job --jid 1 Submission details Job ID: 1 Server URL: http://FBI003:12000/sqoop/ Created by: fulong Creation date: 2014-06-27 12:44:53 CST Lastly updated by: fulong External ID: job_1403248336137_0005 http://FBI003:8088/proxy/application_1403248336137_0005/ 2014-06-27 12:44:53 CST: BOOTING - Progress is not available sqoop:000> status job --jid 1 Submission details Job ID: 1 Server URL: http://FBI003:12000/sqoop/ Created by: fulong Creation date: 2014-06-27 12:44:53 CST Lastly updated by: fulong External ID: job_1403248336137_0005 http://FBI003:8088/proxy/application_1403248336137_0005/ 2014-06-27 12:50:11 CST: RUNNING - 0.00 %
sqoop:000> status job --jid 1 Submission details Job ID: 1 Server URL: http://FBI003:12000/sqoop/ Created by: fulong Creation date: 2014-06-27 12:44:53 CST Lastly updated by: fulong External ID: job_1403248336137_0005 http://FBI003:8088/proxy/application_1403248336137_0005/ 2014-06-27 13:36:49 CST: RUNNING - 25.00 % sqoop:000> status job --jid 1 Submission details Job ID: 1 Server URL: http://FBI003:12000/sqoop/ Created by: fulong Creation date: 2014-06-27 12:44:53 CST Lastly updated by: fulong External ID: job_1403248336137_0005 http://FBI003:8088/proxy/application_1403248336137_0005/ 2014-06-27 13:39:24 CST: RUNNING - 35.00 % sqoop:000> status job --jid 1 Submission details Job ID: 1 Server URL: http://FBI003:12000/sqoop/ Created by: fulong Creation date: 2014-06-27 12:44:53 CST Lastly updated by: fulong External ID: job_1403248336137_0005 http://FBI003:8088/proxy/application_1403248336137_0005/ 2014-06-27 13:47:19 CST: RUNNING - 40.00 % sqoop:000> status job --jid 1 Submission details Job ID: 1 Server URL: http://FBI003:12000/sqoop/ Created by: fulong Creation date: 2014-06-27 12:44:53 CST Lastly updated by: fulong External ID: job_1403248336137_0005 http://FBI003:8088/proxy/application_1403248336137_0005/ 2014-06-27 14:08:36 CST: SUCCEEDED Counters: org.apache.hadoop.mapreduce.JobCounter SLOTS_MILLIS_MAPS: 16499226 TOTAL_LAUNCHED_MAPS: 10 SLOTS_MILLIS_REDUCES: 0 OTHER_LOCAL_MAPS: 15 org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter BYTES_WRITTEN: 421226094 org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter BYTES_READ: 0 org.apache.hadoop.mapreduce.TaskCounter MAP_INPUT_RECORDS: 0 MERGED_MAP_OUTPUTS: 0 PHYSICAL_MEMORY_BYTES: 1185689600 SPILLED_RECORDS: 0 COMMITTED_HEAP_BYTES: 627179520 CPU_MILLISECONDS: 307570 FAILED_SHUFFLE: 0 VIRTUAL_MEMORY_BYTES: 6542311424 SPLIT_RAW_BYTES: 1437 MAP_OUTPUT_RECORDS: 644083 GC_TIME_MILLIS: 5279 org.apache.hadoop.mapreduce.FileSystemCounter FILE_WRITE_OPS: 0 FILE_READ_OPS: 0 FILE_LARGE_READ_OPS: 0 FILE_BYTES_READ: 0 HDFS_BYTES_READ: 1437 FILE_BYTES_WRITTEN: 931780 HDFS_LARGE_READ_OPS: 0 HDFS_WRITE_OPS: 20 HDFS_READ_OPS: 40 HDFS_BYTES_WRITTEN: 431907904 org.apache.sqoop.submission.counter.SqoopCounters ROWS_READ: 644083 Job executed successfully
接下来的问题：如何导入数据到Hbase中？
评论暂时关闭