大数据集群间数据由hbase到impala

2024-07-21 02:51:47

字体：大中小

来源：转载

供稿：网友

10.0.0.11创建测试表create 'LV_TEST',{NAME =>'cf' ,BLOOMFILTER => 'ROW',COMPRESSION => 'SNAPPY',VERSIONS => 1}插入测试数据put 'LV_TEST','rowkey001','cf:name','zhangsan'put 'LV_TEST','rowkey001','cf:age','30'get 'LV_TEST','rowkey001', {COLUMN=>'cf:name'}get 'LV_TEST','rowkey001'put 'LV_TEST','rowkey002','cf:name','lisi'put 'LV_TEST','rowkey002','cf:age','40'get 'LV_TEST','rowkey002'查看测试表scan 'LV_TEST'导出到HDFSfind ./ /opt -name hbasecd /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/binhbase org.apache.hadoop.hbase.mapreduce.Export LV_TEST /data/LV_TESThadoop fs -ls /data/LV_TESThadoop fs -get /data/LV_TEST然后通过FTP下载到本地上传到另一服务器192.168.51.251上传到HDFSsu - hdfshadoop fs -mkdir /datahadoop fs -put /LV_TEST /data/LV_TESThadoop fs -ls /data/LV_TEST创建Hbase表create 'LV_TEST_COPY',{NAME =>'cf' ,BLOOMFILTER => 'ROW',COMPRESSION => 'SNAPPY',VERSIONS => 1}scan 'LV_TEST_COPY'hbase org.apache.hadoop.hbase.mapreduce.Import LV_TEST_COPY /data/LV_TEST建立hive外表hiveCREATE EXTERNAL TABLE LV_TEST_COPY(id string,name string,age string)ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key, cf:name, cf:age")TBLPROPERTIES("hbase.table.name" = "LV_TEST_COPY");impalaimpala-shell -i node90同步元数据INVALIDATE METADATA;SQL查询结果

select * from LV_TEST_COPY;

最后将外部表数据复制到基于parques的impala内部表，完成数据转换。

insert into ***() select * from *****

上一篇：利用 ELK 处理 Percona 审计日志

下一篇：Redis的主从复制