JavaRDD<String> hbaseFile = jsc.textFile(HDFS_MASTER+HBASE_FILE);
JavaPairRDD<ImmutableBytesWritable, KeyValue> putJavaRDD = hbaseFile.mapToPair(line -> convertToKVCol1(line, COLUMN_AGE));
putJavaRDD.sortByKey(true);
putJavaRDD.saveAsNewAPIHadoopFile(stagingFolder, ImmutableBytesWritable.class, KeyValue.class, HFileOutputFormat2.class, conf);
private static Tuple2<ImmutableBytesWritable, KeyValue> convertToKVCol1(String beanString, byte[] column) {
InspurUserEntity inspurUserEntity = gson.fromJson(beanString, InspurUserEntity.class);
String rowKey = inspurUserEntity.getDepartment_level1()+"_"+inspurUserEntity.getDepartment_level2()+"_"+inspurUserEntity.getId();
return new Tuple2<>(new ImmutableBytesWritable(Bytes.toBytes(rowKey)),
new KeyValue(Bytes.toBytes(rowKey), COLUMN_FAMILY, column, Bytes.toBytes(inspurUserEntity.getAge())));
}
The above is my code, it only works for a single column for a row key. Any ideas to create an HFile with multiple columns for one row key?
You can create multiple Tuple2<ImmutableBytesWritable, KeyValue>
for one row, where the key stays the same and KeyValue
s represent individual cell values.
Make sure to order your columns lexicographically as well.
So you should invoke saveAsNewAPIHadoopFile
on a JavaPairRDD<ImmutableBytesWritable, KeyValue>
.
final JavaPairRDD<ImmutableBytesWritable, KeyValue> writables = myRdd.flatMapToPair(record -> {
final List<Tuple2<ImmutableBytesWritable, KeyValue>> listToReturn = new ArrayList<>();
// Add first column to the collection
listToReturn.add(new Tuple2<ImmutableBytesWritable, KeyValue>(
new ImmutableBytesWritable(Bytes.toBytes(record.getRowKey())),
new KeyValue(Bytes.toBytes(record.getRowKey()), Bytes.toBytes("CF"),
Bytes.toBytes("COL1"), System.currentTimeMillis(),
Bytes.toBytes(record.getCol1()))));
// Add subsequent columns
listToReturn.add(new Tuple2<ImmutableBytesWritable, KeyValue>(
new ImmutableBytesWritable(Bytes.toBytes(record.getRowKey())),
new KeyValue(Bytes.toBytes(record.getRowKey()), Bytes.toBytes("CF"),
Bytes.toBytes("COL2"), System.currentTimeMillis(),
Bytes.toBytes(record.getCol2()))));
});
NOTE: This is a major gotcha, you must add your columns to the RDD lexicographically as well.
Essentially this combination: row key + column family + column qualifier should be sorted before you proceeed to push out the HFiles.