Create Value class for Sequence Files at runtime

I have some types of data that I have to upload on HDFS as Sequence Files.

Initially, I had thought of creating a .jr file at runtime depending on the type of schema and use rcc DDL tool by Hadoop to create these classes and use them.

But looking at rcc documentation, I see that it has been deprecated. I was trying to see开发者_Python百科 what other options I have to create these value classes per type of data.

This is a problem as I get to know the metadata of the data to be loaded at runtime along with the data-stream. So, I have, no choice, but to create Value class at runtime and then use it for writing (key, vale) to SequenceFile.Writer and finally saving it on HDFS.

Is there any solution for this problem?

You can try looking other serialization frameworks, like Protocol Buffers, Thrift, or Avro. You might want to look at Avro first, since it doesn't require static code generation, which might be more suitable for you.

Or if you want something really quick and dirty, each record in the SequenceFile can be a HashMap where the key/values are the name of the field and the value.