Generic Parsing of PB in java

Is it possible to parse protobuf in a generic fashion in Java?

I have looked into GeneratedMessage and could not find a way to parse any PB byte buffer into a GeneratedMessage.

Essentially, I a开发者_如何学Pythonm trying to parse a PB byte buffer into GeneratedMessage and then I would use reflection to detect fields inside it.

First of all, you can't parse PB data without knowing the schema. The schema originally comes from a ".proto" file and is typically embedded in the code generated by protoc. However, you can also tell protoc to store the schema in a format that's usable by the Java Protobuf library:

protoc --descriptor_set_out=mymessages.desc mymessages.proto

Then load it in your Java code:

FileInputStream fin = new FileInputStream("mymessages.desc");
Descriptors.FileDescriptorSet set =
  Descriptors.FileDescriptorSet.parseFrom(fin);
Descriptors.Descriptor md = set.getFile(0).getMessageType(0);

Once you have the schema for a message (Descriptor.Descriptor) parsing a message is easy:

byte[] data = ...;
DynamicMessage m = DynamicMessage.parseFrom(md, data);

DynamicMessage has a reflective API that lets you look through the fields.

The messy part is calling out to the protoc tool to convert the ".proto" file into a usable format. The C++ Protobuf library has a way to load ".proto" files directly, but unfortunately the Java Protobuf library does not.

You can use UnknownFieldSet to parse generic protobuf messages.

Then you can get individual fields using provided methods (e.g. asMap(), hasField(), getField())

E.g. (data taken from this question):

    byte[] msg = new byte[] { 0x0a, 0x10, 0x08, 0x7f, (byte)0x8a, 0x01, 0x04, 0x08, 0x02, 0x10, 0x03, (byte)0x92, 0x01, 0x04, 0x08, 0x02, 0x10, 0x03, 0x18, 0x01};
    UnknownFieldSet eee = UnknownFieldSet.parseFrom(msg);
    System.out.println(eee.toString());

Giving:

1: {
  1: 127
  17: {
    1: 2
    2: 3
  }
  18: {
    1: 2
    2: 3
  }
}
3: 1

This is right example:

private static DynamicMessage parseData(byte[] data) throws IOException, DescriptorValidationException {
    FileInputStream fin = new FileInputStream("test.desc");
    DescriptorProtos.FileDescriptorSet set = DescriptorProtos.FileDescriptorSet.parseFrom(fin);
    Descriptor md = Descriptors.FileDescriptor.buildFrom(set.getFile(0), new  Descriptors.FileDescriptor[] {}).findMessageTypeByName("Person");
    return DynamicMessage.parseFrom(md, data);
}

I have working solution tested with last protobuf v.3.1.0

This is upgraded solution raised on previous answers. Thanks to both authors.

import com.example.address.AddressBookManager;
import com.example.address.AddressBookProtos.AddressBook;
import com.google.protobuf.DescriptorProtos.FileDescriptorSet;
import com.google.protobuf.Descriptors;
import com.google.protobuf.Descriptors.Descriptor;
import com.google.protobuf.Descriptors.FileDescriptor;
import com.google.protobuf.DynamicMessage;

import java.io.File;
import java.io.InputStream;

public class DynamicMessageDemo {

    private static final String ADDRESS_BOOK_SOURCE_FILENAME = "test.ab";
    private static final String ADDRESS_BOOK_DESC_FILENAME =
        File.separator + "address.desc";

    public static void main(String[] args) throws Exception {

        InputStream is = DynamicMessageDemo.class
                .getResourceAsStream(ADDRESS_BOOK_DESC_FILENAME);

        FileDescriptorSet set = FileDescriptorSet.parseFrom(is);
        FileDescriptor fd = Descriptors.FileDescriptor.buildFrom(
                set.getFile(0), 
                new Descriptors.FileDescriptor[]{}
                );

        // "AddressBook" is the second message in my *.proto
        // so index must be '1'
        Descriptor messageType = fd.getMessageTypes().get(1);

        // for testing purpose 
        AddressBook book = AddressBookManager.readFromFile(ADDRESS_BOOK_SOURCE_FILENAME);
        byte[] data = book.toByteArray();

        DynamicMessage message = DynamicMessage.parseFrom(messageType, data);

        System.out.println("\n Dynamic message:\n" + message);
    }
}

Here is another way to generically parse a .desc file:

FileInputStream fin = new FileInputStream(descPath);
DescriptorProtos.FileDescriptorSet set = DescriptorProtos.FileDescriptorSet.parseFrom(fin);

List<FileDescriptor> dependencyFileDescriptorList = new ArrayList<>();
for(int i=0; i<set.getFileCount()-1;i++) {
    dependencyFileDescriptorList.add(Descriptors.FileDescriptor.buildFrom(set.getFile(i), new  Descriptors.FileDescriptor[] {}));
}
FileDescriptor desc = Descriptors.FileDescriptor.buildFrom(set.getFile(set.getFileCount()-1), dependencyFileDescriptorList.toArray(new FileDescriptor[0]));