Reverse engineering NSDocument file

I have an unsupported file format from an OS X program. It uses NSDocument, unfortunately the vendor won't help with the format - or allow export.

So what are the first steps in reverse engineering this format based on NSDocument? The program still uses deprecated initWithContentsOfFile methods from os x 10.3. I have access to the program, my data and lots of sample data.

I've done some initial work on disassembly of the program, and revese engineering the format. But haven't made much progress. The disassembly ha开发者_运维百科sn't revealed a thing, but looking at the file with a hex editor has given me the outline of a few o the data structures and the initial fixed header.