How to attach metadata to each frame of a video

I would like to be able to store metadata on each frame of a video file (a key to value map with ~10 entries per frame).

Which video format allows me to store such data ?

Where can I find a c++ library to do this ?

Thank you very much for your answers.

Stupid idea: Could you write your meta data in a separate text file? You could write for every timestamp one line and add your key-value pairs

FFmpeg does such thing.

ffmpeg -i in.avi -metadata title="my title" out.flv

So if you want a library to do this, you can take a look at libavcodec (the leading audio/video codec library).