How to seek to a specific time in a RTP stream?

I am streaming a prerecorded H264 video that has the following structure:

[I] [x] [x] [x] [I] [x] [x] [x] [I]...

In between the IDR (I-s in my structure) I have 32 (only 3 presented here) other frames (all other stuff that is not IDR like SEI, SPS, PPS... X-es)

Now, let assume that the timing of my frames is such:

TIME:   1   2   3   4   5   6   7   8   9
FRAME: [I] [x] [x] [x] [I] [x] [x] [x] [I]...

Now i want to seek to the time 4. If I seek to that frame, and send it, the picture gets messed up because the decoder needs a IDR to decode it properly, so I resorted to finding the appropriate IDR (in this case one with the time 1) and sending it as the frame with the time 4. So now the picture is decoded properly, all is well... but... If my GOV is 32, and I need to send the non IDR frame that has the index 31, and if the time span between it and the corresponding IDR is 3 seconds, I actually get 3 seconds earlier then the time I want. Now, this is not precise, because I cannot seek to the half of the GOV time span. Also, I cant set smaller GOV, so I want other ideas...

My other idea was to send the last known IDR, and then开发者_如何学Python send all other non IDR frames that come before the one I want, only I would set for all of them RTP-TIME to be the same as the corresponding IDR. In this case the picture gets decoded perfectly, but now in the above case, 3 seconds that follow non IDR frame with the wanted time get fast paced in the decoder/player (there is no instantaneous seek)...

Any ideas? Or I can only seek to IDR-s and not the frames in between?