CUDA: cudaEventElapsedTime returns device not ready error

I tried to measure the elapsed time on Tesla (T10 processors) and cudaEventElapsedTime returns device not ready error. But when I tested it on Fermi (Tesla M2090), it gave me the result.

Can anyone tell me what is happening...

Here is my code

cudaError_t err;
        cudaEvent_t start, stop;
        cudaEventCreate(&start);
        cudaEventCreate(&stop);
        err = cudaEventRecord(start, 0);
        f(err != cudaSuccess) {
          printf ("\n\n 1. Error: %s\n开发者_开发技巧\n", cudaGetErrorString(err));
          exit(1);
        }
        // actual code
       cudaThreadSynchronize();
        err = cudaEventRecord(stop, 0);
        if(err != cudaSuccess) {
          printf ("\n\n2. Error: %s\n\n", cudaGetErrorString(err));
          exit(1);
        }
        err = cudaEventElapsedTime(&elapsed_time, start, stop);
        f(err != cudaSuccess) {
          printf ("\n\n 3. Error: %s\n\n", cudaGetErrorString(err));
          exit(1);
        }

It's because cudaEventRecord is asynchronous. It finishes its execution immediately, regardless of the status. Asynchronous functions simply put an order on a "CUDA execution queue". When GPU finishes its current assignment, it pops the next order and executes it. It is all done in a separate thread, handled by the CUDA driver, separate of your program host thread.

cudaEventRecord is an order which says more-or-less something like this: "When you are done all previous work, flag me in this variable".

If your host thread then asks for cudaEventElapsedTime, but the GPU didn't finish its work yet, it gets confused and reports "not ready yet!". cudaEventSynchronize() stalls the current host thread until the GPU reaches the cudaEventRecord order that you placed earlier. After that you are guaranteed that cudaEventElapsedTime will have a meaningful answer for you.

cudaThreadSynchronize() is just a stronger tool: it stalls current thread until GPU finishes all assigned tasks, and not just those until the event.

Even I faced this issue and so based on the answer by @CygnusX1 I keep all my execution code in one cell and the cudaEventElapsedTime in another cell. This solved the issue because Colab (or jupyter notebook) goes to the next cell only if the process in the current cell is done.

Thus,

with torch.no_grad():
  model.eval() # warm up
  model(x)
  start.record() 
  model(x)
  model(x)
  model(x)
  end.record()
  print('execution time in MILLISECONDS: {}'.format(start.elapsed_time(end)/3.0))

raised the error reported in the question i.e. device not ready error and was solved by

with torch.no_grad():
  model.eval()
  model(x) # warm up
  start.record() 
  model(x)
  model(x)
  model(x)
  end.record()
# Shift the print command to next code CELL !!!

 print('execution time in MILLISECONDS: {}'.format(start.elapsed_time(end)/3.0))