CUDA: CPU code in parallel to GPU code

I have a program where I do a bunch of calculations on GPU, then I do memory operations with those results开发者_运维技巧 on CPU, then I take the next batch if data and do the same all over. Now it would be a lot faster if I could do the first set of calculations and then start with the second batch whilst my CPU churned away at the memory operations. How would I do that?

All CUDA kernel calls (e.g. function<<<blocks, threads>>>()) are asynchronous -- they return control immediately to the calling host thread. Therefore you can always perform CPU work in parallel with GPU work just by putting the CPU work after the kernel call.

If you also need to transfer data from GPU to CPU at the same time, you will need a GPU that has the deviceOverlap field set to true (check using cudaGetDeviceProperties()), and you need to use cudaMemcpyAsync() from a separate CUDA stream.

There are examples to demonstrate this functionality in the NVIDIA CUDA SDK -- For example the "simpleStreams" and "asyncAPI" examples.

The basic idea can be something like this:

• Do 1st batch of calculations on GPU

• Enter a loop: {

  Copy results from device mem to host mem

  Do next batch of calculations in GPU (the kernel launch is assynchronous and the control returns immediately to the CPU)

  Process results of the previous iteration on CPU

  }

• Copy results from last iteration from device mem to host mem

• Process results of last iteration

You can get finer control over asynchronous work between CPU and GPU by using cudaMemcpyAsync, cudaStream and cudaEvent.

As @harrism said you need your device to support deviceOverlap to do memory transfers and execute kernels at the same time but even if it does not have that option you can at least execute a kernel asynchronously with other computations on the CPU.

edit: deviceOverlap has been deprecated, one should use asyncEngineCount property.