MSBuild ITaskItem RecursiveDir metadata disappears

I have a custom MSBuild task, which processes a set of files and returns a modified subset of this. Basically, I just create a new ITaskItem array out of the input, skipping some items.