How to read a managed stream into a uint array directly in C#?

I have some data in a stream which is actually in the format of uint. System.IO.Stream only allows me to read the content out to a byte array, but I don't want to read the bytes and then convert them into a uint array with 1/4 the length of the byte array. I want to read it directly into the memory of the uint array so that it saves time.

If it's in unmanaged C++ it's just so easy. Just send over the pointer to the uint array and it'd be done. But in C# it doesn't seem so straightforward. I'm trying to produce a byte array with the same memory address as a开发者_运维技巧nother uint array and then send the byte array as the parameter of Stream.Read() so that when Stream.Read() writes to the byte array the content is just in the uint array as well. But I've looked at the Marshall class and have so far found no solution to this problem. Any ideas?

List<UInt32> array = new List<UInt32>();
System.IO.BinaryReader reader = new System.IO.BinaryReader(stream);
while(stream.Position < stream.Length)
   array.Add(reader.ReadUInt32());

Note that some stream implementations don't support Position and/or Length properties.