What is the "proper" way to allocate variable-sized buffers in C++?

This is very similar to this question, but the answers don't really answer this, so I thought I'd ask again:

Sometimes I interact with functions that return variable-length structures; for example, FSCTL_GET_RETRIEVAL_POINTERS in Windows returns a variably-sized RETRIEVAL_POINTERS_BUFFER structure.

Using malloc/free is discouraged in C++, and so I was wondering:

What is t开发者_如何学运维he "proper" way to allocate variable-length buffers in standard C++ (i.e. no Boost, etc.)?

vector<char> is type-unsafe (and doesn't guarantee anything about alignment, if I understand correctly), new doesn't work with custom-sized allocations, and I can't think of a good substitute. Any ideas?

I would use std::vector<char> buffer(n). There's really no such thing as a variably sized structure in C++, so you have to fake it; throw type safety out the window.

If you like malloc()/free(), you can use

RETRIEVAL_POINTERS_BUFFER* ptr=new char [...appropriate size...];

... do stuff ...

delete[] ptr;

Quotation from the standard regarding alignment (expr.new/10):

For arrays of char and unsigned char, the difference between the result of the new-expression and the address returned by the allocation function shall be an integral multiple of the strictest fundamental alignment requirement (3.11) of any object type whose size is no greater than the size of the array being created. [ Note: Because allocation functions are assumed to return pointers to storage that is appropriately aligned for objects of any type with fundamental alignment, this constraint on array allocation overhead permits the common idiom of allocating character arrays into which objects of other types will later be placed. — end note ]

I don't see any reason why you can't use std::vector<char>:

{
   std::vector<char> raii(memory_size); 
   char* memory = &raii[0];

  //Now use `memory` wherever you want
  //Maybe, you want to use placement new as:

   A *pA = new (memory) A(/*...*/); //assume memory_size >= sizeof(A);
   pA->fun();
   pA->~A(); //call the destructor, once done!

}//<--- just remember, memory is deallocated here, automatically!

Alright, I understand your alignment problem. It's not that complicated. You can do this:

A *pA = new (&memory[i]) A();
//choose `i` such that `&memory[i]` is multiple of four, or whatever alignment requires
//read the comments..

You may consider using a memory pool and, in the specific case of the RETRIEVAL_POINTERS_BUFFER structure, allocate pool memory amounts in accordance with its definition:

sizeof(DWORD) + sizeof(LARGE_INTEGER)

plus

ExtentCount * sizeof(Extents)

(I am sure you are more familiar with this data structure than I am -- the above is mostly for future readers of your question).

A memory pool boils down to "allocate a bunch of memory, then allocate that memory in small pieces using your own fast allocator". You can build your own memory pool, but it may be worth looking at Boosts memory pool, which is a pure header (no DLLs!) library. Please note that I have not used the Boost memory pool library, but you did ask about Boost so I thought I'd mention it.

std::vector<char> is just fine. Typically you can call your low-level c-function with a zero-size argument, so you know how much is needed. Then you solve your alignment problem: just allocate more than you need, and offset the start pointer:

Say you want the buffer aligned to 4 bytes, allocate needed size + 4 and add 4 - ((&my_vect[0] - reinterpret_cast<char*>(0)) & 0x3).

Then call your c-function with the requested size and the offsetted pointer.

Ok, lets start from the beginning. Ideal way to return variable-length buffer would be:

MyStruct my_func(int a) { MyStruct s; /* magic here */ return s; }

Unfortunately, this does not work since sizeof(MyStruct) is calculated on compile-time. Anything variable-length just do not fit inside a buffer whose size is calculated on compile-time. The thing to notice that this happens with every variable or type supported by c++, since they all support sizeof. C++ has just one thing that can handle runtime sizes of buffers:

MyStruct *ptr = new MyStruct[count];

So anything that is going to solve this problem is necessarily going to use the array version of new. This includes std::vector and other solutions proposed earlier. Notice that tricks like the placement new to a char array has exactly the same problem with sizeof. Variable-length buffers just needs heap and arrays. There is no way around that restriction, if you want to stay within c++. Further it requires more than one object! This is important. You cannot make variable-length object with c++. It's just impossible.

The nearest one to variable-length object that the c++ provides is "jumping from type to type". Each and every object does not need to be of same type, and you can on runtime manipulate objects of different types. But each part and each complete object still supports sizeof and their sizes are determined on compile-time. Only thing left for programmer is to choose which type you use.

So what's our solution to the problem? How do you create variable-length objects? std::string provides the answer. It needs to have more than one character inside and use the array alternative for heap allocation. But this is all handled by the stdlib and programmer do not need to care. Then you'll have a class that manipulates those std::strings. std::string can do it because it's actually 2 separate memory areas. The sizeof(std::string) does return a memory block whose size can be calculated on compile-time. But the actual variable-length data is in separate memory block allocated by the array version of new.

The array version of new has some restrictions on it's own. sizeof(a[0])==sizeof(a[1]) etc. First allocating an array, and then doing placement new for several objects of different types will go around this limitation.