Read from a Pipe

I need to 开发者_如何学Pythonread the total result from a C PIPE. I could just create a massive buffer and slowly fill it, but that seems inefficient and very memory hungry, so I'm wondering if anyone has any ideas how to do it.

Here is my code:

FILE* pipe = popen(command, "r");
if (!pipe)
{
    strcpy("FAIL", output);
    return -1;
}

char buffer[128];

while (!feof(pipe))
{
    if (fgets(buffer, 128, pipe) != NULL)
    {
        // should add buffer to main one here
    }
}

Any help would be greatly appreciated.

In C++ I could just add it to a String, but I'm not sure how I'd do that in C.

Edit : I don't know the size of the final buffer.

I could just create a massive buffer and slowly fill it, but that seems inefficient and very memory hungry

If you need everything to be in the memory - that's the only way to go. You can cache interim results in a temporary file, if it takes long time to gather all the data together, but eventually you'll have to have that huge memory buffer filled one way or another. The interim caching can be either at the sender side or the receiver, up to you.

I'm not sure I can follow your dilemma...

edit

After some clarification it appears that the problem is that the amount of the needed memory is not known ahead of time.

For that realloc is indeed one of the ways to handle, using a file as the interim caching method will take care of that as well without the additional memory performance hit that realloc causes (but instead you get a time performance hit because disk I/O is much slower).

You could start with a fairly low buffer size and then double it whenever you run out of space. This should keep buffer size within a reasonable bound but won't take a huge number of reallocations. (I gathered from comments that you don't know the necessary size ahead of time.) I would say that realloc is a fairly safe bet and with this method you'll only call it a few times.