Void pointer to access structs in char array

I have a buffer, essentially an char array, filled with multiple different structs. The structs needs to be passed in a buffer like this because it is something I read/write to a socket. The structs are one header struct and possibly "multiple payload" structs.

Just to illustrate, it looks like this:

unsigned char buffer[buflen];
struct header *head;
struct payload1 *p1;
struct payload2 *p2;

etc...

Now, when I try to fill this buffer, or retrieve from this buffer, I've used a void *ptr where it is first initialized at the position of the buffer then later at the position after the header, like this:

void *ptr;

ptr = &buffer;
ptr += sizeof(header);

This actually works fine - i.e. the pointer points to the correct memory location and I can retrieve (or insert) a new payload struct just fine, but it does however generate a warning in gcc. warning: pointer of type ‘void *’ used in arithmetic.

What can I do to avoid this warning in this case?

Use char * instead of a void *. The problem is it's not obvious what incrementing a void * is supposed to do. I'd expect this to be an error rather than a warning.