Mathematica and C/C++: Exchanging Data

I would like to know how to exchange data between Mathematica and a C/C++ with pipes. In the Mathematica tutorial it says that "when you open a file or a pipe, Mathematica creates a 'stream object' that specifies the open stream associated with the file or pipe".

I know how to create files in both C and Mathematica and I can make each program read and write to them. What I still don't know how to do is how to send output from C through a pipe to another program, much less how to even do this from Mathematica.

Here is a function in which Mathematica that writes a matrix to a binary file as well as reading a file written in that format.

writeDoubleMatrix[obj_, fileName_] := Module[{file},
  file = OpenWrite[fileName, BinaryFormat -> True];
  BinaryWrite[file, Length@obj, "Integer32"];
  BinaryWrite[file, Length@obj[[1]], "Integer32"];
  BinaryWrite[file, Flatten[obj], "Real64"];
  Close[file]
 ]
readDoubleMatrix[fileName_] := Module[{file, obj, m, n},
  file = OpenRead[fileName, BinaryFormat -> True];
  m = BinaryRead[file, "Integer32"];
  n = BinaryRead[file, "Integer32"];
  obj = BinaryReadList[file, "Real64", m*n];
  Close[file];
  Partition[obj, n]
 ]

The first function will write 2 integers to a file (the size of the matrix) and the data of the matrix. I'm not doing any error checking here and thus I'm assuming that the data to be written is specifically in the form {{r11, r12, ..., r1n}, ...., {rm1, rm2, ..., rmn}}. The second function will be able to read the binary file and return the matrix.

Next comes my C program. This program will read the data stored in the file MathematicaData.bin, multiply this matrix by 2 and write data to another file.

// genData.c
#include <stdlib.h>
#include <stdio.h>

int main(int argc, char** argv){
    int m, n, i;
    double* matrix;
    FILE* fin;
    FILE* fout;

    // Reading input file
    fin = fopen(argv[1], "rb");
    fread(&m, sizeof(int), 1, fin);
    fread(&n, sizeof(int), 1, fin);
    matrix = (double*)malloc(m*n*sizeof(double));
    fread(matrix, sizeof(double), m*n, fin);
    fclose(fin);

    //Modifying data
    for (i = 0; i < m*n; ++i) matrix[i] = 2*matrix[i];

    // Writing output file
    fout = fopen(argv[2], "wb");
    fwrite(&m, sizeof(int), 1, fout);
    fwrite(&n, sizeof(int), 1, fout);
    fwrite(matrix, sizeof(double), m*n, fout);
    fclose(fout);

    // De-allocate memory used for matrix.
    free(matrix);
    return 0;
 }

This program does not have any error checking. You need to be careful how you use it otherwise the program may not detect the files or even allocate the amount of memory that you want. In any case, we can compile the program with your compiler of your choice.

gcc -o genData genData.c

And now we can try to use these functions to communicate between the two languages from Mathematica.

matrix = {{1, 2, 3, 4}, {5, 6, 7, 8}, {9, 10, 11, 12}};
writeDoubleMatrix[matrix, "MathematicaData.bin"];
Run["./genData MathematicaData.bin CData.bin"];
readDoubleMatrix["CData.bin"]

If everything went well the output you should get is

{{2., 4., 6., 8.}, {10., 12., 14., 16.}, {18., 20., 22., 24.}}

Yes, this is a very time consuming way of multiplying a matrix by 2 but this is just a simple example to show how to exchange data from Mathematica to C and from C to Mathematica. What I do not like is the fact that everything is stored to a file first and then it is read in the other program. Can someone show me how to exchange data without writing files. I have a feeling that pip开发者_如何学编程es is what I need but I have no idea how to read or write them from neither language. It would be helpful if you could modify this program to adapt it to pipes.

UPDATE:

I found out how to make the C program "pipeable".

//genDataPipe.c
#include <stdlib.h>
#include <stdio.h>

int main(int argc, char** argv){
    int m, n, i;
    double* matrix;

    // Reading input file
    fread(&m, sizeof(int), 1, stdin);
    fread(&n, sizeof(int), 1, stdin);
    matrix = (double*)malloc(m*n*sizeof(double));
    fread(matrix, sizeof(double), m*n, stdin);

    //Modifying data
    for (i = 0; i < m*n; ++i) matrix[i] = 2*matrix[i];

    // Writing output file
    fwrite(&m, sizeof(int), 1, stdout);
    fwrite(&n, sizeof(int), 1, stdout);
    fwrite(matrix, sizeof(double), m*n, stdout);

    // Deallocate memory used for matrix.
    free(matrix);
    return 0;
}

This means that we have to use the program like this:

./genDataPipe < fileIn.bin > fileOut.bin

I have been searching in the documentation on the Mathematica side but all I have figured out that I can open a file by piping an external command. Take OpenWrite for instance:

On computer systems that support pipes, OpenWrite["!command"] runs the external program specified by command, and opens a pipe to send input to it.

This means that I can give my binary input directly to the c program. The problem is that I can't find a way of redirecting the output of the program. What I have come up with is, write a data to a file and make a wrapper to run the external command and read the contents of the output of the external command. Here we assume the existence of writeDoubleMatrix from before.

getDataPipe[fileName_] := Module[{file, obj, m, n},
  file = OpenRead["!./genDataPipe < " <> fileName, 
  BinaryFormat -> True];
  m = BinaryRead[file, "Integer32"];
  n = BinaryRead[file, "Integer32"];
  obj = BinaryReadList[file, "Real64", m*n];
  Close[file];
  Partition[obj, n]
 ]
matrix = {{1, 2, 3}, {4, 5, 6}};
writeDoubleMatrix[matrix, "MData.bin"];
output = getDataPipe["MData.bin"]

Which results in output having the following contents {{2., 4., 6.}, {8., 10., 12.}}.

The only goal that remains now is to find out how to eliminate the need of writeDoubleMatrix and pass the data directly without having to write a file.

The first step would be to create a named pipe, (as mentioned by Artefacto's comment above). This is called a fifo, First In First Out. A pipe makes the output from one file into the input of the other. I'd like to note that these methods are Linux only before I start.

Basically, pipes work like this:

mkfifo mypipe or in C: system ("mkfifo mypipe");

Next step is to write the output to the pipe, because everything in Linux is a file, you can just use the standard i/o operations for that, or even redirect standard input to the pipe. You already have the code for that in both Mathematica and C. So after writing the output of the file to your pipe, the Mathematica version can then read input from the pipe, multiply it, and display it in stdout or wherever you please. You really shouldn't have a problem with this method, seeing as the pipe is emptied after the reading and can be removed easily after you're done. If you want to remove the pipe afterwards, just run system ("rm myfifo");.

If you really don't want any accessory file, even though it's not that bad, try making another file that actually outputs in standard output. Then make one that let's Matematica read from the standard input. Now, to pipe through:

./cprogram | ./mprogram

This means that the output of your C program should be the input of your Mathematica program. To my knowledge, this still creates a pipe, but it will automatically be deleted by the system when it's done, and the average end user probably won't see it.

The closest function that Mathematica has to reading and writing simultaneously from an external program is RunThrough. According to the documentation, though, it does not operate like you want: passing info through Write and Read operations. Instead, it writes the second parameter to a temporary file, executes the command specified in the first parameter passing the temporary file to it, and captures the output of the command. This is clearly not what you're asking for, but it seems to be the closest of the built-in commands. Personally, I'd look into using MathLink, but since you don't seem to want to do that, Johnathon's answer is on the right track.

To expand on his answer a bit, the documentation implies that Mathematica cannot read and write simultaneously to the same file/pipe. This suggests to me, a round about solution. First, you need 2 pipes: one for writing and one for reading, and these can be made via Run, e.g.

Run["mkfifo in"]
Run["mkfifo out"].

Second, open the pipes for i/o

instrm = OpenRead["in"]
outstrn = OpenWrite["out"].

Last, run your external program by either passing the names of the pipes to it, if it is set up to handle command line params,

Run["prog in out"],

or via redirection

Run["prog <in >out"].

MathLink may be the best choice for you. Mathematica and the C program will be separate processes and communicate using pipes or shared memory. You'll have to look at the MathLink examples in your Mathematica install. There's a tool to generate glue code from a template file called mprep, and you'll need to link in the MathLink library, but it's dead easy after that to call a C function with an array of data.

MLPutReal64Array - send array of data to Mathematica

MLGetReal64Array - read array from Mathematica

I'd take one of the examples and start working from there. You'll probably want a Manual ReturnType and Real64List for the ArgumentType.

It's a bit awkward, but you can use JLink:

Needs["JLink`"];
InstallJava[];
LoadJavaClass["java.lang.Runtime"];
runtime = java`lang`Runtime`getRuntime[];
process = runtime@exec["my program"];
input = process@getInputStream[];
output = process@getOutputStream[];

Now you can write to a pipe that's stdin on the other end:

output@write[ToCharacterCode["hi!"]];

And read from the launched program's stdout:

nextByte = input@read[];