How to measure sequential and parallel runtimes of Haskell program

I am taking measurement of the haskell program from this question to produce the following table with runtimes and speedups summary so I can plot in a graph.

#Cores     Runtimes       Speedups
                     Absolute  Relative
Seq        ?         ..        ..
1          3.712     ..        ..
2          1.646     ..        ..

First question

While the runtimes on 1 and 2 cores are taken by compiling the program with the -threaded flag on ([3] and [4] below), I am not sure which time to take for the sequential one ([1] or [2] below):

• should it be the time obtained by compiling without the -threaded flag, or
• that obtained with the flag on but then NOT specifying any number of cores i.e. with no -Nx

Compiling without -threaded flag

        $ ghc --make -O2 test.hs
    [1] $ time ./test           ## number of core = 1
        102334155

        real    0m4.194s
        user    0m0.015s
        sys     0m0.046s

Compiling with -threaded flag

        $ ghc --make -O2 test.hs -threaded -rtsopts
    [2] $ time ./test           ## number of core = not sure?
        102334155

        real    0m3.547s
        user    0m0.000s
        sys     0m0.078s

    [3] $ time ./test +RTS -N1  ## number of core = 1
        102334155

        real    0m3.712s
        user    0m0.016s
        sys     0m0.046s

    [4] $ time ./test +RTS -N2  ## number of core = 2
        102334155

        real    0m1.646s
        user    0m0.016s
        sys     0m0.046s

Second question

As can be seen from above, I am using the time command to measure the runtimes. I am taking the 'real' time. But if I run the program with the -sstderr flag on, I get more detailed information:

    $ ghc --make -O2 test.hs -rtsopts
    $ ./test +RTS -sstderr 
    102334155
             862,804 bytes allocated in the heap
               2,432 bytes copied during GC
              26,204 bytes maximum residency (1 sample(s))
              19,716 bytes maximum slop
                   1 MB total memory in use (0 MB lost due to fragmentation)

      Generation 0:     1 collections,     0 parallel,  0.00s,  0.00s elapsed
      Generation 1:     1 collections,     0 parallel,  0.00s,  0.00s elapsed

      INIT  time    0.00s  (  0.00s elapsed)
      MUT   time    3.57s  (  3.62s elapsed)
      GC    time    0.00s  (  0.00s elapsed)
      EXIT  time    0.00s  (  0.00s elapsed)
      Total time    3.57s  (  3.62s elapsed)

      %GC time       0.0%  (0.0% elapsed)

      Alloc rate    241,517 bytes per MUT second

      Productivity 100.0% of total user, 98.6% of total elapsed

开发者_C百科I believe that the -sstderr provides a more accurate time which I should use instead of the time command. Am I correct? Also, which of the 'Total time' (3.57s or 3.62s) should I use?

And finally, any general advice/good practice while taking measurements like this? I am aware that there are some packages that allow us to benchmark our program, but I am mainly interested in taking the measurements manually (or using a script to do that for me).

Also: the runtimes are the median of running the program 3 times.

I would use -N1 for the single-core time. I believe that also constrains the GC to use one core (which seems fitting for the benchmark, I think?), but others may know more.

As for your second question, the answer to benchmarking in Haskell is nearly always to use criterion. Criterion will allow you to time one run of the program, and you can then wrap it in a script which runs the program with -N1, -N2, etc. Taking the median of 3 runs is okay as a very quick and rough indicator, but if you want to rely on the results then you'll need a lot more runs than that. Criterion runs your code enough and performs the appropriate statistics to give you a sensible average time, as well as confidence intervals and standard deviation (and it tries to correct for how busy your machine is). I know you asked about best practice for doing it yourself, but Criterion already embodies a lot of it: use clock time, benchmark a lot, and as you realised, don't just take a simple mean of the results.

Criterion requires very little change to your program if you want to benchmark the whole thing. Add this:

import Criterion.Main

main :: IO ()
main = defaultMain [bench "My program" oldMain]

where oldMain is whatever your main function used to be.