How to compare results of two RSpec suite runs?

I have a pretty big spec suite (watirspec), I am running it against a Ruby gem (safariwatir) and there are a lot of failures:

1002 examples, 655 failures, 1 pending

When I make a change in the gem and run the suite again, sometimes a lot of previously failing specs pass (52 in this example):

1002 examples, 603 failures, 1 pending

I would like to know which previously failing specs are now passing, and of course if any of the previously passing specs are now failing. What I do now to compare the results is to run the tes开发者_高级运维ts with --format documentation option and output the results to a text file, and then diff the files:

rspec --format documentation --out output.txt

Is there a better way? Comparing text files is not the easiest way to see what changed.

Just save the results to file like you're doing right now and then just diff those results with some random diff-ing tool.

I don't know of anything out there that can do exactly that. Said that, if you need it so badly you don't mind spending some time hacking your own formatter, take a look at Spec::Runner::Formatter::BaseFormatter.It is pretty well documented.

I've implemented @Serabe's solution for you. See the gist: https://gist.github.com/1142145.

Put the file my_formatter.rb into your spec folder and run rspec --formatter MyFormatter. The formatter will compare current run result with previous run result and will output the difference as a table.

NOTE: The formatter creates/overwrites file result.txt in the current folder.

Example usage:

D:\Projects\ZPersonal\equatable>rspec spec --format MyFormatter
..........
No changes since last run

Finished in 0.011 seconds
10 examples, 0 failures

No changes since last run line was added by the formatter.

And now I intentionally broken one and rerun rspec:

D:\Projects\ZPersonal\equatable>rspec spec --format MyFormatter
..F.......
Affected tests (1).
PS CS Description
.  F  Equatable#== should be equal to the similar sock
PS - Previous Status
CS - Current Status

Failures:

  1) Equatable#== should be equal to the similar sock
     Failure/Error: subject.should == Sock.new(10, :black, 0)
       expected: #<Sock:0x2fbb930 @size=10, @color=:black, @price=0>
            got: #<Sock:0x2fbbae0 @size=10, @color=:black, @price=20> (using ==)
       Diff:
       @@ -1,2 +1,2 @@
       -#<Sock:0x2fbb930 @color=:black, @price=0, @size=10>
       +#<Sock:0x2fbbae0 @color=:black, @price=20, @size=10>
     # ./spec/equatable_spec.rb:30:in `block (3 levels) in <top (required)>'

Finished in 0.008 seconds
10 examples, 1 failure

Failed examples:

rspec ./spec/equatable_spec.rb:29 # Equatable#== should be equal to the similar sock

The table with affected specs was added by the formatter:

Affected tests (1).
PS CS Description
.  F  Equatable#== should be equal to the similar sock
PS - Previous Status
CS - Current Status

If some spec status is different between current and previous run, the formatter outputs previous status, current status and spec description. '.' stands for passed specs, 'F' for failed and 'P' for pending.

The code is far from perfect, so feel free to criticize and change it as you want.

Hope this helps. Let me know if you have any questions.