Testing strategy: Assert on 'features' of response or assert on full response? [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but thi开发者_运维问答s question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance. Closed 11 years ago.

I've had a "philosophy" disagreement with a coworker and I would like to hear community thoughts on the two sides presented (or, an even better third option).

Basically: We have a JSON API that returns a friends list. The result looks something like:

[{"name":"Bob", "uid":12345, "level":4}, {"name":"George", "uid":23456, "level":6}]

There are the normal sort of "mutual friend" requirements in place, that influence the response.

The disagreement is basically over which is better,

• Tests that assert on "features" of the response:

  def test_results_are_sorted_by_name():
      .. <setup 2 friends> ..
  
      response = controller.getFriendsList()
      assertLessThan(response[0].name, response[1].name)
  
  def test_blocked_users_are_not_returned():
      .. <setup some friends and block one, storing the id in blocked_uid> ..
  
      response = controller.getFriendsList()
      for friend in response:
          assertNotEqual(friend.uid, blocked_uid)
  

• Tests that assert on a literal response

  def test_results_are_sorted_by_name():
      .. <setup 2 friends> ..
  
      response = controller.getFriendsList()
      expectedResponse = {...}
      assertEqual(response, expectedResponse)
  
  def test_blocked_users_are_not_returned():
      .. <setup some friends and block one, storing the id in blocked_uid> ..
  
      response = controller.getFriendsList()
      expectedResponse = {...}
      assertEqual(response, expectedResponse)
  

Which is better, and why?

Are there other options which are better than both?

---

Pure opinion here, but:

Testing literals encourages bad behavior - literal tests are so fragile and break so often that people learn to respond to test failures by updating the literal. Even if they broke it, they may convince themselves it's correct, update the literal, and move on, unintentionally cementing a bug in with a test, ensuring that it will not be fixed.

Feature testing is the right way to go. There are ways to go wrong with it - you can write a feature test easily that looks right, but passes even when your code is broken - but the best solution to that is to also use integration tests with your feature tests, to ensure that the service actually does the thing you expect.

---

You need to test based on features, not literal responses.

In principle, this allows people to change the format (add fields, etc) in ways that don't break what you have already written tests for. When they do this, your tests won't break, which is what they should do. (If it's bad for new fields to be added, write a test for that.)

In practice, literal text tests sometimes get fixed by just pasting in the new string, and real problems get, well, pasted over without anyone giving them enough thought.

If you want to make sure the response isn't changed in some way that passes tests but then doesn't work with the recipients of the data, include some basic level of integration test.