Should changing the documented performance of a method be considered a breaking change? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.

---

Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.

Closed 5 years ago.

Improve this question

It seems a common rule in software development is that once you have deployed a public API, especially if you've documented that API (which—yeah, you should have), you must think long and hard before making any changes that could break backwards compatibility with that API as such changes would be breaking changes. In fact I'm sure many developers would argue that you just flat-out should not do it at all, regardless of how long and hard you may think about it.

Occasionally a developer will actually document the performance of a method. A lot of MSDN docs do this, for example, using Big O notation. Would changing this also constitute a "breaking" change? Maybe that's not a clear question; put another way: should it be avoided?

It seems to me that in cases where, e.g., you might have developed a superior algorithm to solve a problem, if you had previously documented that this algorithm was O(N²) you might be forgiven for improving your API by replacing it with this superior algorithm which is, say, O(log N). Users of your API might notice this and would, I imagine, only feel glad for improvement.

On the other hand I wonder whether it would ever be considered "excusable" to worsen the performance of a component of your API for other reasons—e.g., you optimize for the most common use case which results in worse performance in the general case; you optimize for memory over CPU cycles or vice versa; you eliminate an external dependency that had caused other issues, etc.

My intuition tells me that improving performance is almost certainly always OK. What about the opposite? Is it OK, assuming you update your documentation? Is it just wrong? I'm just looking for some rea开发者_开发知识库sonable guidelines, here.

---

Well, if you make the performance better I doubt you'll get any complaints.

Otherwise, I'd say it depends. If you have access to the source code for all your clients, the upright thing to do would be to look through them all and verify that your change won't cause them any trouble.

If it is more complicated than that, and you are liable to make some use-case's performance significantly worse, then you should go talk to your clients and warn them of the change.

If you can't do that either (eg: a shrink-wrapped API of some kind), then yeah, it would probably be best to implement the change as an alternative call or something, and leave the old call alone.

---

A Breaking Change is one that breaks ... specifications.

So, if you are dealing with a piece of software where timing (average case, worst case, variance, or whatever) or asymptotic behavior is specified, you should just avoid breaking that. I believe this kind of specifications should be part of any project, and surely they will be there if you have real time dependencies.

You are free to improve performance, as long as functionality is respected, and I guess nobody will complain.

As to worsen performance, (again within specs and for reasonable causes) depends on the balanced impact of choices. It's a trade-off, and as such should be analyzed in a case by case basis. How could be a general rule for that if your processing time impact may vary from an unnoticed to an infuriated user?

---

I note that the C++ standard requires that some functions of the STL have performance no worse than a given Big-O.

I speak from experience that if you are performing long duration batch-processes that have large data sets (2-3 hours, N~10⁷), an inadvertent O(N²) can be unacceptable. If an important relied upon API changed its performance from O(N) to O(N²), such a program could have its computation time so greatly extended (a 2 hours computation time could become a 4000 year computation time) that the program would be considered broken. So, I'd say that worsening Big-O performance was a breaking change.

---

Improving performance of an implementation is formally OK if you have documented it as big-O. If your new implementation is O(log n), then it is also O(n^2), as documented. The other way around, relaxing the limit is a breaking change from the POV of the user (since the previous property is no longer true).

So to permit better algorithms in future, don't document performance as big-Theta. In any case, it seems silly to me to guarantee that any operation will take at least a certain amount of time, unless you're publishing the cycle counts for CPU opcodes, interrupt frequencies from a device, or similar hardware-level stuff where exact timing is important. Just don't specify a minimum, and if a user comes to you saying, "we really need a guaranteed lower-bound time complexity on this", then write an alternative interface to the functionality:

int myfunc_omega_n_squared(n, args) {
    sleep(n*n); // well, actually in a loop because of spurious wakes
    return myfunc(n, args);
}

They can call this instead of myfunc, and their peculiar problem is solved ;-)

Do beware though that if you are defining an interface which others may implement (perhaps as well as your own implementation), then changing the interface to a stricter complexity requirement is of course a breaking change from the POV of the implementer, since their formerly-valid implementation no longer is.

That tells you whether the change is breaking in terms of what's documented. However, it's almost inconceivable that anyone would literally rely only on your implementation's asymptotic complexity, as opposed to its actual runtime. Absurd example, but if you change from an O(n^2) that takes an hour to process 100GB of input data, to an O(log n) that takes a week to process 100GB of data, you're probably going to get complaints from any users processing 100GB of data, regardless of the fact that you haven't actually broken your explicit interface.

What API users want is for their code to still work, so in general if you're going to make something much slower, regardless of whether it's big-O-slower or not, then there had better be a good reason. I'd want to issue a new interface rather than withdrawing and replacing the old one, if possible.