Mysql Update performance suddenly abysmal

MySQL 5.1, Ubuntu 10.10 64bit, Linode virtual machine.

All tables are InnoDB.

One of our production machines uses a MySQL database containing 31 related tables. In one table, there is a field containing display values that may change several times per day, depending on conditions.

These changes to the display values are applied lazily throughout the day during usage hours. A script periodically runs and checks a few inexpensive conditions that may cause a change, and updates the display value if a condition is met. However, this lazy开发者_运维百科 method doesn't catch all posible scenarios in which the display value should be updated, in order to keep background process load to a minimum during working hours.

Once per night, a script purges all display values stored in the table and recalculates them all, thereby catching all possible changes. This is a much more expensive operation.

This has all been running consistently for about 6 months. Suddenly, 3 days ago, the run time of the nightly script went from an average of 40 seconds to 11 minutes.

The overall proportions on the stored data have not changed in a significant way.

I have investigated as best I can, and the part of the script that is suddenly running slower is the last update statement that writes the new display values. It is executed once per row, given the (INT(11)) id of the row and the new display value (also an INT).

update `table` set `display_value` = ? where `id` = ?

The funny thing is, that the purge of all the previous values is executed as:

update `table` set `display_value` = null

And this statement still runs at the same speed as always.

The display_value field is not indexed. id is the primary key. There are 4 other foreign keys in table that are not modified at any point during execution.

And the final curve ball: If I dump this schema to a test VM, and execute the same script it runs in 40 seconds not 11 minutes. I have not attempted to rebuild the schema on the production machine, as that's simply not a long term solution and I want to understand what's happening here.

Is something off with my indexes? Do they get cruft in them after thousands of updates on the same rows?

Update

I was able to completely resolve this problem by running optimize on the schema. Since InnoDB doesn't support optimize, this forced a rebuild, and resolved the issue. Perhaps I had a corrupted index?

mysqlcheck -A -o -u <user> -p

There is a chance the the UPDATE statement won't use an index on id, however, it's very improbable (if possible at all) for a query like yours.

Is there a chance your table are locked by a long-running concurrent query / DML? Which engine does the table use?

Also, updating the table record-by-record is not efficient. You can load your values into a temporary table in a bulk manner and update the main table with a single command:

CREATE TEMPORARY TABLE tmp_display_values (id INT NOT NULL PRIMARY KEY, new_display_value INT);

INSERT
INTO    tmp_display_values
VALUES
(?, ?),
(?, ?),
…;

UPDATE  `table` dv
JOIN    tmp_display_values t
ON      dv.id = t.id
SET     dv.new_display_value = t.new_display_value;