Python/MySQL batch UPDATE optimisation
I'm currently trying to determine the best means of performing a bulk UPDATE into a MySQL DB, in terms of speed. Unfortunately, each row needs a separate update and I'm basing it off the ID (auto-incremented) of the table. Hence the need for one UPDATE statement per row. The table itself is around 12m records and I probably only need to update around 100,000 per day.
For batch INSERT statements, I am able to use the executemany() function, but unfortunately the same function does not seem to affect UPDATE statements. I have read that this is due to the fact that the optimisations in MySQL which allow this function to work are hardcoded for INSERTS.
Currently I am considering creating a temporary table which will store a key-value pair of ID and update_value which will then be updated into the table requiring an update via an update join. Is this a good idea, conside开发者_Go百科ring the database is a master for a chained replication topology?
I am currently using MyISAM as the storage engine for this table as it is read frequently (but not at the same time as the update). Admittedly I have not benchmarked it using InnoDB. I realise that MyISAM has table level locking and that an UPDATE will lock the entire table for the duration of that particular update. If I switch to InnoDB is MySQL smart enough to perform the updates concurrently if provided via executemany() and am I likely to see any speed increase?
精彩评论