How to efficiently merge two large MySQL tables?
Here's my problem. I have two MySQL tables, one with 3.3 million records and one with 700,000 records. All 700,000 records that are in the second table exist in the first and the two tables have one column in common. I want to select * from both tables for all 700,000 records and insert them into a new table. I've created an index for both of the columns in common. Here's what I would normally do
INSERT INTO MergedTable (FirstName TEXT, LastName TEXT, Address TEXT) SELECT FirstNameFromTable1, LastName, Address FROM Table1, Table2 WHERE FirstNameFromTable1 = FirstNameFromTable2
But the size of the tables causes this statement to hang and eventually the request is interrupted. Any ideas on how to do this more efficiently? For reference, here are the commands I used to make the two tables in question. Thanks for your help.
CREATE TABLE Table1 ( FirstNameFromTable1 varchar(300), LastName TEXT, index(FirstNameFromTable1) );
CREATE TABLE Table2 ( FirstNameFromTable2 varchar(300), Address TEXT, index(FirstNameF开发者_运维问答romTable2) );
Chunk it:
insert into MergedTable (...)
select top 10000 ... from Table1,Table2
where FirstNameFromTable1=FirstNameFromTable2
and not exists (select * from MergedTable where FirstName=FirstNameFromTable1)
run this until row_count() is 0, you'll want an index on MergedTable.FirstName obviously
精彩评论