PHP, MySQL CSV Import - How would you do this?

So I'm working on a PHP project that needs to allow the user to import CSV files into MySQL. The CSV files contain a "column" with a unique ID... there are duplicates. Here is where the tricky part comes in... The duplicates need to go into a separate table and not into the main table.

I have written code to do this but there has to be a more efficient way... Right now, the script reads the CSV and for each row, queries the db to see if the ID already exists, then puts it in the proper table. These CSV files contain 50k+ rows each so this gets to be extra time consuming...

Any Idea's would be great! 开发者_C百科Thanks!

Bulk load the CSV into MySQL, then run a query with GROUP BY id HAVING COUNT(id) > 1 and pull those rows out.

Make sure your MySQL table is set to not accept duplicate values for the ID column, then do the following:

1. Run your INSERT statement as if the ID doesn't exist
2. Test the return. It will fail if the ID already exists
3. If the test failed, run a new INSERT statement to put the record into your second table

This way, if 98% of your records aren't duplicates, you will see no speed decrease, and only the few duplicates will run the extra tests.