Translation of data during import
I am importing approx 350,000 lines from a CSV file into MySQL using PHP (Symfony Framework if that matters) I am reading in each line and inserting into the SQL table as I go. This will be a daily process - and the number of lines will increase.
I have to translate 2 of the columns to a number (which is a foreign key). I want to know the most efficient way of doing it. I am experimenting with using a lookup table in MySQL and also an array in PHP - but I am wondering if there is a more efficient way.
There are approx 60 different combinations the 2 columns could be, here is an example of the translation table I have
Feature Type CODE
ANZIRL Voice Call 8
BT2DT2 Data Call 6
BT2I开发者_开发技巧L2 Voice Call 1
BT2UK2 Voice Call 2
DG2DG2 Voice Call 4
DG2DT2 Data Call 3
DG2EC Voice Call 1
DG2EDQ Voice Call 5
DG2EZY Voice Call 7
And the CSV will be as follows
DG2DG2,x,x,x,Voice Call,x,x
DG2DT2,x,x,x,Data Call,x,x
etc...
I need to insert that into my table as
x,4,x,x
x,3,x,x
etc ....
I would batch import the CSV file into MySQL as is (i.e. create a table that mirrors CSV field layout), and then run a query against the CSV import table to get the foreign key code.
Something like:
INSERT into sometable (code,other,fields)
SELECT code, other, fields
FROM codes c
JOIN csvtable ct
WHERE ct.feature = c.feature AND ct.type = c.type
Basic idea is to not perform processing on row-by-row basis in PHP or MySQL, but to get all the data into MySQL and then run some version of above query to efficiently do the foreign key translation in 1-step.
If you go row-by-row with 350,000+ records, that will be a fairly slow process, php is not terribly efficient...
精彩评论