开发者

Translation of data during import

I am importing approx 350,000 lines from a CSV file into MySQL using PHP (Symfony Framework if that matters) I am reading in each line and inserting into the SQL table as I go. This will be a daily process - and the number of lines will increase.

I have to translate 2 of the columns to a number (which is a foreign key). I want to know the most efficient way of doing it. I am experimenting with using a lookup table in MySQL and also an array in PHP - but I am wondering if there is a more efficient way.

There are approx 60 different combinations the 2 columns could be, here is an example of the translation table I have

Feature         Type          CODE
ANZIRL          Voice Call    8
BT2DT2          Data Call     6
BT2I开发者_开发技巧L2          Voice Call    1
BT2UK2          Voice Call    2
DG2DG2          Voice Call    4
DG2DT2          Data Call     3
DG2EC           Voice Call    1
DG2EDQ          Voice Call    5
DG2EZY          Voice Call    7

And the CSV will be as follows

DG2DG2,x,x,x,Voice Call,x,x
DG2DT2,x,x,x,Data Call,x,x

etc...

I need to insert that into my table as

x,4,x,x
x,3,x,x

etc ....


I would batch import the CSV file into MySQL as is (i.e. create a table that mirrors CSV field layout), and then run a query against the CSV import table to get the foreign key code.

Something like:

INSERT into sometable (code,other,fields)
SELECT code, other, fields
FROM codes c
JOIN csvtable ct
WHERE ct.feature = c.feature AND ct.type = c.type

Basic idea is to not perform processing on row-by-row basis in PHP or MySQL, but to get all the data into MySQL and then run some version of above query to efficiently do the foreign key translation in 1-step.

If you go row-by-row with 350,000+ records, that will be a fairly slow process, php is not terribly efficient...

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜