How to import large amounts of data into Rails?
To load small amounts of data, I've been using rake tasks to important data from CSVs into Rails:
desc "Import users."
task :import_users => :environment do
File.open("users.txt", "r").each do |line|
name, age, profession = line.strip.split("\t")
u = User.new(:name => name, :age => age, :profession => profession)
u.save
end
end
For larger files (about 50,000 re开发者_C百科cords), though, this is incredibly slow. Is there a faster way to import the data?
You might want to take a look at activerecord-import and check out this similar thread.
Without extra libraries (I agree that a bulk import with AR extensions should be quicker)(though AR:Extension skips model validations) you can add a little bit of concurrency and take advantage of a multicore machine
# Returns the number of processor for Linux, OS X or Windows.
def number_of_processors
if RUBY_PLATFORM =~ /linux/
return `cat /proc/cpuinfo | grep processor | wc -l`.to_i
elsif RUBY_PLATFORM =~ /darwin/
return `sysctl -n hw.logicalcpu`.to_i
elsif RUBY_PLATFORM =~ /win32/
# this works for windows 2000 or greater
require 'win32ole'
wmi = WIN32OLE.connect("winmgmts://")
wmi.ExecQuery("select * from Win32_ComputerSystem").each do |system|
begin
processors = system.NumberOfLogicalProcessors
rescue
processors = 0
end
return [system.NumberOfProcessors, processors].max
end
end
raise "can't determine 'number_of_processors' for '#{RUBY_PLATFORM}'"
end
desc "Import users."
task :fork_import_users => :environment do
procs = number_of_processors
lines = IO.readlines('user.txt')
nb_lines = lines.size
slices = nb_lines / procs
procs.times do
subset = lines.slice!(0..slices)
fork do
subset.each do |line|
name, age, profession = line.strip.split("\t")
u = User.new(:name => name, :age => age, :profession => profession)
u.save
end
end
end
Process.waitall
end
on my machine with 2 cores and the fork version I get
real 1m41.974s
user 1m32.629s
sys 0m7.318s
while with your version:
real 2m56.401s
user 1m21.953s
sys 0m7.529s
You should try FasterCSV. It is quite fast and dead easy to use for me.
精彩评论