What is the best way to parse this page using nokogiri?
I am trying to figure out the best way to parse a search results screen that consists of 25 repeating chucks similar this:
Name: JOHN DOE / COMPANY NAME
Status: ACTIVE
Date Joined: 2007-08-17
Address: 123 MAIN STREET
City: ANYTOWN State/Territory/Other: NEW YORK Country: US
Postal Code/Zip Code: 10101
I managed to parse and clean-up the page to return one of the 25 result sets and I'm stuck with how to return the rest. I thought of implementing a variable that would increment from 9 to 33 but was unable to get it to work. The code I'm using looks like this:
require "nokogiri"
class String
def astrip
self.gsub(/([\x09|\x0D|\n|\t])|(\xc2\xa0开发者_运维百科){1,}/u, '').strip
end
end
i = 9
f = File.open("testpage.html", "r:iso-8859-1:utf-8")
doc = Nokogiri::HTML(f)
NAME = doc.css(":nth-child(" + i.to_s + ") div:nth-child(1) a").text.astrip.split("/")
NAME_URL = doc.css(":nth-child(" + i.to_s + ") div:nth-child(1) a").map { |link| link['href'] }
STATUS = doc.css(":nth-child(" + i.to_s + ") div:nth-child(2) a").text
JOINED = doc.css(":nth-child(" + i.to_s + ") div:nth-child(3)").text.gsub("Date Joined:", "").astrip.strip
ADDRESS1 = doc.css(":nth-child(" + i.to_s + ") div:nth-child(4)").text.gsub("Address:", "").astrip.strip
ADDRESS2 = doc.css(":nth-child(" + i.to_s + ") div:nth-child(5)").text.astrip.gsub("City:", "").gsub("State/Territory/Other", "").gsub("Country", "").split(":")
ZIPCODE = doc.css(":nth-child(" + i.to_s + ") div:nth-child(6)").text.gsub("Postal Code/Zip Code:", "").astrip.strip
Output = NAME[0].strip, NAME[1].strip, NAME_URL[0].to_s.strip, STATUS, JOINED, ADDRESS1, ADDRESS2[0].strip, ADDRESS2[1].strip, ADDRESS2[2].strip, ZIPCODE
p Output
It returns an output that I'm happy with that looks like this:
["JOHN DOE", "COMPANY NAME", "http://linktoprofile/johndoe", "ACTIVE", "2007-08-17", "123 MAIN STREET", "ANYTOWN", "NEW YORK", "US", "10101"]
Without sample HTML our ability to provide a working solution is really limited.
This should give you a starting point to work from:
require 'nokogiri'
html = <<EOT
<html>
<body>
<div>
<p><b>Name:</b> JOHN DOE / COMPANY NAME</p>
<p><b>Status:</b> ACTIVE</p>
<p><b>Date Joined:</b> 2007-08-17</p>
<p><b>Address:</b> 123 MAIN STREET</p>
<p><b>City:</b> ANYTOWN <b>State/Territory/Other:</b> NEW YORK <b>Country:</b> US</p>
<p><b>Postal Code/Zip Code:</b> 10101</p>
</div>
</body>
</html>
EOT
doc = Nokogiri::HTML(html)
data = doc.search('div').map { |div|
name = div.at('//p[1]').text[/:(.+)/, 1].strip
status = div.at('//p[2]').text[/:(.+)/, 1].strip
date_joined = div.at('//p[3]').text[/:(.+)/, 1].strip
address = div.at('//p[4]').text[/:(.+)/, 1].strip
city_state_country = div.at('//p[5]').text
postal_code = div.at('//p[6]').text[/:(.+)/, 1].strip
city, state, country = (city_state_country.match(%r{City:(.+) State/Territory/Other:(.+) Country:(.+)}).captures).map{ |s| s.strip }
{
:name => name,
:status => status,
:date_joined => date_joined,
:address => address,
:city => city,
:state => state,
:country => country,
:postal_code => postal_code
}
}
The resulting output looks like:
require 'pp'
pp data
# >> [{:name=>"JOHN DOE / COMPANY NAME",
# >> :status=>"ACTIVE",
# >> :date_joined=>"2007-08-17",
# >> :address=>"123 MAIN STREET",
# >> :city=>"ANYTOWN",
# >> :state=>"NEW YORK",
# >> :country=>"US",
# >> :postal_code=>"10101"}]
If you want an array of arrays, use this in the map block:
[
name,
status,
date_joined,
address,
city,
state,
country,
postal_code
]
which will generate:
# >> [["JOHN DOE / COMPANY NAME",
# >> "ACTIVE",
# >> "2007-08-17",
# >> "123 MAIN STREET",
# >> "ANYTOWN",
# >> "NEW YORK",
# >> "US",
# >> "10101"]]
An alternate way to do the lookups, which I think is easier to maintain because it's more DRY, is:
data = doc.search('div').map { |div|
name,
status,
date_joined,
address,
city,
state,
country,
postal_code = [
'Name',
'Status',
'Date Joined',
'Address',
'City',
'State/Territory/Other',
'Country',
'Postal Code/Zip Code'
].map { |t|
div.at( %Q(//p/b[text()="#{t}:"]) ).next.text.strip
}
加载中,请稍侯......
精彩评论