XPath expression to extract weblinks from a chinese search (baidu.com)
Does anyone know of an XPath expression which will allow me to extract all the search results returned from baidu.com?
For example, in 开发者_Python百科R, I would usually do something like this this:
# load libraries
library(RCurl)
library(XML)
# get webpage
doc <- getURL("http://www.baidu.com/s?rn=100&bs=chivas+regal&f=8&wd=chivas+regal")
# html structure
html <- htmlTreeParse(doc, useInternalNodes = TRUE, error=function(...){})
# use xpath expression to get links
nodes <- getNodeSet(html, "//a[@href]")
However this only gets about 10 links, when I should have around 100. So I think this means there is something about the baidu html structure which is not clear to me.
Many thanks in advance for your time.
If Xpath is not an absolute requirement try an approach based on regular expressions. The following assumes all links start with http:// and are in double quotes. It uses strapply
to match the indicated regular expression and extract out the back reference, i.e. the part within parentheses.
URL <- "http://www.baidu.com/s?rn=100&bs=chivas+regal&f=8&wd=chivas+regal"
Lines <- readLines(URL)
library(gsubfn)
links <- strapply(Lines, '"(http://[^"]*)"', simplify = c)
精彩评论