how to return a list of url from the coding below
as title shows, how do i return a list of urls under (a href) reference and display it in a text file ? The code below return the html form a a website.
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;
public class Main {
public static void main(String[] args) {
try 开发者_运维问答{
URL my_url = new URL("http://www.placeofjo.blogspot.com/");
BufferedReader br = new BufferedReader(
new InputStreamReader(my_url.openStream()));
String strTemp = "";
while(null != (strTemp = br.readLine())){
System.out.println(strTemp);
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
You sound like you want to be using an HTML parsing library like HtmlUnit, rather than getting into the hassle of parsing the HTML yourself. The HtmlUnit code would be as simple as:
final WebClient webClient = new WebClient();
webClient.setJavaScriptEnabled(false);
final HtmlPage page = webClient.getPage("http://www.placeofjo.blogspot.com/");
// Then iterate through
for (DomElement element : page.getElementsByTagName("a")){
String link = ((HtmlAnchor)element).getHrefAttribute();
System.out.println(link);
}
Gives output of:
http://www.twitter.com/jozefinfin/
http://www.facebook.com/jozefinfin/
http://placeofjo.blogspot.com/2008_08_01_archive.html
... etc etc
http://placeofjo.blogspot.com/2011_02_01_archive.html
http://endlessdance.blogspot.com
http://blogskins.com/me/aaaaaa
http://weheartit.com
You might want to try parsing the HTML with jsoup and collect all the anchor tags from the page.
Edit (2)
If you're looking for a robust solution (or might need to extend to parsing more HTML), then check out one of the other answers here. If you just want a quick-and-dirty, one time solution you might consider regex.
If I understand you correctly, you want to extract the href
values for all <a>
tags in the HTML you're fetching.
You can use regular expressions. Something like
String regex = "<a\s.*href=['\"](.*?)['\"].*?>";
Pattern p = Pattern.compile(regex);
Matcher m = p.matcher(text);
while (m.find())
{
String urlStr = m.group();
}
Edit (1)
Corrected the regex - we want reluctant quantifiers otherwise we'll end up getting everything!
精彩评论