开发者

how to return a list of url from the coding below

as title shows, how do i return a list of urls under (a href) reference and display it in a text file ? The code below return the html form a a website.

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;

public class Main {
    public static void main(String[] args)  {
        try 开发者_运维问答{
            URL my_url = new URL("http://www.placeofjo.blogspot.com/");
            BufferedReader br = new BufferedReader(
               new InputStreamReader(my_url.openStream()));
            String strTemp = "";
            while(null != (strTemp = br.readLine())){
                System.out.println(strTemp);
            }
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
}


You sound like you want to be using an HTML parsing library like HtmlUnit, rather than getting into the hassle of parsing the HTML yourself. The HtmlUnit code would be as simple as:

final WebClient webClient = new WebClient();
webClient.setJavaScriptEnabled(false);
final HtmlPage page = webClient.getPage("http://www.placeofjo.blogspot.com/");

//  Then iterate through
for (DomElement element : page.getElementsByTagName("a")){
    String link = ((HtmlAnchor)element).getHrefAttribute();
    System.out.println(link);
}

Gives output of:

http://www.twitter.com/jozefinfin/
http://www.facebook.com/jozefinfin/
http://placeofjo.blogspot.com/2008_08_01_archive.html
... etc etc
http://placeofjo.blogspot.com/2011_02_01_archive.html
http://endlessdance.blogspot.com
http://blogskins.com/me/aaaaaa
http://weheartit.com


You might want to try parsing the HTML with jsoup and collect all the anchor tags from the page.


Edit (2)

If you're looking for a robust solution (or might need to extend to parsing more HTML), then check out one of the other answers here. If you just want a quick-and-dirty, one time solution you might consider regex.


If I understand you correctly, you want to extract the href values for all <a> tags in the HTML you're fetching.

You can use regular expressions. Something like

String regex = "<a\s.*href=['\"](.*?)['\"].*?>";
Pattern p = Pattern.compile(regex);
Matcher m = p.matcher(text);
while (m.find())
{
    String urlStr = m.group();
}

Edit (1)

Corrected the regex - we want reluctant quantifiers otherwise we'll end up getting everything!

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜