Software for building a sitemap

If I had to create a content inventory for a website that doesn't have a sitemap, and I do not have access to modify the website, but the site is very large. How can I build a sitemap out of that website without having to browse it entirely ? I tried with Visio's sitemap builder, but it fails great time. Let's say for example: I Want to create a sitemap of Stackoverflow. Do you guys k开发者_StackOverflow社区now a software to build it ?

You would have to browse it entirely to search every page for unique links within the site and then put them in an index.

Also for each unique link you find within the site you then need to visit that page and search for more unique links.

You would use a tool such as HtmlAgilityPack to easily grab urls and extract links from them.

I have written an article which touches on the extracting links part of the problem:

• http://runtingsproper.blogspot.com/2009/11/easily-extracting-links-from-snippet-of.html

I would register all your pages in a Database, and then just output them all on a page (php - sql). Maybe even indexing software could help you! First of all, just make sure all your pages are linked up and submit it to google still!

Just googled and found this one. http://www.xml-sitemaps.com/ Looks pretty interesting!

There is a pretty big collection of XML Sitemaps generators (assuming that's what you want to generate -- not a HTML sitemap page or something else?) at http://code.google.com/p/sitemap-generators/wiki/SitemapGenerators

In general, for any larger site, the best solution is really to grab the information directly from the source, for example from the database that powers the site. By doing that you can get the most accurate and up-to-date Sitemap file. If you have to crawl the site to get the URLs for a Sitemap file, it will take quite some time for a larger site and it will load the server during that time (it's like someone visiting all pages in your site). Crawling the site from time to time to determine if there are crawlability issues (such as endless calendars, content hidden through forms, etc) is a good idea, but if you can, it's generally better to get the URLs for the Sitemap file directly.