diff --git a/docs/sitemap b/docs/sitemap new file mode 100644 index 0000000..d9ec760 --- /dev/null +++ b/docs/sitemap @@ -0,0 +1,11 @@ +# Sitemap support + +This class allow to read the sitemaps files available in websites. This files +allow to provide web crawlers all the interesting URL to crawl, some optional +other sitemaps files and some parameters concerning the last update of the +pages. + +Two formats are available : a textual one and a XML one. +The texutal format is simple : all the URL are written, one by line. +The XML allow to provide more data like the change frequency, the priority and +the last modification date