|
|
330dfc3993
|
robotstxt : the analyze must be done at start. Use the constructor
git-svn-id: https://svn.fournier38.fr/svn/ProgSVN/trunk@5344 bf3deb0d-5f1a-0410-827f-c0cc1f45334c
|
2019-06-12 07:03:44 +00:00 |
|
|
|
f574476427
|
robotsTxt : update Tests
git-svn-id: https://svn.fournier38.fr/svn/ProgSVN/trunk@5343 bf3deb0d-5f1a-0410-827f-c0cc1f45334c
|
2019-06-11 18:59:44 +00:00 |
|
|
|
ded1628c1f
|
robotstxt : rewrite all the parser with the logic defined in RFC
git-svn-id: https://svn.fournier38.fr/svn/ProgSVN/trunk@5342 bf3deb0d-5f1a-0410-827f-c0cc1f45334c
|
2019-06-11 09:08:53 +00:00 |
|
|
|
da4215dbf2
|
Update robotstxt file
git-svn-id: https://svn.fournier38.fr/svn/ProgSVN/trunk@5340 bf3deb0d-5f1a-0410-827f-c0cc1f45334c
|
2019-06-10 14:38:23 +00:00 |
|
|
|
1f466ffa47
|
robotstxt : Add support to understand the robots.txt files
git-svn-id: https://svn.fournier38.fr/svn/ProgSVN/trunk@5339 bf3deb0d-5f1a-0410-827f-c0cc1f45334c
|
2019-06-10 14:35:49 +00:00 |
|
