Building a Better Robots.txt

October 29, 2014 at 10:06 AM

Thanks for the heads up on the new Google arrest warrant for our site files: http://www.reviewsforjoomla.com/forum/index.php?topic=27328.0

I would like to suggest you add /modules/ and /libraries/ to the proposed robots.txt you posted in that Announcement, to cover customers who use libraries like Gantry and modules like JFBConnect, EasySocial & AJAX Search. Also don't forget about the JReviews AJAX trigger (first line below):

User-agent: Mediapartners-Google
Allow: /
User-agent: *
Allow: /index.php?option=com_jreviews&format=ajax
Allow: /components/*.js
Allow: /components/*.css
Allow: /components/*.png
Allow: /components/*.jpg
Allow: /components/*.gif
Allow: /components/*.woff
Allow: /components/*.svg
Allow: /components/*.eot
Allow: /components/*.ttf
Disallow: /components/
Allow: /templates/*.js
Allow: /templates/*.png
Allow: /templates/*.jpg
Allow: /templates/*.gif
Allow: /templates/*.css
Allow: /templates/*.woff
Allow: /templates/*.svg
Allow: /templates/*.eot
Allow: /templates/*.ttf
Disallow: /templates/
Allow: /media/*.js
Allow: /media/*.css
Allow: /media/*.png
Allow: /media/*.jpg
Allow: /media/*.gif
Allow: /media/*.woff
Allow: /media/*.svg
Allow: /media/*.eot
Allow: /media/*.ttf
Disallow: /media/
Allow: /images/*.png
Allow: /images/*.jpg
Allow: /images/*.gif
Disallow: /images/
Allow: /modules/*.js
Allow: /modules/*.css
Allow: /modules/*.png
Allow: /modules/*.jpg
Allow: /modules/*.gif
Allow: /modules/*.woff
Allow: /modules/*.svg
Allow: /modules/*.eot
Allow: /modules/*.ttf
Disallow: /modules/
Allow: /libraries/*.js
Allow: /libraries/*.css
Allow: /libraries/*.png
Allow: /libraries/*.jpg
Allow: /libraries/*.gif
Allow: /libraries/*.woff
Allow: /libraries/*.svg
Allow: /libraries/*.eot
Allow: /libraries/*.ttf
Disallow: /libraries/
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /logs/
Disallow: /plugins/
Disallow: /tmp/
Disallow: /?
Disallow: /component/
Disallow: /images/sampledata/
Disallow: /images/watermarks/
Disallow: /*by:
Disallow: /*order:
Disallow: /*type:
Disallow: /*user:
Disallow: /*view:
Disallow: /*by=
Disallow: /*order=
Disallow: /*type=
Disallow: /*user=
Disallow: /*view=
Disallow: /*month=
Disallow: /*url=
Disallow: /*wid=
Disallow: /*?searchword
Disallow: /*search-results?=
Disallow: /*sort=
Disallow: /*option=
Disallow: /*component/content/article
Disallow: /component/jreviews/upload/
Disallow: /?page

From "Disallow: /?" on down are some of my suggestions to stay out of trouble with some common Joomla SEO pitfalls and also prevent so many JReviews pages getting indexed, for example all the ordered pages. Be sure to remove any of the lines of items you would prefer to be indexed.

I think this is a pretty good robots.txt for JReviews users. If anyone has anything to add/criticize feel free to share.

October 29, 2014 at 10:35 AM