Jump to content

Building a Better Robots.txt


Steven Koontz

Recommended Posts

Thanks for the heads up on the new Google arrest warrant for our site files: http://www.reviewsforjoomla.com/forum/index.php?topic=27328.0

 

I would like to suggest you add /modules/ and /libraries/ to the proposed robots.txt you posted in that Announcement, to cover customers who use libraries like Gantry and modules like JFBConnect, EasySocial & AJAX Search. Also don't forget about the JReviews AJAX trigger (first line below):

 

User-agent: Mediapartners-Google
Allow: /
User-agent: *
Allow: /index.php?option=com_jreviews&format=ajax
Allow: /components/*.js
Allow: /components/*.css
Allow: /components/*.png
Allow: /components/*.jpg
Allow: /components/*.gif
Allow: /components/*.woff
Allow: /components/*.svg
Allow: /components/*.eot
Allow: /components/*.ttf
Disallow: /components/
Allow: /templates/*.js
Allow: /templates/*.png
Allow: /templates/*.jpg
Allow: /templates/*.gif
Allow: /templates/*.css
Allow: /templates/*.woff
Allow: /templates/*.svg
Allow: /templates/*.eot
Allow: /templates/*.ttf
Disallow: /templates/
Allow: /media/*.js
Allow: /media/*.css
Allow: /media/*.png
Allow: /media/*.jpg
Allow: /media/*.gif
Allow: /media/*.woff
Allow: /media/*.svg
Allow: /media/*.eot
Allow: /media/*.ttf
Disallow: /media/
Allow: /images/*.png
Allow: /images/*.jpg
Allow: /images/*.gif
Disallow: /images/
Allow: /modules/*.js
Allow: /modules/*.css
Allow: /modules/*.png
Allow: /modules/*.jpg
Allow: /modules/*.gif
Allow: /modules/*.woff
Allow: /modules/*.svg
Allow: /modules/*.eot
Allow: /modules/*.ttf
Disallow: /modules/
Allow: /libraries/*.js
Allow: /libraries/*.css
Allow: /libraries/*.png
Allow: /libraries/*.jpg
Allow: /libraries/*.gif
Allow: /libraries/*.woff
Allow: /libraries/*.svg
Allow: /libraries/*.eot
Allow: /libraries/*.ttf
Disallow: /libraries/
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /logs/
Disallow: /plugins/
Disallow: /tmp/
Disallow: /?
Disallow: /component/
Disallow: /images/sampledata/
Disallow: /images/watermarks/
Disallow: /*by:
Disallow: /*order:
Disallow: /*type:
Disallow: /*user:
Disallow: /*view:
Disallow: /*by=
Disallow: /*order=
Disallow: /*type=
Disallow: /*user=
Disallow: /*view=
Disallow: /*month=
Disallow: /*url=
Disallow: /*wid=
Disallow: /*?searchword
Disallow: /*search-results?=
Disallow: /*sort=
Disallow: /*option=
Disallow: /*component/content/article
Disallow: /component/jreviews/upload/
Disallow: /?page

 

From "Disallow: /?" on down are some of my suggestions to stay out of trouble with some common Joomla SEO pitfalls and also prevent so many JReviews pages getting indexed, for example all the ordered pages. Be sure to remove any of the lines of items you would prefer to be indexed.

 

I think this is a pretty good robots.txt for JReviews users. If anyone has anything to add/criticize feel free to share.

Link to post
  • 2 weeks later...
  • 3 months later...
  • 2 years later...

This section is only visible with a valid subscription. If you have a valid subscription, please login.

Edited by AndrewG
Link to post
  • 3 months later...
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue. Please refer to our Privacy Policy.