What's the best way to handle the robots.txt file with a Miva site that has static mirror pages? Here's the situation:
I recently used Phosphor's Merchant Optimizer module to create static pages on a Miva 5 site, and it works very nicely. I used a robots.txt file to disallow the mm5 directory to prevent spiders from thinking I was producing duplicate content. But I see in google's webmaster tools that it wants to index certain resources within the mm5 directory, and I am concerned that I may be missing out on search results -- for example if all the product images are inside the mm5 directory (inside images/00000001) then they would not show up on google image search, right?
Should I not be blocking the entire mm5 directory?
Thanks in advance,
--Lee
I recently used Phosphor's Merchant Optimizer module to create static pages on a Miva 5 site, and it works very nicely. I used a robots.txt file to disallow the mm5 directory to prevent spiders from thinking I was producing duplicate content. But I see in google's webmaster tools that it wants to index certain resources within the mm5 directory, and I am concerned that I may be missing out on search results -- for example if all the product images are inside the mm5 directory (inside images/00000001) then they would not show up on google image search, right?
Should I not be blocking the entire mm5 directory?
Thanks in advance,
--Lee
Comment