Is it still necessary under today's setup to still block /Merchant2/ in the robots.txt?
Announcement
Collapse
No announcement yet.
Robots.txt question
Collapse
X
-
Necessary? Well maybe yes, maybe no. Advisable...yes. Why have search engines wasting their time indexing things you don't need them to index.Bruce Golub
Phosphor Media - "Your Success is our Business"
Improve Your Customer Service | Get MORE Customers | Edit CSS/Javascript/HTML Easily | Make Your Site Faster | Get Indexed by Google | Free Modules | Follow Us on Facebook
phosphormedia.com
-
Thanks Bruce. One reason I asked is when poking around some stores here in our community I didn't see any disallows for /mm5/ or /merchant2/ around. Thought went thru my mind that maybe under new miva this was somehow not important any longer or worse yet a disadvantage. Thanks for stepping in and saying best to leave it as is.Dan
Girlfriends Lingerie - "Keeping It Sexy!"
Sexy Lingerie - Twitter - Facebook- Pinterest - YouTube
Comment
-
Actually, this strategy is no longer the best as over a year ago, Google started marking down sites where googlebot doesn't have complete access to areas that Google feels it should. I assume they spider the blocked directory and then make a determination, based on an algorithm. Evidently they feel that complete access is worth considerably more to them than the cost of extra spidering. The solution is to ensure all product and category pages contain the link rel="canonical" tag then allow googlebot to spider all directories. This is especially so when the CSS and JavaScript files are in sub directories of the mm5/Merchant2 directory (as compared to being in the "root" i.e. the mm5/Merchant2 directory) (which they are in V9). Over the past few years Google has been transforming itself from a "Do no evil" company into a "What is good for Alphabet, is good for the world" company. I will get off my soapbox now.
Comment
-
Interesting Pete. So this still is up in the air then it appears?... So what is the the general consensus among other folks in the community then. With it becoming increasingly difficult to rank well these days I wish this part of the equation could be put to rest once and for all. Running a test tonight via bing for mobile friendly list, Bing came up with images /Merchant2/ blocked. So i tossed in a
User-agent: bingbot
Allow: /Merchant2/graphics/
Allow: /Merchant2/images/
and it passed with no image blocks... Yet at the end of the day, should /mm5/ or in my case with a new site /Merchant2/ be blocked at all. Perhaps more can chime in on this as i think allot including myself would love to know more on this issue.Last edited by Datagg; 03-20-16, 08:57 PM.Dan
Girlfriends Lingerie - "Keeping It Sexy!"
Sexy Lingerie - Twitter - Facebook- Pinterest - YouTube
Comment
-
You should not block Google from any resources (css, javascript and/or images) used by landing pages/documents that you want to market via organic listings, Adwords or Google Shopping: https://webmasters.googleblog.com/20...webmaster.html
This all plays into Google's push for speed and usability.
Comment
-
Is it necessary in today's set-up to keep a /merchant2_old/ directory or any of the other folders that have become outdated such as /some-old-third-party-module-folder/
Comment
Comment