Robots.txt question

[Datagg]

Is it still necessary under today's setup to still block /Merchant2/ in the robots.txt?

[Bruce - PhosphorMedia]

Necessary? Well maybe yes, maybe no. Advisable...yes. Why have search engines wasting their time indexing things you don't need them to index.

[Datagg]

Thanks Bruce. One reason I asked is when poking around some stores here in our community I didn't see any disallows for /mm5/ or /merchant2/ around. Thought went thru my mind that maybe under new miva this was somehow not important any longer or worse yet a disadvantage. Thanks for stepping in and saying best to leave it as is.

[Pete McNamara]

Actually, this strategy is no longer the best as over a year ago, Google started marking down sites where googlebot doesn't have complete access to areas that Google feels it should. I assume they spider the blocked directory and then make a determination, based on an algorithm. Evidently they feel that complete access is worth considerably more to them than the cost of extra spidering. The solution is to ensure all product and category pages contain the link rel="canonical" tag then allow googlebot to spider all directories. This is especially so when the CSS and JavaScript files are in sub directories of the mm5/Merchant2 directory (as compared to being in the "root" i.e. the mm5/Merchant2 directory) (which they are in V9). Over the past few years Google has been transforming itself from a "Do no evil" company into a "What is good for Alphabet, is good for the world" company. I will get off my soapbox now.

[Datagg]

Interesting Pete. So this still is up in the air then it appears?... So what is the the general consensus among other folks in the community then. With it becoming increasingly difficult to rank well these days I wish this part of the equation could be put to rest once and for all. Running a test tonight via bing for mobile friendly list, Bing came up with images /Merchant2/ blocked. So i tossed in a
User-agent: bingbot
Allow: /Merchant2/graphics/
Allow: /Merchant2/images/

and it passed with no image blocks... Yet at the end of the day, should /mm5/ or in my case with a new site /Merchant2/ be blocked at all. Perhaps more can chime in on this as i think allot including myself would love to know more on this issue.

[Dan - Glendale Designs]

You should not block Google from any resources (css, javascript and/or images) used by landing pages/documents that you want to market via organic listings, Adwords or Google Shopping: https://webmasters.googleblog.com/20...webmaster.html

This all plays into Google's push for speed and usability.

[Datagg]

Thanks Dan. I shall remove Disallow: /Merchant2/ and /mm5/

[alphabet]

Is it necessary in today's set-up to keep a /merchant2_old/ directory or any of the other folders that have become outdated such as /some-old-third-party-module-folder/

[Dan - Glendale Designs]

Is it necessary in today's set-up to keep a /merchant2_old/ directory or any of the other folders that have become outdated such as /some-old-third-party-module-folder/

As long as they are truly no longer necessary, i.e. there are no resources used or needed for reference and any associated links have been accounted for. Over the years we have had a lot of clients still calling images in from these old directories despite our best efforts to advise otherwise. Year or two down the road they clean house only to find that their images are no longer working. It's a good idea to look at modified datestamps and any recent activity for those directories, when it doubt ask the host if they see anything before removing them. (make baskups as needed)

[alphabet]

Good advice. Thank you!