Robots.txt

[dkmcm4]

Does Miva Merchant create a robots.txt file or is that something I need to do myself?

Thanks

[ILoveHostasaurus]

Re: Robots.txt

Merchant does not, and unless you have something you want to prevent search engines from spidering, you don't need to create one. There are some side uses for it though, you can use it to make some search engines crawl your site slower rather than it's originally designed purpose of telling them not to index certain things.

[dkmcm4]

Re: Robots.txt

I'm trying to create a google friendly SMAP because it want's XML which isn't what Miva creates when it makes an SMAP. I keep running into roadblocks and figured I should ask.

Thanks for the help though.

[ILoveHostasaurus]

Re: Robots.txt

I think by default they look for the file as /sitemap.xml on your site, so unless you place it somewhere else or name it differently, shouldn't need the robots.txt for that. As far as creating a google friendly smap, don't know much about that. :)

[koala]

Re: Robots.txt

Just go and create a sitemap for free with a handy tool like this:

http://www.auditmypc.com/xml-sitemap.asp

[Gilbert]

Re: Robots.txt

Template Data Feed from Emporium Plus HERE – Froogle (Google base), Yahoo, Shopzilla, Google Sitemap, etc.

[Barrett]

Re: Robots.txt

Google does not require xml. They will accept a simple text file with up to 50,000 urls.

A Sitemap should contain a list of your site's URLs - up to 50,000 of them. If you have a large site with more than 50,000 URLs, you should create multiple Sitemaps and submit a Sitemap index file <>.
You can provide Google with a simple text file that contains one URL per line. For example:
http://www.example.com/file1.html

http://www.example.com/file2.html For best results, follow these guidelines:

1. You must fully specify URLs as Google attempts to crawl them exactly as provided.
2. Each text file can contain a maximum of 50,000 URLs. If you site includes more than 50,000 URLs, you can separate the list into multiple text files and add each one separately.
3. The text file must use UTF-8 encoding. You can specify this when you save the file (for instance, in Notepad, this is listed in the Encoding menu of the Save As dialog box).
4. The text file should contain no information other than the list of URLs.
5. The text file should contain no header or footer information.
6. You can name the text file anything you wish. Google recommends giving the file a .txt extension to identify it as a text file (for instance, sitemap.txt).

You should upload the text file to your server, generally to the highest-level directory you want search engines to crawl. Once you've created this file, you can submit it as a Sitemap. This process, while manual, is the simplest and is probably best if you're not familiar with scripting or managing your web server.

[dkmcm4]

Re: Robots.txt

Thanks you explained it perfectly

[daniellim]

Re: Robots.txt

I found the following on my robot.txt:

User-Agent: Googlebot
Disallow: *.csi
Disallow: /Merchant2/
Allow: /Merchant2/graphics/

I'm not a tech savvy, can someone tell me is there any issue with my robot.txt?
Would the line of "Allow: /Merchant2/graphics/" still work with the 'Disallow: /Merchant2/' line already before it, limiting access to the /Merchant2?

thanks,
Dan

[Krsullivan]

Re: Robots.txt

Hey Everyone,

This may be off topic a bit, but it is about Robot.txt files. We use McAfee Secure for PCI compliance- no problem there, anyway, our bank sent us a letting telling us they set up a mandatory account for us with Trustwave (PCI company) and would be deducting the cost from our merchant account. (yes mandatory, but only $60 buck real cheap so I don't care)

Anyway, we failed our first scan of course with trustwave (needed to clear up the false positives) and found that one of the Failures was using Disallows in our robot.txt file. This was a priority 4 failure. Has anyone heard or experienced anything on this? I was surprized, yes I wrote a justification for using desallows for seach engines spiders, not sensitive data was in any of those folders. I have not heard back, I have found a few post on some other forums about this, but those showed this as a low prioity fault and not enough to flag PCI failure. I'm not sure if this is just a simple mistake on their part or not. The d

Thanks

Kevin
www.automotiveworkwear.com
www.sullivanuniforms.com