If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.
Merchant does not, and unless you have something you want to prevent search engines from spidering, you don't need to create one. There are some side uses for it though, you can use it to make some search engines crawl your site slower rather than it's originally designed purpose of telling them not to index certain things.
I'm trying to create a google friendly SMAP because it want's XML which isn't what Miva creates when it makes an SMAP. I keep running into roadblocks and figured I should ask.
I think by default they look for the file as /sitemap.xml on your site, so unless you place it somewhere else or name it differently, shouldn't need the robots.txt for that. As far as creating a google friendly smap, don't know much about that. :)
Google does not require xml. They will accept a simple text file with up to 50,000 urls.
A Sitemap should contain a list of your site's URLs - up to 50,000 of them. If you have a large site with more than 50,000 URLs, you should create multiple Sitemaps and submit a Sitemap index file <>.
You can provide Google with a simple text file that contains one URL per line. For example: http://www.example.com/file1.html
You must fully specify URLs as Google attempts to crawl them exactly as provided.
Each text file can contain a maximum of 50,000 URLs. If you site includes more than 50,000 URLs, you can separate the list into multiple text files and add each one separately.
The text file must use UTF-8 encoding. You can specify this when you save the file (for instance, in Notepad, this is listed in the Encoding menu of the Save As dialog box).
The text file should contain no information other than the list of URLs.
The text file should contain no header or footer information.
You can name the text file anything you wish. Google recommends giving the file a .txt extension to identify it as a text file (for instance, sitemap.txt).
You should upload the text file to your server, generally to the highest-level directory you want search engines to crawl. Once you've created this file, you can submit it as a Sitemap. This process, while manual, is the simplest and is probably best if you're not familiar with scripting or managing your web server.
I'm not a tech savvy, can someone tell me is there any issue with my robot.txt?
Would the line of "Allow: /Merchant2/graphics/" still work with the 'Disallow: /Merchant2/' line already before it, limiting access to the /Merchant2?
This may be off topic a bit, but it is about Robot.txt files. We use McAfee Secure for PCI compliance- no problem there, anyway, our bank sent us a letting telling us they set up a mandatory account for us with Trustwave (PCI company) and would be deducting the cost from our merchant account. (yes mandatory, but only $60 buck real cheap so I don't care)
Anyway, we failed our first scan of course with trustwave (needed to clear up the false positives) and found that one of the Failures was using Disallows in our robot.txt file. This was a priority 4 failure. Has anyone heard or experienced anything on this? I was surprized, yes I wrote a justification for using desallows for seach engines spiders, not sensitive data was in any of those folders. I have not heard back, I have found a few post on some other forums about this, but those showed this as a low prioity fault and not enough to flag PCI failure. I'm not sure if this is just a simple mistake on their part or not. The d
Comment