View Full Version : Where is my robots.txt file?
When I looked in to my stats I have seen that 404 error for robots.txt file. (my site mathsupporter.com set up on this week only.) Do we manually set up one file or is it alrady there?
Any help is welcome
capitolh
01-08-2007, 10:48 PM
Well you have to write one yourself, but that won't make the error go away, that error is some sort of global server error I believe.
felgall
01-09-2007, 02:13 AM
The top section of the error stats are errors that could belong to anyone on the server but the software can't work out who so it displays them to everyone in the hope that the person they belong to will realise it and fix the error.
Setting up a robots.txt file with the following one line will ensure that your site isn't the one producing the 404 errors for that file without affecting search engines at all (apart from those that wont search your site at all if the file is missing which will be able to include you once you add the file):
User-agent: *
Well, I am using one software called EWISOFT for editing my pages. Actually What is a robot.txt file? I think it is some thing for thr search engine. How can I make one? and what should I include in it?
Thanks
Early Out
01-09-2007, 08:57 AM
What is a robot.txt file? I think it is some thing for thr search engine. How can I make one? and what should I include in it?
Thanks
Read all about it here (http://www.robotstxt.org/).
This site is very helpful... :)
Thanks
Powered by vBulletin® Version 4.1.10 Copyright © 2012 vBulletin Solutions, Inc. All rights reserved.