PDA

View Full Version : Where is my robots.txt file?



Math
01-08-2007, 10:45 PM
When I looked in to my stats I have seen that 404 error for robots.txt file. (my site mathsupporter.com set up on this week only.) Do we manually set up one file or is it alrady there?

Any help is welcome

capitolh
01-08-2007, 10:48 PM
Well you have to write one yourself, but that won't make the error go away, that error is some sort of global server error I believe.

felgall
01-09-2007, 02:13 AM
The top section of the error stats are errors that could belong to anyone on the server but the software can't work out who so it displays them to everyone in the hope that the person they belong to will realise it and fix the error.

Setting up a robots.txt file with the following one line will ensure that your site isn't the one producing the 404 errors for that file without affecting search engines at all (apart from those that wont search your site at all if the file is missing which will be able to include you once you add the file):

User-agent: *

Math
01-09-2007, 08:51 AM
Well, I am using one software called EWISOFT for editing my pages. Actually What is a robot.txt file? I think it is some thing for thr search engine. How can I make one? and what should I include in it?
Thanks

Early Out
01-09-2007, 08:57 AM
What is a robot.txt file? I think it is some thing for thr search engine. How can I make one? and what should I include in it?
Thanks
Read all about it here (http://www.robotstxt.org/).

Math
01-09-2007, 11:19 AM
This site is very helpful... :)
Thanks