Crawler access
If your site has content
you don't want Google or other search engines to access, use a
robots.txt file to specify how search engines should crawl your site's
content.
If private or out-of-date content is appearing in Google search
results, use the Removal URL tool to request its removal (but first,
make sure to review our removal requirements).
1. Choose default crawler access
2. Specify additional rules (optional)
| Action | User-agent | Directories and Files | |||
If you want this rule to
affect all files and directories, type /. To specify individual files
or directories, type them one per line. Example:
|
3. Download your new robots.txt file
4. Save your robots.txt file and upload to your site's top-level directory