The secure CMS.

Welcome to Geeklog, Anonymous Sunday, December 22 2024 @ 03:51 pm EST

Geeklog Forums

Prevent search engines from indexing

 New Topic  Post Reply
 09/27/06 06:27am (Read 1,919 times)  
Status: offline

Robin

Forum User
Full Member
Registered: 02/15/02
Posts: 725
Hello,
currently Geeklog provides this functionality with this file, robots.txt:
Text Formatted Code
User-agent: *
Disallow: /comment.php
Disallow: /docs
Disallow: /pingback.php
Disallow: /submit.php
Disallow: /trackback.php
 


It means: All bots leave the following files and directories.

I would like to prevent bots from indexing the whole site. Is it enough and whether this works:
Text Formatted Code
User-agent: *
Disallow: /

 

TIA
Robert

Geeklog Polish Support Team
 Quote
 09/27/06 07:55am  
Status: offline

jmucchiello

Forum User
Full Member
Registered: 08/29/05
Posts: 985
Yep. Assuming they obey the robots.txt file. Almost all search engines do but you can't really stop the ones that don't.
 Quote
 09/27/06 09:29am  
Status: offline

Robin

Forum User
Full Member
Registered: 02/15/02
Posts: 725
Quote by jmucchiello: Yep. Assuming they obey the robots.txt file.


Thanks Smile Is there another way to achieve that?
Geeklog Polish Support Team
 Quote
 09/27/06 02:48pm  
Status: offline

Dirk

Site Admin
Admin
Registered: 01/12/02
Posts: 13073
Location:Stuttgart, Germany
What exactly are you trying to accomplish?

The big search engines (Google, Yahoo, MSN, Ask.com, ...) follow the robots.txt standard. There are always a few clueless bots by smaller companies, but those don't really matter anyway.

Unless you had security or spam prevention in mind, but that would be different topics.

bye, Dirk
 Quote
 New Topic  Post Reply
All times are EST. The time is now 03:51 pm.
  • Normal Topic
  • Sticky Topic
  • Locked Topic
  • New Post
  • Sticky Topic W/ New Post
  • Locked Topic W/ New Post
  •  View Anonymous Posts
  •  Able to post
  •  Filtered HTML Allowed
  •  Censored Content