After the latest round of Copiepresse vs. Google; Google posted some great informational articles on the classic robots.txt file as well as other ways to keep your information from being indexed.
- Controlling how search engines access and index your website
- The Robots Exclusion Protocol
- Robots Exclusion Protocol: now with even more flexibility
Danny Sullivan has several write ups on robots.txt and Copiepresse as well.
- Search Engines, Permissions & Moving Forward In Copyright Battles – Daggle.com
- Google’s Belgium Fight: Show Me The Money, Not The Opt-Out, Say Publishers – Search Engine Watch