Making changes to your website is scary stuff. You don’t have to launch until you’re ready, but you should be aware that, if you’re putting together new stuff that’s not quite ready for primetime, you’ll need to include a robots.txt file.
Robots (also called “bots” or “spiders”) are programs that search engines use to crawl the many websites and pages on the web by retrieving linked pages. The way to exclude robots from looking at particular files on a server (where your website lives) is to…
…Include a robots.txt file on your server.
For a great "how to," check out this article on TheSiteWizard.com, paying close attention to the subheader "How To Set Up A Robots.txt File." You can find additional information through Google’s Webmaster Central by reading this article, titled "Block Or Remove Pages Using A Robots.txt File," as well as the three articles that follow it.
Keep in mind, however, that the only way you can truly restrict access to information on your server is to create a password protected directory on your web server.
To prevent a rogue search bot from adding your junk to their search results against your will, you’ll need this additional safeguard in place that is more brick wall than stop sign. The most typical setup is a pair of files called .htaccess and .htpasswd. Many web hosts have a panel that allows a person without a computer science degree to set up a folder with password protection. A brief, relatively pain-free call to your web host’s tech support will allow them to guide you through the short process of putting up your robot anti-venom. Just remember that this is not air tight security from anything other than killer robots, so it’s probably not an awesome idea to backup your sensitive financial data in the same folder. A rogue human could find the unencrypted data.
