|
|
|
Doing Business from the Heart - or - What Connects an Old Box with Web Site Design
A few days ago, my business partner was researching some
professional organisations for us to join. Being the website
expert that he is, he didn't just look at what the various
associations had to offer in terms of membership...
Good Web Design
One of the primary keys to a successful website is a good design. A poorly designed site actually discourages visitors from even getting to your site, and to leave quickly when (and if) they arrive. A well designed site encourages people to visit...
I’m Gettin’ Really Torqued!
For those of you who have been living in a cave for most of your lives, you can translate the “torqued” in the heading of this article to “mad” or “angry”. For today’s discussion of anger, let’s quantify the intensity of our focus with the words...
"I'm just me" - An Overview of an Web er.. Designer, Developer, Consultant and Friend
"I'm just me" - An Overview of an Web er.. Designer, Developer, Consultant and Friend.
I am going to give you a breakdown of my ... er ... work ... playtime .... income ....
I'm a UK based Web Designer, Web Developer, Internet Consultant. I...
Low cost website design company
There are many companies who build websites, all of which who want your business. This puts you in a very strong position when trying to obtain a good deal, a professional looking website at a low affordable price. The world wide web is an...
|
|
| |
|
|
|
|
|
from our sponsors
Internet Home Business Training Course
Your privacy is safe.
|
Why Robots.txt?
I am sure that a lot of you have heard of the file named robots.txt (also called a "robot exclusion file") before. But what does this file really pertain to? Basically you can think of a robots.txt file as a list of rules that search engines follow when they spider your site. A robots.txt file gives you the Webmaster a say in what does and does not get indexed when spiders come to your little corner of the web.
Okay I can hear a few people asking why anyone would want to keep some things from being indexed. I thought the goal was to get indexed, right? Well yes and no, there are quite a few instances when blocking spider access to certain areas or pages is almost a must. Here are several examples of what a person might want to restrict access to: temporary files or directories, presentations, information with a specific sequential order, testing directories or cgi-bin. As you can see just from these few examples there are definitely files that you would most certainly want to keep from being indexed. While there is a Meta tag (<meta name="Robots" content="attributes">) available that does in essence the same thing as a robots.txt file it is not currently 100% supported by search engines. Another drawback is that the tag needs to go on every page you do not want indexed, as opposed to one central point of control. Writing 101
All right I have given you a few vague examples as to what might be included in such a file, essentially there is never going to be a set list of things that should and should not be indexed, a robots.txt file needs to be tailored to your site and your content. There is however a very specific format that needs to be followed when creating a robots.txt file.
Step 1: First a robots.txt file needs to be created in Unix format, or Unix line ender mode. The reason for this is to ensure that there are no carriage returns inserted into your file. I would suggest looking at Notepad++, my personal favorite text editor due to the amount of languages and formatting it supports. Notepad++ is able to create a document directly in Unix format by selecting the "Convert to Unix Format" from the "Format" option. Other plain text editors should be able to achieve the same results however stay away from editors like WordPad or Microsoft Word when creating your robots.txt file. Also I do not recommend using HTML editors for this task.
Step 2: Now lets begin adding some content to our file. A robots.txt file is made up of two fields. The first line is the User-agent line. This line specifies the spider/robot that we are intending to limit or allow. An example of this would be:
User-agent: googlebot
In addition to allowing or restricting specific spiders you can use a wildcard and target all spiders coming to your site. To do this you simply need to place an asterisk (*) in for your User-agent. Example:
User-agent: *
Step 3: Now we will begin to disallow our desired content; either a file or a whole directory can be kept from being index with a robots.txt file. We will do this with the second line of our file the Disallow: directive line. Here is an example:
Disallow: /cgi-bin/
Or for a file:
Disallow: /temp/temp.html
Moreover you are not limited to just one Disallow per User-agent and in fact you can get pretty granular as to what you give spiders access to. Just make sure that you give each Disallow its own line. If you leave the Disallow field empty (i.e.
CONTINUE BELOW...
Didn't Find What You Were Looking For?
Try a Specialized Search HERE
MOST RECENT ARTICLES
Article Dashboard: Internet Business | Web Design
Web Design articles from Article Dashboard
How to Create a Web Site the Smart and Easy Way Posted By : Ryan Gutierrez
Don't let college professors, web developers and other professionals fool you: creating a web site in this day and age is easy! These people are perpetrating the myth that not only is creating a web site hard, it requires technical knowledge such as HTML, CSS and PHP. While it is useful to some extent to know these things, if you're just trying to put together a web site for your club/organization, family or event, learning these skills is not only overkill, it's a waste of time.
Some Tips For Search Engine Friendly Website Design! Posted By : uvi pann
A search engine friendly website with unique content is an extremely important asset for to have more traffic and success on the internet. Cluttered and unorganized content turns people off. However without traffic, even a well designed site will not produce results for you.
How to Increase Visibility Value of your Website with Flash Website Design Po...
Curb the online competition by adding flash design to your website. Its application uplifts the image of your company and gives it a hi-tech look.
Make your Website Valuable for your Business with Good Web Design. Posted By ...
A website can serve as an asset for any business organization provided the web design building services are done efficiently and accordingly.
Graphic Designing for Elevating your Website Image and Profit Posted By : Spi...
Graphic design is a known concept to all, but to produce perfect results it requires great deal of experience and creativity along with market knowledge.
Newsfeed display by CaRP
CONTINUE HERE...
Disallow: ) you are giving permission for all files and directories to be indexed.
One word of caution when writing your robots exclusion file; if you are not careful you can shut one or all spider's access to your site off completely. This would be done by prohibiting access at the root level by using a slash (/). Example:
Disallow: /
If you were to use the asterisk wildcard to specify your User-agent with the above example you would block all search engines from every part of your site.
Step 4: That is all there is to creating a robots.txt file. The final step is to upload it to the root directory of your site: www.yoursite.com/. Make sure that you upload it as ASCII just like all other text files and you are done.
Step 5: Writing a robots.txt file is pretty straightforward after you get comfortable with the files configuration. Once your file is complete and uploaded it is good practice to have it validated; you can do this through www.searchengineworld.com.
Notes: Aside from search engine specific information you are also able to comment your robots.txt file. This is achieved by using the pound sign (#). Though you can place a comment after the Disallow field it is not recommended. Instead make sure that you begin your comments on a new line starting with the pound sign. Example:
# Just making a comment
User-agent: googlebot
Disallow: /cgi-bin/
If you are hesitant about the different steps involved in creating a robots.txt file there are applications available that will help you through the creation process. One application that does this is RoboGen from Rietta Solutions. RoboGen provides you with an Explorer like view that lets you browse the files and directories that you want to restrict access to and creates the robot exclusion file as you go. In Closing As with all things there are going to be some drawbacks you will need to contend with. With the robots.txt file it is the road map effect that it causes; for those with the desire to attempt to see what you do not want made publicly available the file provides them with a prime place to begin looking. Since all robot exclusion files are named the same and are always in the same place probing people will know where to find it.
Still the pros out weight the cons. And by having a robots.txt file present on your site you keep important or private information from ending up in a search engine's cache making it publicly available to a mass audience. This is what the file is there for. If on the other hand you have something that not only needs to be kept private but also needs to be protected you should make sure that access is restricted through much more secure and appropriate means. Robot exclusion files were designed as a method for Webmasters to delimit the access robots have to their sites, providing robots with one central place to look when they begin the task of indexing. To this end the file serves it purpose extremely well and when used properly it makes the job of a Webmaster much easier.
About the Author
Matt Benya is a co-owner of Primate Studios (www.primatestudios.com) an independent development house
focusing on CGI illustration, Web design and multimedia. With 20+ years of art experience and a degree in Network administration Matt is well suited to translate your needs to the Web.
|
|
|
|
|
|