Elementary SEO: All about robots.txt

Posted by Posted on by
0

SEO is about making your pages rank higher on search results. But there are certain pages on your website which you don’t want users to arrive at from search results. The robots.txt file is used to exclude such pages from showing up in search results.

Search engines use bots or robots to crawl websites and learn about them, so that they’ll know which websites should show up for a particular keyword. Whenever such bots arrive at a website, the first thing they look for is the robots.txt file, because it contains instructions from the website’s owner. Now, there are good bots and bad bots. The bad ones especially, like the malware bots that are on the lookout for security vulnerabilities, pay no attention to the robots.txt file.

What is the role of robots.txt?

It contains two important information. Which bots are allowed to crawl this website, and which pages on this site should not be crawled.

How to create a robots.txt?

It can be created using any text-editor. The name of this file is case-sensitive, so it should be lower-case only. The robots.txt file should be put in the root folder of your website, along with the index or welcome page, so that the path of this file always is, www.yourdomainname.com/robots.txt .

It usually has two commands. User-agent is to specify the bot to which the following instructions apply. Disallow specifies the pages which are restricted.

A simple example of a robots.txt file is as below.

User-agent: *
Disallow: /

So, in the example above, the “*” beside User-agent says, the following commands apply to every kind of bot that lands on this site.
The “/” beside Disallow, represents all sub-directories in the root folder are restricted to bots. That means, no page inside the root folder should be crawled, by any bot.

Here are a few examples. To permit select bots, and keep the rest out,

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /

To restrict select directories on a website from being crawled, the commands would be,

User-agent: *
Disallow: /directory/

To block files of a specific type,

User-agent: Googlebot

Disallow: /*.gif$

To block a particular directory, and everything in it,

User-agent: Googlebot-Image

Disallow: /images/dogs.jpg

Alternate method – META TAG:
You can also include a robots <meta> in the header of every page on your site. The syntax is,

<META NAME=”ROBOTS” CONTENT=”NOINDEX, FOLLOW”>
<META NAME=”ROBOTS” CONTENT=”INDEX, NOFOLLOW”>

The FOLLOW / NOFOLLOW attribute is for the links on that page. If it is NOFOLLOW, then bots should not be following any links on that page. If no meta tag is included, then it implies INDEX and FOLLOW, so there is no need to explicitly mention this.

As with Zoho Sites, you can access the crawler specification in the SEO settings page.

The commands entered here will automatically be saved in the robots.txt file of your website, and can be accessed at www.yourdomainname.com/robots.txt .

Search Engine Optimization: Elementary Essentials

Posted by Posted on by
0

The topics discussed here are very simple changes to your webpages, which when combined, have a notable impact on how your website is treated by search engines.

Here is what matters most, and why.

Keywords:

It all starts with this, even though it is not exposed to users. It is the term that people search for, in order to arrive at a website. You first need to identify the keywords which are both, relevant to what you offer, and are likely to bring you traffic. Keywords of a page are different from keywords of another page, because each page has its own focus terms. Identifying keywords is the most important task of them all, because the title, description and URL of that page depend on, and contain the keywords.

The more specific your keywords are, the better. For example,if you own a bakery in Seattle, and specialize in muffins, then Muffins in Seattle is a better option than Cakes.

Users are more likely to click on a search result that contains the term they just searched for. That is why the title and description draw attention if when contain these keywords.

Title:

The title of your page is enclosed in <title> tags, and is exposed as marked in the image above. There are some guidelines an optimum title would always follow.

1. Uniqueness – No two pages of your website should have the same title.
2. Readability – It should be a meaningful sentence, long enough (preferably 65 characters), and is not spammy.
3. Keyword – The title should contain the keyword which that page focuses on. The sooner it appears in the title, the better.

Description:

Description appears below the title on search results, and should support the claim you’ve made in your title.

1. Preferably 160 characters long.
2. Should also contain the keyword focused on that page.
3. Should be unique, informative and interesting
4. Should sum up the content on the entire page.

URL Structure:

Easily understandable URL’s have two main advantages. They make your website easily navigable to users, and simplifies managing your pages. The characteristics of a good URL are,

1. few, easy words, no numbers
2. Not too long
3. Use specific keywords like home.html, contact.html, etc. instead of generic ones like page1.html, page2.html, etc.

Headings:

It is best to have only one H1 heading tag per page, which should contain the same keyword used in title and description tags.

SEO with Zoho Sites:

All these are enclosed in meta tags like <title>, <description>, etc. If you are managing your own HTML files on Zoho Sites, you are probably familiar with these tags. Otherwise, if you are using the drag-and-drop builder, things are even easier. Zoho Sites lets you enter meta details for every page on your site.

In the Settings page, you have SEO options.

Pick the page from the list, and enter the meta details for that page. Once done, save the changes, and pick another page to optimize.

In addition to these meta information, there is another essential optimization technique, but that happens automatically on Zoho Sites; the sitemap. For every website published, Zoho Sites automatically generates the sitemap.xml file. A sitemap helps search engines to identify the entire structure of your website, so that no page goes unnoticed. It also helps in identifying the navigation path to every page.

All the above parameters focus only on the text on a webpage. There are numerous other techniques to drive more traffic into your website, which we will gradually get into. Try optimizing your web pages by filling these parameters, and see how it affects your site’s ranking on search engines.