Learning About Robots.txt File?
What is a Robots.txt?
Robots exclusion protocol Generally known as Robots.txt is a Text file With Few Lines Of Code It is Created in the Root Folder of the Blog or Site.
Robots.txt is used to instruct the web crawlers or Search bots to Restrict some parts of the site like pages, Posts, admin Area etc from indexing in Search Engines and how to index and crawl your blog or site. Web crawlers scan the robots.txt file before crawling any website or Blog So Now You can Think How Important is Robots.txt file.
If you add some wrong Code then There is a chance of Your Website Disappearing From Search Engines So Use This Carefully.
Robots.txt is used to instruct the web crawlers or Search bots to Restrict some parts of the site like pages, Posts, admin Area etc from indexing in Search Engines and how to index and crawl your blog or site. Web crawlers scan the robots.txt file before crawling any website or Blog So Now You can Think How Important is Robots.txt file.
If you add some wrong Code then There is a chance of Your Website Disappearing From Search Engines So Use This Carefully.
What is a Blogger Custom Robots.txt File and How To Use It?
Thanks To Blogger That It Creates Robots.txt File For Every Blog And Every Blog Has Default Robots.txt Just The Sitemap Will Be Changed To Your Domain Name i.e
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://www.trendingwidgets.com/sitemap.xml
Explanation About Blogger Default Robots.txt:
The Robots.txt looks Simple in Reading but if you are a Newbie then it is difficult in understanding for the first time but we will make sure that after reading this article you will understand every robots.txt whether it is on blogger or any other platform.
For Your Understanding We Will divide This Robots.txt into Three Parts That are as follows:
- User-agent: Mediapartners-Google
- User-agent: *
- Sitemap: http://www.trendingwidgets.com/sitemap.xml
1. User-agent: Mediapartners-Google:
This User-agent Mediapartners-Google is for Adsense Crawler To Provide Better and Relevant ads on your blogs or Websites. To know More About Adsense Crawler Then Visit This Link:
Now If you are thinking That Either you are not using Adsense or Banned By Adsense Then What Should I do With this.
If you listen to me Leave it It like this only as this is just to display relevant ads.
2. User-agent: *
Asterisk (*) here means All Which means This is For All Crawlers or Robots. Here in this Line We are Selecting All the Crawlers and say them to do disallow and Allow Something.
Disallow: /search
Here Disallow keyword will instruct the Crawler or Search Robot To Not To Visit & Ignore this pages From Crawling and indexing it in the Search Engines.
Disallow: /search means the pages or any link having keyword search just after the domain name will be ignored and not be crawled and indexed by Search Bots in the Search Engines.
For Example: This type of pages will not be index by Search Engines:
http://www.trendingwidgets.com/search/label/Blogging%20Guide
See The Example Above This Type of Pages will not Be Crawled and will not be index by Google or other Search Engines. As This This pages is starting with Keyword /search after the domain name
if we remove Disallow: /search from the above code then crawlers will Crawl your entire blog or site and index this pages also.
Allow: /
This Allow: / keyword tell search robot or Crawlers to visit Home Page and All the pages on your website except All the Disallow Pages.
2. Sitemap: http://www.yoursite.com/sitemap.xml
Here The Last part means your website’s sitemap this is after Allow:/ keyword so here we are telling the search robots to index and crawl all posts of your site. Means whenever the web crawlers scan our robots.txt file they will find a path to our sitemap where all the links of our published posts present. Web crawlers will find it easy to crawl all of our posts. So Adding the sitemap in the robots.txt is very good for SEO as whenever Search Robots will scan your Robots.txt they should also go through your sitemap.
How To Add Static Pages in the Sitemap:
By Default Blogger Added only Posts Sitemap in the Robots.txt But You can Add Your Static-pages sitemap just add this Sitemap URL after the first sitemap.
OR
Sitemap: https://your-blog.blogspot.com/sitemap-pages.xmlIf you have custom Domain then follow the first instance and change the URL with your domain name
If you have Blogspot Blog Structure Then Follow The Second Instance and change the URL with your domain name
How To Add Custom Robots.txt File correctly In Blogger?
STEP 1: Log in to your Blogger account and Go to your Blogger Dashboard
STEP 2: Now go to Settings-> Search Preference -> As Shown in the Image Below
STEP 4: Now Click on Save Changes to save your Default Robots.txt File. Now This File Will replace your Default Robots.txt file So Don't make any mistake.
Now You can Check your New Robots.text file in action by entering
http://www.trendingwidgets.com/robots.txt
Change the domain name with your domain name and paste it in the browser url field and test.
That's It Thanks For Following our Tutorial '' How To Use Robots.txt File Correctly in Blogger Blogs & Boost SEO! '' Along With us If you found any difficulty Please Comment and Share Your Valuable Opinion. And Stay tuned for More Tutorials Like This and Share this with your friends.
No comments:
Post a Comment