 |
| Crawling and Indexing issue |
How to Solve Crawling & Indexing issue in Blogger | How to add custom robot.txt
Before diving into the issue and the solution, I will tell you what is crawling and indexing so you can easily understand the further procedure.
What is Crawling?
When search engine bots visit a website or any page of the website to check what is on the page, it is called crawling and it helps the search engine to understand which type of content is present on the page and then index that page and content on their database.
What is Indexing?
Organizing the information for a quick response after crawling a page or site is called indexing. Search Engines store the crawled information in their databases and show it to relevant searches.
What is Sitemap?
A Sitemap is a file through which you give information to search engines about the pages and other data like videos and images on your website. It helps a search engine to crawl the site more efficiently.
Now let's come to the main topic - Solving Crawling and Indexing issues in Blogger.
1- Copy the code below:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://yoursite.com/atom.xml?redirect=false&start-index=1&max-results=500
2- Go to your Blogger settings, scroll down and look for Crawlers and indexing.
3- Enable Custom robots.txt and then click the custom robot.txt.
4- Paste the Copied Code in the pop-up, and make sure to replace the URL in the copied code with your website's URL.
Now after this in the very next line Enable custom robot header tags.
1- Click home page tags and enable (all and noodp) and then click Save.
2- Click Archive and search page tags and enable (noindex and noodp) and then click Save.
3- Click Post and page tags and enable (all and noodp) and then click Save.
After doing all this stuff, switch to Google Search Console, and click Sitemaps from the left menu.
Paste your Sitemap address there and click on Submit. You can copy the Sitemap address below but make sure to replace the URL with your website URL.
http://yoursite.com/atom.xml?redirect=false&start-index=1&max-results=500
Here you are done with the procedure and your indexing issue has been resolved.