21 Jun Crawl Optimization
Ever heard of a crawl budget? You might be surprised to learn that Google does not have the time and resources to search every single page on the web. Crawl budget represents the set amount of time that search engines will spend going through pages on your site and the number of pages they will crawl. This can have an impact on your visibility to search engines if crawlers have trouble navigating your site.
Your site’s crawl budget is roughly based on page rank, meaning the pages that have the most links and activity are naturally going to have the most crawl budget given to them. If your website exceeds your crawl budget, it’s possible that key pages may not be crawled and indexed.
In order to avoid this from happening, you need to make your site as easy for crawlers to process as possible by keeping crawl optimization in mind and addressing the areas where crawlers are most likely to get stuck.
Here are six quick methods for improving the crawlability of your site.
1. The XML Sitemap
All URLs listed in the sitemap should be the most up-to-date, canonical versions. Having crawlers dig through old and outdated links will take up space in your crawl budget. Keeping your XML sitemap updated when changes are made is a simple and easy way to take some of the pressure off crawlers.
In order for crawlers to understand the relationship between pages in series, rel=prev/next tags should be used on each page. If you are unfamiliar with these tags, basically they send signals to Google indicating that pages are part of a series, so that crawlers know which page to index. When you use these tags crawlers will not waste time crawling deeper, but will just send users to the first page in the series. Using the Robots.txt file to block off a certain section in a paginated series is also an option.
3. Internal Site Search
Using your Robots.txt file to block internal site search from being crawled is an excellent way to save crawlers’ time. Crawlers can waste a significant amount of time going through individual search result pages on your site. Having these individual searches indexed often provides no value, therefore time can be better spent elsewhere.
4. Internal Redirects
Making sure all internal links are up-to-date is important for obtaining the highest possible rank for your pages. Every redirect that search engines have to follow create more work for the crawlers, and also saps a slight amount of SEO value from that page. Screaming Frog or other similar tools are an easy way to identify redirects across your site.
It is a common misconception that applying a canonical tag prevents a page from being crawled by Google. Using a canonical tag prevents your page from being indexed, but pages with canonical tags are still being crawled, and eating up your budget in the process. Therefore, be conscious of where you are using canonical tags versus blocking the page in the robot.txt file
6. Site Speed
Last, but not least, is site speed. One of the most effective measures of how long a crawler is able to spend on your site is how quickly it is able to pass through your pages. There are a variety of ways to increase speed on your site, including using a CDN, image compression, and browser caching. One of the best tools in identifying where to start when it comes to site speed it the PageSpeed Insights tool in Google’s Search Console.
Don’t leave all the hard work for the crawlers! Always remember, that while it may be easy for a human to understand your website, crawlers are a different story, since they are essentially robots. If you keep all these factors in mind, Google will be able to crawl your website with ease.
However, simply getting your entire website crawled and indexed is no guarantee that your pages will automatically start ranking. For more SEO and PPC tips, visit related articles in the sidebar.