Understanding How Google Works: Key Insights into Search Engine Functionality
The first step in learning SEO is to learn how search engines like Google actually work. So in this lesson we’ll focus on three stages or three functions of google. These stages include crawling, indexing, and serving, which I’ve explained below.
You’ll discover the importance of this lesson later on after you finish the course. When you open your search console, you’ll see this section, which shows how many of your pages are crawled or indexed. That day you won’t be able to understand your search console’s data if you don’t know what indexing or crawling is.
So this lesson will be all about these three important functions of Google.
What happens at Google at a glance:
At its core, Google’s primary function is to help users find information. Well, that was kind of given. But how does Google do that? Google has a huge set of computers and a program that is called Googlebot (also known as a crawler, robot, bot, or spider).
These bots crawl all over the internet looking for links to fetch. Then, all these links will be analyzed and when the user enter a query into the search box, google brings up a set of results that match that query based on many, many, many, many factors. Now let cover each of these stages im more depth.
Google’s stages:
Crawling:
as I told you, Google uses automated programs known as “crawlers” or “spiders” to scan the internet. These crawler’s function is to search the web and find new URLs. Simply because there isn’t a central registry of all web pages.
So these crawlers crawl and follow links from one page to another to add them to Google’s list of known pages. They use an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site (we call this crawl budget).
During this process, google runs any JavaScript it finds (we call this rendering). Rendering is important because websites often rely on JavaScript to bring content to the page, and without rendering Google might not see that content. So Google renders the page to have it ready for analysis for the next stage.
Apart from the crawling budget that I mentioned above, other factors can prevent Google bots from crawling, like when the site owners disable Google bots. Wondering why one might want to do that? It can have multiple reasons.
For instance, sometimes we do this because were redesigning our website, and we don’t want Google to see all the mess, or when we make pages only for our user’s convenience but we don’t want them to rank, like the registration pages.
Indexing:
Once a page is crawled, it is indexed. This means that the information collected is organized and stored in a massive database.
Google’s index is comparable to a library’s catalog, where each web page is stored with its relevant details, allowing quick retrieval when needed. Indexing involves analyzing various elements of a page, such as keywords, links, and multimedia content.
Ranking (or serving):
When a user enters a search query, Google sifts through its index to find the most relevant results. The ranking process involves a complex algorithm that considers hundreds of factors, known as ranking signals.
These signals can include keyword relevance, site authority, user engagement, page load speed, and mobile-friendliness, among others. The goal is to provide users with the most pertinent results at the top of the search results page.
