How Search Works

Before you can improve a page for search, you need a clear picture of what a search engine actually does with it. Many SEO myths come from people skipping this step. Google publishes how its Search works, and the basic process has three stages: crawling, indexing and serving results.

In this lesson you will learn each stage, why some pages never appear in Google, what affects which results are shown, and you will build a tiny crawler and index in Python to see the ideas working.

Stage 1: Crawling

The web has no central list of pages, so Google has to discover them. It uses an automated program called Googlebot (a crawler, also called a bot or spider) that downloads pages and follows the links on them to find more pages.

Googlebot finds new URLs in three main ways:

• Links from pages it already knows, on your site or other sites.
• Sitemaps that you submit, which list your important URLs (see Technical SEO).
• Revisits of known pages to check for changes.

Crawlers follow normal HTML links. A link that only works through JavaScript click handlers may not be followed:

<!-- Crawlable: a real link with an href -->
<a href="/courses/spoken-english">Spoken English classes</a>

<!-- Not reliably crawlable: no href, only a click handler -->
<span onclick="openPage('spoken-english')">Spoken English classes</span>

During crawling, Google also renders the page, which means it runs the page's JavaScript much like a browser does, so it can see content that scripts add.

Stage 2: Indexing

After downloading a page, Google tries to understand it: the text, headings, <title>, images, videos and structured data. It stores what it learns in the Google index, a huge database of pages.

During indexing, Google also groups pages that have the same or very similar content and picks one as the canonical, the version most likely to be shown in results. The others are treated as duplicates.

Indexing is not guaranteed. Google states clearly that it does not promise to crawl, index or show every page, even if the page follows its guidelines. Common reasons a page is not indexed:

• The page has a noindex rule.
• The content is thin, duplicated or low quality.
• No links point to the page, so Google rarely finds it.
• The server returned an error when Googlebot visited.

You can check the indexing status of any of your own URLs with the URL Inspection tool in Google Search Console.

Stage 3: Serving results

When someone searches, Google looks through its index for the most relevant, useful pages and decides the order. Many factors go into this, including:

• Meaning of the query: what the person is really looking for, including spelling mistakes and synonyms.
• Relevance: whether the page's content matches the query.
• Quality: signs that the content is helpful, reliable and trustworthy.
• Usability: whether the page works well, for example on a phone.
• Context: the searcher's location, language and device. "Tuition centre near me" in Kochi gives different results from the same search in Jaipur.

The results page is not only ten blue links. Depending on the query, it can include a map with local businesses, images, videos, product listings, AI Overviews and ads. Ads are labelled as sponsored and are sold through Google Ads; Google says it does not accept payment to crawl a site more often or to rank it higher in the unpaid results.

Simulate a crawler

A crawler starts from one page, reads its links, visits those pages, and repeats. This Python program models a small website as a dictionary of pages and the links on each page. It finds every page reachable from the homepage, and the orphan pages that nothing links to.

site = {
    "/": ["/courses", "/about", "/contact"],
    "/courses": ["/courses/spoken-english", "/courses/ielts"],
    "/courses/spoken-english": ["/contact"],
    "/courses/ielts": ["/contact"],
    "/about": ["/"],
    "/contact": [],
    "/courses/interview-skills": ["/contact"],   # nobody links here
}

to_visit = ["/"]
found = set()
while to_visit:
    page = to_visit.pop(0)
    if page in found:
        continue
    found.add(page)
    to_visit.extend(site.get(page, []))

print("Found by following links:", len(found), "pages")
orphans = sorted(set(site) - found)
print("Orphan pages:", orphans)

Found by following links: 6 pages
Orphan pages: ['/courses/interview-skills']

The interview-skills page exists, but no page links to it, so a crawler starting from the homepage never reaches it. Real SEO audits look for exactly this problem. The fix is simple: link to the page from the courses page, and list it in your sitemap.

Simulate an index

An index lets a search engine answer quickly without reading every page again. A simple version is an inverted index: a dictionary from each word to the pages that contain it.

pages = {
    "/courses/spoken-english": "spoken english classes for beginners in lucknow",
    "/courses/ielts": "ielts coaching classes in lucknow weekend batch",
    "/about": "our trainers have taught english for many years",
}

index = {}
for url, text in pages.items():
    for word in set(text.split()):
        index.setdefault(word, set()).add(url)

def search(query):
    results = None
    for word in query.lower().split():
        urls = index.get(word, set())
        results = urls if results is None else results & urls
    return sorted(results or [])

print(search("english classes"))
print(search("classes lucknow"))
print(search("french classes"))

['/courses/spoken-english']
['/courses/ielts', '/courses/spoken-english']
[]

Google's real index is vastly more advanced: it understands meaning, not just matching words, and it ranks results instead of listing them alphabetically. But the core idea holds. If your page does not clearly contain the topic people search for, it is hard for any search engine to match it.

Common mistakes

• Expecting instant results. New pages can take days or weeks to be crawled and indexed.
• Believing "submit to Google" services. There is no paid submission. Use Search Console and a sitemap, both free.
• Hiding important text in images. Search engines read text best. Put key information in real HTML text.
• Orphan pages. Every important page should be reachable through normal links.

Key points

• Google Search works in three stages: crawling, indexing and serving results.
• Googlebot discovers pages mainly through links and sitemaps, and follows normal <a href> links.
• Indexing analyses the page and chooses a canonical among duplicates; indexing is never guaranteed.
• Results depend on query meaning, relevance, quality, usability and the searcher's context.
• Google does not sell higher rankings in its unpaid results; ads are separate and labelled.

site = {
    "/": ["/courses", "/about", "/contact", "/blog"],
    "/courses": ["/courses/spoken-english", "/courses/ielts", "/courses/interview-skills"],
    "/courses/spoken-english": ["/contact"],
    "/courses/ielts": ["/contact"],
    "/about": ["/"],
    "/contact": [],
    "/courses/interview-skills": ["/contact"],
    "/blog": ["/blog/ielts-tips"],
    "/blog/ielts-tips": ["/courses/ielts"],
}
to_visit, found = ["/"], set()
while to_visit:
    page = to_visit.pop(0)
    if page not in found:
        found.add(page)
        to_visit.extend(site.get(page, []))
print("Orphan pages:", sorted(set(site) - found))

Orphan pages: []