I.24 Search Engines

⬆️ Back to: P-I.I. Introduction to Computing

🧭

Page Outline — click a heading to jump. Past-paper questions sit in the last section.

🎯

Exam pattern (2016–2026): 2023 Q2(c) role of crawling, indexing and searching (6).

---

CSS Computer Science (Optional) — Paper-I, Section-A, Topic I, p.61-64. Covers: Search Engines.

Definition

A search engine is a software system designed to search the World Wide Web and return relevant results (e.g., Google, Bing).

Classification of Search Engines

By Underlying Method

Other Classification Axes

• By privacy model: Tracking/personalized (Google, Bing — tailor results using search history/profile) vs. Privacy-focused (DuckDuckGo, Startpage — no tracking or personalization).
• By scope: General-purpose (Google, Bing — search the whole web) vs. Vertical/specialized (Google Scholar for academic papers, dedicated image/video search, e-commerce search engines).
• By regional dominance: Global (Google, Bing) vs. Regional/market-specific (Baidu in China, Yandex in Russia, Naver in South Korea) — a reminder that "the world's most popular search engine" isn't uniformly true in every country.
• Exam angle: distinguishing crawler-based from directory-based is a classic historical question — the early web (mid-1990s) relied heavily on human-edited directories (e.g., the original Yahoo!) before crawler-based indexing (Google's model) became dominant.

History of Search Engines

• 1990 — Archie: created by Alan Emtage at McGill University; the first search tool of any kind, but it indexed only FTP file names (the World Wide Web did not yet exist) — it could not search file content or accept natural-language queries.
• 1993 — World Wide Web Wanderer: built by Matthew Gray at MIT; the first automated web crawler/robot, originally created to measure the size of the growing web rather than to power a search product.
• 1994 — Yahoo!: founded by Jerry Yang and David Filo (Stanford graduate students) as "Jerry and David's Guide to the World Wide Web," renamed Yahoo! in March 1994. Started as a human-edited directory (pages organized by category), not a crawler-based index — Yahoo added its own search function only in 1995.
• 1994 — WebCrawler: created by Brian Pinkerton; the first search engine to index the full text of web pages rather than just their titles/URLs.
• 1994 — Lycos: created by Michael Mauldin at Carnegie Mellon University; one of the first major crawler-based search engines with a large index, and one of the first to rank results partly by links pointing to a page.
• 1995 — AltaVista: launched 15 December 1995 by Digital Equipment Corporation; introduced a fast, multi-threaded crawler (nicknamed "Scooter") and was the first large-scale, searchable full-text database with a simple interface — a major leap in speed and coverage over earlier engines.
• 1996 — Ask Jeeves: founded by Garrett Gruener and David Warthen; distinguished by accepting natural-language questions rather than requiring keyword-style queries.
• 1998 — Google: founded by Larry Page and Sergey Brin at Stanford University; introduced the PageRank algorithm (originally named BackRub), ranking pages by the quantity and quality of links pointing to them — it eventually became, and remains, the dominant global search engine.

Exam angle

The exam sometimes tests which engine was "the first" at something specific — Archie (first search tool, 1990), World Wide Web Wanderer (first web crawler, 1993), WebCrawler (first full-text index, 1994), and Google (link-based PageRank ranking, 1998) are each "first" at a different, precise thing; read the question carefully rather than assuming Google was first at everything.

How They Work — Three-Stage Process

1. Crawling: automated bots ("crawlers"/"spiders") systematically browse the web, following links to discover pages
2. Indexing: the content of discovered pages is analyzed and stored in a massive searchable index/database
3. Ranking: when a user submits a query, the search engine retrieves matching pages from its index and orders ("ranks") them by relevance, using algorithms that weigh factors like keyword match, page authority, and freshness

Major Search Engines

• Google Search — the dominant global search engine.
  • Founded: 1998, by Larry Page and Sergey Brin at Stanford University.
  • Ranking history: original algorithm named BackRub, renamed PageRank (ranks pages partly by inbound links; today one of hundreds of signals).
  • Known for: scale/freshness of crawling and index; integration with other Google services (Maps, Images, News); the "featured snippet" answer box shown above normal results.
  • Typical use-case: general-purpose everyday web search.
  • Advantage: very large index and strong relevance tuning.
• Microsoft Bing — Microsoft's search engine.
  • Launched: 2009, as a rebrand/successor to the earlier MSN Search and Live Search.
  • Default in: Windows and Microsoft Edge.
  • Known for: strong image/video search; a built-in rewards program; supplies results/AI-chat integration used elsewhere in Microsoft products.
  • Typical use-case: default search inside Microsoft-centric environments and devices.
  • Market position: second-most-used engine globally.
• Yahoo Search — one of the earliest major web portals with integrated search.
  • Origin: began as a human-edited directory in 1994; added its own search function in 1995.
  • Current index: results today are largely powered by Bing's index, not an independent Yahoo crawler.
  • Typical use: a legacy user base, mainly as part of the broader Yahoo portal (mail, news, finance).
• DuckDuckGo — a search engine that markets itself on privacy.
  • Founded: 2008, by Gabriel Weinberg.
  • Defining feature: by design, does not track users, log search history, or personalize/filter results based on a user profile.
  • Advantage: no "filter bubble"; stronger privacy.
  • Disadvantage: smaller independent crawl than Google and no personalization, so results can be less tailored for some queries.
• Baidu — the dominant search engine in China.
  • Optimized for: the Chinese language.
  • Operates within: China's domestic internet regulatory environment (where Google is largely inaccessible).
  • Relevance: an example of how search engine dominance varies by country/region rather than being globally uniform.

---

MCQ Traps

• Crawling ≠ Indexing: crawling is discovering pages; indexing is storing/organizing their content for retrieval — these are two separate stages, not one.
• A search engine's results are ranked by algorithm, not by manual curation — relevance signals (keywords, links, freshness) drive the order.
• DuckDuckGo's defining feature is privacy/no tracking, not simply "a different design" — do not confuse it with Bing or Yahoo, which do personalize/track.
• Yahoo Search results are largely powered by Bing, not by Yahoo's own independent index — a detail sometimes tested to check whether students understand market consolidation among engines.
• BackRub, not PageRank, was Google's original algorithm name — Page and Brin renamed it PageRank; a trap for questions asking about Google's early history.
• Directory-based ≠ crawler-based — a human-edited directory (the historical Yahoo Directory) is a different classification axis from an automated crawler-based index (Google); don't treat all pre-Google engines as "crawler-based."
• "First search engine" ≠ "first web search engine" ≠ "first web crawler" — Archie (1990) predates the web itself and only indexed FTP file names; the World Wide Web Wanderer (1993) was the first web crawler; don't collapse these into one "first" fact.
• Yahoo! predates Google by four years but started as a human-edited directory, not a crawler-based search engine — a scenario question about "the first search engine" should not default to Google just because it's dominant today.

Frequently Asked MCQs (Past Papers)

• Q: What is the correct three-stage order in which a search engine processes and returns results? — A: Crawling → Indexing → Ranking.
• Q: What is the name given to the automated programs that browse the web to discover pages for a search engine? — A: Web crawlers (also called spiders or bots).
• Q: In what year was Google founded, and by whom? — A: 1998, by Larry Page and Sergey Brin at Stanford University.
• Q: What was Google's original link-based ranking algorithm called before it became PageRank? — A: BackRub.
• Q: Which search engine markets itself specifically on privacy and does not track users or personalize results? — A: DuckDuckGo.
• Q: Whose search index actually powers Yahoo Search's results today? — A: Microsoft Bing.
• Q: Which search engine dominates in China, where Google is largely inaccessible? — A: Baidu.
• Q: What distinguishes a directory-based search engine from a crawler-based one? — A: A directory-based engine organizes pages into categories using human editors, rather than discovering them automatically via bots.
• Q: What is a metasearch engine? — A: One that submits a single query to multiple other search engines and combines their results (e.g., Dogpile).
• Q: Which search engine is the default in Windows and Microsoft Edge, and when was it launched? — A: Bing, launched in 2009.
• Q: What is generally considered the first search tool, and what did it index? — A: Archie (1990), created by Alan Emtage; it indexed only FTP file names, not web pages.
• Q: What was the first automated web crawler, and who built it? — A: The World Wide Web Wanderer (1993), built by Matthew Gray at MIT.
• Q: In what year was Yahoo! founded, and what was its original format? — A: 1994, founded by Jerry Yang and David Filo; it began as a human-edited directory, not a crawler-based search engine.
• Q: Which search engine was the first to index the full text of web pages rather than just titles/URLs? — A: WebCrawler (1994), created by Brian Pinkerton.
• Q: Which company launched AltaVista, and in what year? — A: Digital Equipment Corporation, in 1995.
• Q: Which early search engine was distinguished by accepting natural-language questions instead of keyword queries? — A: Ask Jeeves (1996).

✅ Self-Test Questions (with Answers)

Q1. Describe the three-stage process by which a search engine returns results.

Ans: Every crawler-based search engine works through three sequential stages: Crawling → Indexing → Ranking. The first two happen continuously in the background, long before any user types a query; only the third happens in response to the search.

1. Crawling (discovery). Automated programs called crawlers, spiders or bots — Google's is Googlebot — systematically browse the web. Starting from a seed list of known URLs, a crawler fetches each page, extracts every hyperlink on it, and adds those links to a queue of pages to visit next, repeating endlessly. This link-following is how new pages are discovered without anyone submitting them. Crawlers respect the robots.txt file, by which a site owner can exclude parts of the site, and sitemap.xml, which advertises pages for crawling.
2. Indexing (storage and organisation). The fetched content is parsed — text, titles, headings, image alt text, metadata — and stored in an enormous inverted index, a data structure that maps each word to the list of documents containing it. This is the same principle as a book's index: rather than scanning every page at query time, the engine looks up the term and instantly retrieves the document list. Without this step a web-scale search could never return results in under a second.
3. Ranking (retrieval and ordering). When a user submits a query, the engine retrieves the matching documents from the index and orders them by relevance using a ranking algorithm weighing hundreds of signals — keyword match and placement, link authority (PageRank), freshness, page quality, mobile-friendliness, page speed, user location and language. The ordered list is returned as the SERP (Search Engine Results Page).

The key exam point: the user's query touches only stage 3. The engine never searches the live web at query time — it searches its own pre-built index, which is why results can appear in milliseconds and why a very newly published page may not yet be findable.

Q2. What is the difference between crawling and indexing?

Ans: Crawling is discovery; indexing is storage and organisation. They are two distinct stages and conflating them is a standard MCQ trap. Crawling is the act of a bot finding and fetching a page — following a hyperlink to a URL and downloading its HTML. Its output is raw retrieved content and a list of further URLs to visit. Indexing is what happens to that content afterwards: it is parsed, analysed and stored in the searchable index (the inverted index mapping words to documents), so that it can later be retrieved by keyword.

The relationship is one-directional but not automatic: a page must be crawled before it can be indexed, but being crawled does not guarantee being indexed. A search engine may crawl a page and then decline to index it because it is duplicate content, judged low quality, marked with a noindex directive, or blocked from indexing by the site owner. Conversely, a page blocked in robots.txt is not crawled at all, though it may still appear in results as a bare URL if other sites link to it.

Q3. Name one factor that influences how a search engine ranks results.

Ans: One of the most important factors is link authority — the quantity and quality of inbound links (backlinks) pointing to a page. This is the principle behind Google's PageRank algorithm, devised by Larry Page and Sergey Brin in 1998 (originally named BackRub). Its insight was to treat a hyperlink as a vote of confidence: a page linked to by many other pages is probably valuable, and a link from an already-authoritative page counts for far more than a link from an obscure one. This link-based model is what made Google dramatically more relevant than the keyword-frequency engines that preceded it, and it remains one signal among hundreds today.

Other major ranking factors worth naming:

• Keyword relevance and placement — whether query terms appear in the title, headings, URL and body, and how the engine judges the page's topical match to the query's intent.
• Content quality and depth — original, substantive, well-structured content over thin or duplicated pages.
• Freshness — recency matters greatly for news and fast-changing topics, far less for reference material.
• User experience signals — page load speed, mobile-friendliness, HTTPS security, absence of intrusive interstitials.
• Personalisation and context — the user's location, language, device and search history (on engines that personalise).
• Engagement behaviour — click-through rate and whether users quickly return to the results page.

The defining point is that ordering is produced algorithmically, not by manual human curation — which is what distinguishes a crawler-based search engine from a human-edited directory.

Q4. Compare Google and DuckDuckGo in terms of their approach to user privacy.

Ans: The two engines embody opposite philosophies. Google, founded in 1998, operates an advertising-funded, personalised model: it stores search history, associates queries with a signed-in Google account, and uses cookies, location, device and cross-service data (YouTube, Maps, Chrome, Android) to build a user profile. That profile both targets advertisements and personalises results, so two people searching the same term may see different pages in different orders. DuckDuckGo, founded in 2008 by Gabriel Weinberg, is built on the opposite premise: it does not store search history, does not create user profiles, and does not personalise results by user. Every user searching a given term sees the same results. It still shows advertisements, but they are contextual — matched to the current query alone, not to any stored profile.

The trade-off is genuine, not one-sided. Google's data gives it superior personalisation, local results and query understanding; DuckDuckGo's refusal to profile avoids the filter bubble — the phenomenon, named by Eli Pariser, in which personalisation progressively narrows what a user is shown to what the algorithm predicts they already agree with — at the cost of less tailored results and a smaller independent index.

CSS angle: This is directly relevant to Pakistan's data-protection debate. The long-pending Personal Data Protection Bill, and the existing framework of PECA 2016 and PTA regulation, engage exactly these questions of consent, data minimisation and cross-border transfer of citizens' personal data by foreign platforms.

Q5. Why might a search engine be dominant in one country but not in another? Give an example.

Ans: Search-engine dominance is not uniform worldwide, because market share is shaped by factors far beyond raw technical quality:

• Government regulation and market access. A state may block or restrict a foreign engine, or impose content requirements a foreign firm will not accept, leaving the field to a domestic competitor.
• Language and script optimisation. An engine tuned for a national language — its morphology, script and idioms — returns better results than a generic one. This matters greatly for Chinese, Russian, Korean, Arabic and Urdu.
• Local content and local knowledge. Dominance depends on indexing the domestic web deeply: local news, businesses, maps, government sites and payment or booking services.
• First-mover advantage and habit. The engine people learned to use becomes the default verb for searching.
• Bundling and distribution. Pre-installation on devices, browsers and operating systems shapes defaults powerfully — Bing is the default in Windows and Microsoft Edge.
• Cultural and commercial integration. An engine woven into local ecosystems of payments, maps, ride-hailing and messaging is far harder to displace.

Example: Baidu dominates in China, where Google is largely inaccessible following Google's 2010 withdrawal from the mainland market over censorship and cyberattack concerns. Baidu is optimised for Chinese-language search and operates within China's domestic regulatory environment. Two further examples: Yandex leads in Russia, largely because it handles Russian morphology (a highly inflected language) better than Google historically did, and Naver leads in South Korea by indexing curated in-house Korean content such as its own blogs, Q&A and news.

The lesson for an exam answer is that the claim "Google is the world's most popular search engine" is a global aggregate, not a universal fact — in several major markets a domestic engine leads, for regulatory, linguistic and cultural reasons rather than purely technical ones. In Pakistan, by contrast, Google is overwhelmingly dominant, since there is no significant domestic search alternative, though Urdu-language and local-content indexing remains comparatively thin — a recognised gap in the Digital Pakistan agenda.

📜 Past-Paper Questions & MCQs (CSS 2016–2026)

2023 · Paper-I · Q2(c) · 6 marks

Q. Discuss the role of web crawling, indexing and searching operations for a search engine.

Quick revision — key points

• Crawling discovers and fetches web documents
• Indexing extracts terms and constructs an inverted index
• Searching processes a query and retrieves matching pages
• Ranking orders results by relevance and other signals
• Show the sequence with an example

Exam-ready answer (6 marks)

Introduction (1 mark)

A search engine is an information-retrieval system that helps users find relevant web pages in response to their queries. Its core workflow consists of web crawling, indexing and searching. These are connected operations, but they perform different tasks: crawling discovers information, indexing organises it, and searching retrieves and ranks results.

1. Web crawling — discovering web pages (1.5 marks)

Web crawling is the automated process of visiting web pages using programs called crawlers, spiders or bots (for example, Googlebot).

• Discovery: A crawler begins with known URLs, visits accessible pages and follows hyperlinks and sitemaps to discover additional URLs.
• Fetching: It requests page content from web servers, commonly through HTTP/HTTPS, and can schedule repeat visits to detect updates.
• Crawl management: The engine prioritises useful URLs, avoids repeatedly fetching duplicates and considers robots.txt directives and server capacity.

Example: When a university publishes a new admissions notice and links it from its homepage, a crawler may discover and fetch the notice. Crawling alone does not guarantee that the notice will appear in search results.

2. Indexing — organising discovered information (1.5 marks)

After fetching a page, the search engine analyses its content to build a searchable index, rather than scanning the entire live web for every query.

• Content processing: It extracts usable text, headings, titles, links and metadata; it may also analyse images and structured data.
• Tokenisation and normalisation: Text is divided into searchable terms, with appropriate handling of spelling, language and word forms.
• Inverted index: An important data structure maps each term to the documents containing it, often alongside positions and other information used for ranking.
• Quality control: Duplicate or canonical pages may be consolidated, and pages that cannot or should not be indexed may be excluded.

Simple inverted-index example:

• admissions → Page A, Page C
• scholarship → Page B, Page C

If a user searches for admissions scholarship, the engine can quickly identify Page C as containing both terms. Not every crawled page is necessarily indexed.

3. Searching — retrieving and ranking results (1.5 marks)

When a user submits a query, the search engine uses its index to find potentially relevant documents and display an ordered results page.

1. Query interpretation: It processes the search words, language, spelling and likely intent.
2. Retrieval: It looks up relevant terms and concepts in its indexes and selects candidate pages.
3. Ranking: It orders candidates using signals such as textual relevance, content quality, usefulness, link-related signals and freshness. Location may also matter for local searches.
4. Presentation: It shows titles, links and snippets on the search engine results page (SERP), sometimes with specialised answers.

Example: For the query NUST scholarship admission, the engine retrieves relevant indexed pages and aims to rank official, current admissions or scholarship pages above unrelated mentions.

4. Integration and conclusion (0.5 mark)

Workflow: Web pages → Crawling → Content processing and indexing → User query → Retrieval and ranking → Search results.

These processes work together continuously: crawling keeps the source collection current, indexing makes the information searchable, and searching helps the user find the most relevant answer efficiently. A page may exist on the web but remain absent from results if it has not been discovered, is not eligible for indexing, or is not selected among the relevant results.

Examiner's guidance (6 marks): Distinguish all three stages clearly, explain the inverted index, show a short workflow and include one real-world example. Avoid claiming that robots.txt is an absolute access-control system or that every crawled page must be indexed.