Information Gathering

Every security assessment starts with questions. Which machines exist? Which software do they run? Which pages and APIs does the web app have that are not linked from the menu? This first phase is called reconnaissance or information gathering, and good testers spend a lot of time on it, because you cannot test what you have not found.

In this lesson you will learn the difference between passive and active recon, the sources testers use, and how to gather information about the practice apps in your own lab. You will also write Python helpers that pull useful details out of robots.txt, HTML and JavaScript.

Passive and active reconnaissance

Even passive recon belongs inside an authorised engagement. Collecting information about real people (their phone numbers, addresses, family details) is also covered by privacy law, including the DPDP Act, so professional testers gather only what the engagement needs and protect it carefully.

Passive sources

• DNS records: dig yourdomain.in ANY or separate queries for A, MX, TXT and NS records show servers, mail providers and email security policies.
• WHOIS: whois yourdomain.in shows the registrar and dates. Many registrants now hide personal details.
• Certificate Transparency (CT) logs: every publicly trusted TLS certificate is recorded in public logs, so searching them (for example on crt.sh) for your own domain reveals subdomains such as staging. or vpn. that you may have forgotten. Defenders should run this search on their own domains regularly.
• The organisation's own pages: careers pages often list technologies ("experience with Laravel and AWS"), which tells a tester what to look for.

dig example.com MX +short
dig example.com TXT +short
whois example.com

Active recon in the lab

Start by finding live hosts in the lab network. You will learn Nmap properly in the next lesson; this "ping scan" only checks which addresses respond:

nmap -sn 192.168.56.0/24

Next, identify the web technology. curl -I shows response headers, and whatweb (included in Kali) recognises frameworks and versions:

curl -I http://192.168.56.101:4280/login.php
whatweb http://192.168.56.101:4280

Then explore the application the way a curious user would, with your browser's developer tools open:

• robots.txt tells search engines what not to index. It is public, so it often points straight at interesting paths.
• Page source and HTML comments sometimes contain notes developers forgot to remove.
• JavaScript files in modern single-page apps such as Juice Shop list the API endpoints the front end calls.

This helper reads a robots.txt file and lists the paths it hides:

robots = """User-agent: *
Disallow: /admin/
Disallow: /backup/
Allow: /public/
# old staging copy, remove before launch
Disallow: /staging-2024/
Sitemap: http://192.168.56.101:4280/sitemap.xml"""

hidden, sitemap = [], None
for line in robots.splitlines():
    line = line.strip()
    if line.lower().startswith("disallow:"):
        hidden.append(line.split(":", 1)[1].strip())
    elif line.lower().startswith("sitemap:"):
        sitemap = line.split(":", 1)[1].strip()
    elif line.startswith("#"):
        print("Comment found:", line)

print("Paths to check:", hidden)
print("Sitemap:", sitemap)

Comment found: # old staging copy, remove before launch
Paths to check: ['/admin/', '/backup/', '/staging-2024/']
Sitemap: http://192.168.56.101:4280/sitemap.xml

robots.txt is a polite request, not a security control. Anything it lists can be visited by anyone, so it must never be used to "hide" sensitive pages.

Pulling details out of HTML

Python's built-in html.parser can collect every link, form and comment from a page you saved from your lab app:

from html.parser import HTMLParser

page = """<html><body>
<!-- dev note: remove test account demo/demo123 -->
<a href="/products">Products</a>
<a href="/account/orders">My orders</a>
<form action="/search" method="get"><input name="q"></form>
<form action="/api/feedback" method="post"><input name="comment"><input type="hidden" name="userId" value="7"></form>
</body></html>"""

class Recon(HTMLParser):
    def __init__(self):
        super().__init__()
        self.links, self.forms, self.inputs, self.comments = [], [], [], []
    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "a":
            self.links.append(a.get("href"))
        elif tag == "form":
            self.forms.append((a.get("method", "get").upper(), a.get("action")))
        elif tag == "input":
            self.inputs.append((a.get("name"), a.get("type", "text")))
    def handle_comment(self, data):
        self.comments.append(data.strip())

r = Recon()
r.feed(page)
print("Links:   ", r.links)
print("Forms:   ", r.forms)
print("Inputs:  ", r.inputs)
print("Comments:", r.comments)

Links:    ['/products', '/account/orders']
Forms:    [('GET', '/search'), ('POST', '/api/feedback')]
Inputs:   [('q', 'text'), ('comment', 'text'), ('userId', 'hidden')]
Comments: ['dev note: remove test account demo/demo123']

Each form and input is a place where the app accepts data, so each is something to test later. The hidden userId field is worth a note: if the server trusts it, a user could change it and act as someone else (you will test this in Access Control, CSRF & SSRF). The comment leaks test credentials, a real and common finding.

Finding API endpoints in JavaScript

Single-page apps download large JavaScript bundles that contain the paths of every API the app uses. A regular expression can list them:

import re

bundle = '''fetch("/rest/products/search?q="+e);this.http.get(this.host+"/api/Users/"+id);
const t="/rest/admin/application-configuration";this.http.post("/api/Feedbacks/",data);
fetch("/rest/user/whoami")'''

endpoints = sorted(set(re.findall(r'"(/(?:api|rest)/[A-Za-z0-9_/-]*)', bundle)))
for path in endpoints:
    flag = "  <- admin path, check access control" if "admin" in path else ""
    print(path + flag)

/api/Feedbacks/
/api/Users/
/rest/admin/application-configuration  <- admin path, check access control
/rest/products/search
/rest/user/whoami

An admin endpoint shipped to every visitor's browser is not a vulnerability by itself, but it tells you exactly which access-control check to test.

Directory discovery

Some pages are not linked anywhere. Directory discovery tools such as gobuster and ffuf request thousands of common names from a wordlist and report the ones that exist. This creates a lot of traffic and log entries, so run it only against your lab:

gobuster dir -u http://192.168.56.101:4280 -w /usr/share/wordlists/dirb/common.txt -t 10

-t 10 limits it to 10 parallel requests. On a real engagement you agree the speed with the client, because aggressive discovery can slow down a production site.

Keeping recon notes

Write everything down as you go: host, port, technology and version, interesting paths, forms, parameters and anything odd. Good notes turn into the "scope and discovery" section of your final report, and they stop you testing the same thing twice.

Common mistakes

• Treating "public" as "permitted". Public information about an organisation is still gathered only inside an authorised engagement.
• Collecting personal data you do not need. Gather only what the test requires and protect it.
• Running aggressive tools before reading the app. Ten minutes of clicking through the app with developer tools open often finds more than an hour of brute-force discovery.
• Not saving evidence. Save requests, responses and screenshots with timestamps as you find things.

Key points

• Reconnaissance is the first phase of any assessment; you cannot test what you have not found.
• Passive recon uses public sources such as DNS, WHOIS and Certificate Transparency logs; active recon sends traffic to the target.
• Both need authorisation; active steps in this course target only your lab.
• robots.txt, HTML comments, hidden fields and JavaScript bundles often reveal paths and parameters.
• Directory discovery is noisy; limit its speed and use it only in scope.

from html.parser import HTMLParser

page = '<script src="/main.js"></script><form><input type="hidden" name="orderId" value="1042"></form>'

class Recon(HTMLParser):
    def __init__(self):
        super().__init__()
        self.scripts = []
    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "script" and a.get("src"):
            self.scripts.append(a["src"])
        if tag == "input" and a.get("type") == "hidden" and a.get("name", "").lower().endswith("id"):
            print("Warning: hidden ID field", a["name"], "=", a.get("value"))

r = Recon()
r.feed(page)
print("Scripts:", r.scripts)