SpiderFoot and Recon-ng in Practice: Automate OSINT Gathering Without Drowning in Noise

TL;DR: Automate OSINT Without the Noise

Yes, you can automate OSINT without drowning in junk—but only if you configure SpiderFoot and Recon-ng deliberately: scope every target tightly, enable only the modules that matter for your engagement, add API keys for high-value sources, and filter output by risk and confidence before it ever reaches a report. Out-of-the-box, both tools will happily bury you in thousands of low-signal findings. Tuned correctly, they deliver a prioritized attack-surface picture in hours, not weeks.

What SpiderFoot and Recon-ng Actually Do (and Where They Differ)

Traditional reconnaissance has a well-established rhythm: pick a target, enumerate subdomains, harvest emails, check breach dumps, map exposed services. SpiderFoot and Recon-ng automate that rhythm—but they automate it in fundamentally different ways, and choosing the wrong tool for the job is the first source of noise.

SpiderFoot (maintained by Intel471, which acquired it in 2022) is breadth-first automation. You give it a target—an IP, domain, email, or Bitcoin address—and it fans out across 200+ modules, correlating everything into a relationship graph. It’s designed to answer “what does this target’s footprint look like?” with minimal operator input. That power is also its problem: run it unscoped and unfiltered, and you’ll get 40,000 data elements for a mid-size company, most of them irrelevant.

Recon-ng, developed by Tim Tomes (LaNMaSteR53) and now maintained under the Recon-ng project on GitHub, is a modular, Metasploit-style framework. Workspaces, marketplace-installed modules, and a database backend make it a targeted instrument. You decide which recon domain—hosts, contacts, credentials, profile information—and which data source serves it. Where SpiderFoot is a firehose, Recon-ng is a scalpel.

The practical distinction: use SpiderFoot for broad discovery and relationship mapping; use Recon-ng for focused, repeatable collection you can defend line-by-line in a report.

Lab Setup and Legal Ground Rules

Before you type a single command: automated OSINT is only legal against assets you own or are explicitly authorized to test. Passive sources still impose terms-of-service limits—scraping Shodan or breach data against their usage policies can violate contracts even when the data itself is “public.” The CISA Known Exploited Vulnerabilities catalog and OWASP’s testing guides exist partly because the industry learned the hard way that unauthorized enumeration is not a gray area. For authoritative framing, review the OWASP Web Security Testing Guide (https://owasp.org/www-project-web-security-testing-guide/) and your local computer-misuse statutes before scoping an engagement. When practicing, stick to designated targets: HackerOne-sanctioned programs, DVWA, vulnweb.com, or your own registered domains.

Both tools ship in Kali Linux. If you’re starting clean:

• Recon-ng (Docker or pip): git clone https://github.com/lanmaster53/recon-ng.git && cd recon-ng && pip install -r REQUIREMENTS && ./recon-ng
• SpiderFoot (Docker, recommended): docker run -p 5001:5001 secureheritage/spiderfoot, then browse to http://localhost:5001. Alternatively: git clone https://github.com/smicallef/spiderfoot.git && pip3 install -r spiderfoot/requirements.txt && python3 spiderfoot/sf.py -l 127.0.0.1:5001

Use stable releases—Recon-ng’s versioning (currently the 5.x line) changes module APIs between major versions, and stale tutorials will hand you broken syntax.

Configuring Recon-ng: Workspaces, Marketplace, and Keys

Recon-ng’s workspace model is your first noise filter. Everything—hosts, contacts, credentials—scopes to a workspace, so findings never cross-contaminate between clients.

workspaces create acme_test
keys add shodan_api <YOUR_KEY>
marketplace search
marketplace install recon/domains-hosts/hackertarget
marketplace install recon/domains-hosts/shodan_hostname
marketplace install recon/contacts-accumule/...

Module selection principle: install for the data you need, not the modules that exist. A domain-scope engagement needs host-enumeration modules (hackertarget, shodan_hostname, crtsh) and contact modules. You do not need profile modules or geolocation modules for that engagement—every enabled source you don’t need is another source of duplicates and false positives later.

Free-tier keys cover the basics; a Shodan API key (paid tiers especially) unlocks most of the high-value infrastructure data. Check key status with keys list before every run—dead keys fail silently or produce empty result sets that look like clean recon.

Running a Scoped Recon-ng Workflow on a Test Domain

Against a test domain (we’ll use example.com—swap in your own authorized target):

db insert domains   # enter example.com
modules load recon/domains-hosts/hackertarget
options set SOURCE example.com
run

Sample output:

[*] Category: ...
[-] -------------------
[*] 4 total hosts found.
example.com
www.example.com
mail.example.com
dev.example.com

Chain additional modules against the stored hosts:

modules load recon/hosts-hosts/shodan_host
run
modules load recon/contacts-accumule/...
run

Everything lands in the workspace database—query it with query select * from hosts or pivot with the hosts, contacts, and credentials tables. That structure is precisely why Recon-ng output is so much easier to report on: it’s already normalized.

Configuring SpiderFoot: Scopes, Modules, and API Keys

SpiderFoot’s web UI makes it dangerously easy to just paste a target and hit “Run by use case.” Don’t. Open Settings first and do three things:

1. Set the target allowlist under Scan Configuration → Target. Add your authorized domains to “Approved domains” and—critically—use the denylist to exclude adjacent domains that share your target’s naming but aren’t in scope (shared hosting, parked lookalikes). Scope leaks are the number-one legal risk in automated OSINT.
2. Disable module sets you don’t need. SpiderFoot groups modules by use case (Footprint, Integrity, Leak, Research). For an external attack-surface assessment, disable the cryptocurrency, dark-web-heavy, and social-media modules unless the engagement calls for them. Each disabled module is noise you’ll never have to filter.
3. Add API keys under Settings → API Keys: Shodan, VirusTotal, SecurityTrails, Have I Been Pwned. The free modules work, but the correlation quality improves dramatically with 3–5 quality sources feeding the graph.

Tuning Down the Noise: Filters, Correlation, and False Positives

Even scoped, a SpiderFoot run on a real domain produces thousands of events. Cut the junk in layers:

• Scope enforcement first. Verify the denylist before launch. One misconfigured regex and SpiderFoot will enumerate an unauthorized third-party CDN for twenty minutes.
• Confidence and visibility levels. SpiderFoot tags every data element with a confidence score (0–100) and a risk rating (0–5). In results, filter to confidence ≥ 50 and review risk 3+ items first. Low-confidence, zero-risk events (DNS churn, CDN endpoints) belong in raw logs, not reports.
• Deduplicate before reporting. SpiderFoot’s correlation rules—its “correlation” result type—already merge duplicate findings across modules. Prefer a single correlation result over the ten raw events behind it.
• Human-validate the scary stuff. An exposed-admin-panel flag from one source is a hypothesis; confirmed by Shodan service data and a manual HTTP check, it’s a finding. Recon-ng is the right tool for that validation step.

False positives in OSINT cluster around three patterns: expired DNS records still resolving, subdomains wildcard-matched but parked, and name collisions in breach data. All three are cheap to verify manually—so build that verification into the workflow instead of letting unverified flags into your report.

Cross-Tool Workflow: When to Use Which

The tools aren’t competitors—they’re stages:

• SpiderFoot: initial broad sweep of an authorized domain, producing an asset inventory and risk-flagged anomalies.
• Recon-ng: targeted validation and enrichment—enumerate the specific hosts SpiderFoot flagged, pull contacts, and confirm exposures.
• Pipe between them: export SpiderFoot’s discovered subdomains as CSV, then feed them back into Recon-ng via db insert hosts or a bulk import so your validated dataset stays in one queryable place.

Turning Raw Findings into a Report

SpiderFoot exports HTML, CSV, and GEXF directly from the scan results page; Recon-ng produces CSV/HTML reports via reporting modules (e.g., reporting/html). Neither export is a report—they’re raw material. Structure the deliverable in three tiers:

1. Critical / Act now: confirmed exposures—leaked credentials, exposed admin interfaces, vulnerable services. Each finding gets evidence, source, timestamp, and remediation.
2. Monitor: unverified flags and low-confidence anomalies worth periodic re-checking.
3. Context: the attack-surface inventory itself—subdomain counts, technology footprint, contact surface—framing the findings above.

Every critical claim should trace to a primary source. “SpiderFoot said” is not evidence; “Shodan banner data captured 2024-XX-XX, verified manually” is.

Common Mistakes and Troubleshooting

• Rate-limit lockouts. Run high-volume modules overnight or reduce concurrency in SpiderFoot’s settings; Shodan free tiers throttle hard.
• Silent key failures. keys list in Recon-ng and SpiderFoot’s API settings page before every engagement; empty results usually mean a dead key, not clean recon.
• Scope leaks. Third-party infrastructure (CDNs, SaaS, shared hosting) shares naming with your target. Denylist aggressively and spot-check resolved IPs against in-scope netblocks.
• Module overload. If a SpiderFoot scan takes four hours and yields 50,000 events, you enabled everything. Fewer, higher-quality sources beat maximum coverage.
• Stale installs. Both projects evolve; a tutorial’s syntax from two years ago may not survive the current release. Pull from the official GitHub repos, not mirrors.

Key Takeaways and Next Steps

SpiderFoot and Recon-ng reward configuration discipline: scope everything, enable selectively, filter by confidence and risk, and validate before you report. Build the habit on safe targets—your own registered domains, DVWA, or a CTF range—so your first client engagement isn’t also your first tuning exercise. Once recon automation is producing clean signal, the natural next step is turning that asset inventory into continuous attack-surface monitoring.

Frequently Asked Questions

Is SpiderFoot or Recon-ng better for beginners?

Recon-ng, if you’re comfortable with a command-driven workflow—it forces you to understand each data source you enable. SpiderFoot’s web UI is friendlier for broad automated sweeps, but its ease of use makes it easier to generate noise you don’t yet know how to filter.

Is automated OSINT scanning legal?

Only against assets you own or are explicitly authorized to test. Passive collection still carries terms-of-service limits on the data sources themselves—scraper restrictions apply even to “public” data. Get written authorization and read each API provider’s usage policy.

Do I need API keys for SpiderFoot and Recon-ng?

Both work with free modules out of the box, but keys for Shodan, VirusTotal, SecurityTrails, and Have I Been Pwned unlock most of the high-value data sources. Even a single paid Shodan tier meaningfully improves infrastructure findings.

How do I stop SpiderFoot from flooding me with irrelevant results?

Enforce allowlist/denylist scope settings, disable low-value module sets, and filter output by risk score and confidence level before anything reaches a report. Validate risk-3+ flags manually.

Can SpiderFoot and Recon-ng be used together?

Yes—and they’re better together. Use SpiderFoot for broad discovery and relationship mapping, Recon-ng for targeted validation and enrichment, then merge findings into a single risk-tiered report.