SpiderFoot is an open source intelligence automation tool. Its goal is to automate the process of gathering intelligence about a given target.
Purpose
There are three main areas where SpiderFoot can be useful:
- If you are a pen-tester, SpiderFoot will automate the reconnaisance stage of the test, giving you a rich set of data to help you pin-point areas of focus for the test.
- Understand what your network/organisation is openly exposing to the outside world. Such information in the wrong hands could be a significant risk.
- SpiderFoot can also be used to gather threat intelligence about suspected malicious IPs you might be seeing in your logs or have obtained via threat intelligence data feeds.
Features
- Utilises a shedload of data sources; over 40 so far and counting, including SHODAN, RIPE, Whois, PasteBin, Google, SANS and more.
- Designed for maximum data extraction; every piece of data is passed on to modules that may be interested, so that they can extract valuable information. No piece of discovered data is saved from analysis.
- Runs on Linux and Windows. And fully open-source so you can fork it on GitHub and do whatever you want with it.
- Visualisations. Built-in JavaScript-based visualisations or export to GEXF/CSV for use in other tools, like Gephi for instance.
- Web-based UI. No cumbersome CLI or Java to mess with. Easy to use, easy to navigate. Take a look through the gallery for screenshots.
- Highly configurable. Almost every module is configurable so you can define the level of intrusiveness and functionality.
- Modular. Each major piece of functionality is a module, written in Python. Feel free to write your own and submit them to be incorporated!
- SQLite back-end. All scan results are stored in a local SQLite database, so you can play with your data to your heart’s content.
- Simultaneous scans. Each footprint scan runs as its own thread, so you can perform footprinting of many different targets simultaneously.
- So much more.. check out the documentation for more information.
Data Sources
This is an ever-growing list of data sources SpiderFoot uses to gather intelligence about your target. A few require API keys but they are freely available.
| Source | Location | Notes |
|---|---|---|
| abuse.ch | http://www.abuse.ch | Various malware trackers. |
| AdBlock | https://easylist-downloads.adblockplus.org/easylist.txt | AdBlock pattern matches |
| AlienVault | https://reputation.alienvault.com | AlienVault’s IP reputation database. |
| Autoshun.org | http://www.autoshun.org | Blacklists. |
| AVG Site Safety Report | http://www.avgthreatlabas.com | Site safety checker. |
| Bing | http://www.bing.com | Scraping but future version to also use API. |
| Blocklist.de | http://lists.blocklist.de | Blacklists. |
| Checkusernames.com | http://www.checkusernames.com | Look up username availability on popular sites. |
| DNS | Your configured DNS server. | Defaults to your local DNS but can be configured to whatever IP address you supply SpiderFoot. |
| DomainTools | http://www.domaintools.com | |
| DroneBL | http://www.dronebl.org | |
| DuckDuckGo | http://www.duckduckgo.com | |
| http://www.facebook.com | Scraping but future version to also use API. | |
| FreeGeoIP | http://freegeoip.net | |
| Github | http://www.github.com | |
| http://www.google.com | Scraping but future version to also use API. | |
| Google+ | http://plus.google.com | Scraping but future version to also use API. |
| Google Safe Browsing | http://www.google.com/safebrowsing | Site safety checker. |
| IPCat | https://raw.githubusercontent.com/client9/ipcat/master/datacenters.csv | IP Categorisation. |
| http://www.linkedin.com | Scraping but future version to also use API. | |
| malc0de.com | http://malc0de.com | Blacklists. |
| malwaredomainlist.com | http://www.malwaredomainlist.com | Blacklists. |
| malwaredomains.com | http://www.malwaredomains.com | Blacklists. |
| McAfee SiteAdvisor | http://www.siteadvisor.com | Site safety checker. |
| NameDroppers | http://www.namedroppers.org | |
| Notepad.cc | http://www.notepad.cc | |
| Nothink.org | http://www.nothink.org | Blacklists. |
| Onion.City | http://onion.city | Search engine for the dark web. |
| OpenBL | http://www.openbl.org | Blacklists. |
| PasteBin | http://www.pastebin.com | Achieved through Google scraping. |
| Pastie | http://www.pastie.org | |
| PGP Servers | http://pgp.mit.edu/pks/ | PGP public keys. |
| PhishTank | http://www.phishtank.org | Identified phishing sites. |
| Project Honeypot | http://www.projecthoneypot.org | Blacklists. API key needed. |
| PunkSPIDER | http://www.punkspider.org | |
| RIPE/ARIN | http://stat.ripe.net/ | |
| Robtex | http://www.robtex.com | |
| SANS ISC | http://isc.sans.edu | Internet Storm Center IP reputation database. |
| SHODAN | http://www.shodanhq.com | API key needed. |
| SORBS | http://www.sorbs.net | Blacklists. |
| SpamHaus | http://www.spamhaus.org | Blacklists. |
| ThreatExpert | http://www.threatexpert.com | Blacklists. |
| TOR Node List | http://torstatus.blutmagie.de | |
| TotalHash.com | http://www.totalhash.com | Domains/IPs used by malware. |
| UCEPROTECT | http://www.uceprotect.net | Blacklists. |
| VirusTotal | http://www.virustotal.com | Domains/IPs used by malware. API key needed. |
| WayBack Machine | http://www.archive.org | |
| Whois | Various | Whois servers for different TLDs. |
| XSSposed | http://www.xssposed.org | |
| Yahoo | http://www.yahoo.com | Scraping but future version to also use API. |
| Zone-H | http://www.zone-h.org | Easy to get black-listed. Log onto the site in a browser from the IP you’re scanning from first and enter the CAPTCHA, then it should be fine. |