SpiderFoot is a reconnaissance tool that automatically queries over 100 public data sources (OSINT) to gather intelligence on IP addresses, domain names, e-mail addresses, names and more. You simply specify the target you want to investigate, pick which modules to enable and then SpiderFoot will collect data to build up an understanding of all the entities and how they relate to each other.
What is OSINT?
OSINT (Open Source Intelligence) is data available in the public domain which might reveal interesting information about your target. This includes DNS, Whois, Web pages, passive DNS, spam blacklists, file meta data, threat intelligence lists as well as services like SHODAN, HaveIBeenPwned? and more. See the full list of data sources SpiderFoot utilises.
What can I do with SpiderFoot?
The data returned from a SpiderFoot scan will reveal a lot of information about your target, providing insight into possible data leaks, vulnerabilities or other sensitive information that can be leveraged during a penetration test, red team exercise or for threat intelligence. Try it out against your own network to see what you might have exposed!
New Modules
The growing numbrs of OSINT sources out there is mind-boggling, and most remain free or at least provide API keys free of charge for low query volumes. In this release, eight new modules have been introduced:
- SecurityTrails (sfp_securitytrails): One of my favourite recent discoveries, SecurityTrails has truly a shedload of DNS and Whois data that any threat intelligence analyst, security analyst or investigator should look into. This module will query their API for IP addresses, domain names, e-mail addresses and owned netblocks to identify co-hosted sites, domains registered under the same e-mail address and more. An API key is required, however limited free usage is provided. Check out their blog post about the integration.
- FullContact.com (sfp_fullcontact): FullContact.com has loads of data about people and companies. This module uses their API (API key required) to look up domain names, e-mail addresses and names in an attempt to identify further e-mail addresses and names, but also physical locations and phone numbers.
- ARIN (sfp_arin): ARIN (American Registry for Internet Numbers) is similar to RIPE (for which SpiderFoot already has a module - sfp_ripe) in that they provide an API to query information about network ranges. But more interestingly from an OSINT perspective, you can query by first and last name, and likewise query by domain name to get affiliated names. This module will take any identified domain name and return a list of human names and ARIN registry data, which will then be scanned by other modules to idenify potential e-mail addresses and hostnames. It will also look up any names to identify potential relevant data.
- Hacked-Emails.com (sfp_hackedemails): Similar to haveibeenpwned.com, hacked-emails.com provides a free service to identify e-mail addresses mentioned in data leaks. This module will query their API for any e-mail address identified during a scan.
- Citadel.pw (sfp_citadel): As above, citadel.pw provides a way to search a large number of leaks for mention of an e-mail address, which is what this module will do. Thanks to citadel.pw - at - protonmail.com for this contribution and for providing a public API key free of charge!
- CIRCL.LU (sfp_circllu): CIRCL.LU (Computer Incident Response Center, Luxembourg) provide a free, however upon-request API to query their rich database of historical SSL and DNS data. This module will take hostnames, owned netblocks, IP addresses and domain names and identify further IP addresses and hostnames, plus SSL certificates and co-hosts related to your target.
- Quad9.net (sfp_quad9): Quad9.net aggregate a number of threat intelligence data sources and integrate them into their resolver, which anyone can point to (9.9.9.9). The resolver will not resolve anything malicious according to the data feeds they have integrated. This module will attempt to resolve identified hostnames, affiliates and co-hosts using 9.9.9.9, and if they fail to resolve there but do resolve using the configured resolver, will report them as malicious.
- RiskIQ / PassiveTotal (sfp_riskiq): RiskIQ provide a threat intelligence platform with an API (API key required) to query their passive DNS and other data. This module will query their API for any hostname, IP address, domain name or e-mail address identified, and return owned netblocks, further IP addresses, co-hosted sites and domain names also registered by the provided e-mail address (reverse Whois).
Improvements
- Dockerfile is now using the Alpine Linux base image, plus some other improvements to bring the image down from about 500MB to 90MB. See this tutorial to try it out.
- Stopped reporting IPv6 addresses from the sfp_ripe module, as it made the malicious modules spin forever on the huge IPv6 address spaces identified. This will be re-visited sometime when IPv6 sees wider adoption.
- Updated the sfp_robtex module to honor throttling and be more configurable.
- Improved sfp_ripe’s ability to identify netblocks possibly owned by the target.
- Handle re-directions when looking for S3 buckets, which will result in many more being found as Amazon returns 30x in many cases, which before was being ignored by SpiderFoot.
- sfp_whois will now perform Whois lookups for affiliate domains and co-hosted sites.
- sfp_onioncity updated to use onion.link.
Enhancements / Bug fixes
- Misc. minor bug fixes, performance improvements and tweaks.