theHarvester is a tool for gathering subdomain names, e-mail addresses, virtual hosts, open ports/ banners, and employee names from different public sources (search engines, pgp key servers).
Is a really simple tool, but very effective for the early stages of a penetration test or just to know the visibility of your company in the Internet.
The sources are:
Passive:
- threatcrowd: Open source threat intelligence - https://www.threatcrowd.org/
- crtsh: Comodo Certificate search - www.crt.sh
- google: google search engine - www.google.com (With optional google dorking)
- googleCSE: google custom search engine
- google-profiles: google search engine, specific search for Google profiles
- bing: microsoft search engine - www.bing.com
- bingapi: microsoft search engine, through the API (you need to add your Key in the discovery/bingsearch.py file)
- dogpile: Dogpile search engine - www.dogpile.com
- pgp: pgp key server - mit.edu
- linkedin: google search engine, specific search for Linkedin users
- vhost: Bing virtual hosts search
- twitter: twitter accounts related to an specific domain (uses google search)
- googleplus: users that works in target company (uses google search)
- yahoo: Yahoo search engine
- baidu: Baidu search engine
- shodan: Shodan Computer search engine, will search for ports and banner of the discovered hosts (http://www.shodanhq.com/)
- hunter: Hunter search engine (you need to add your Key in the discovery/huntersearch.py file)
- google-certificates: Google Certificate Transparency report
Active:
- DNS brute force: this plugin will run a dictionary brute force enumeration
- DNS reverse lookup: reverse lookup of ip´s discovered in order to find hostnames
- DNS TDL expansion: TLD dictionary brute force enumeration
Modules that need API keys to work:
- googleCSE: You need to create a Google Custom Search engine(CSE), and add your Google API key and CSE ID in the plugin (discovery/googleCSE.py)
- shodan: You need to provide your API key in discovery/shodansearch.py (one provided at the moment)
- hunter: You need to provide your API key in discovery/huntersearch.py (none is provided at the moment)
Dependencies:
- Requests library (http://docs.python-requests.org/en/latest/)
pip install requests - Beautiful Soup 4 (https://pypi.org/project/beautifulsoup4//)
pip install beautifulsoup4
Changelog in 3.0.0:
- Subdomain takeover checks
- Port scanning (basic)
- Improved DNS dictionary
- Shodan DB search fixed
- Result storage in Sqlite
Comments? Bugs? Requests?
cmartorella@edge-security.com