Dr. ROBOT - Tool To Enumerate The Subdomains Associated With A Company By Aggregating The Results Of Multiple OSINT Tools

• Run gather using Sublist3r and Aquatone and Shodan

  python drrobot.py example.domain gather -sub -aqua -shodan 

• Run gather using Sublist3r with Proxy

  python drrobot.py --proxy http://some.proxy:port example.domain gather -sub

• Run inspect using Eyewitness

  python drrobot.py example.domain inspect -eye

• Run inspect using httpscreenshot and grabbing headers

  python drrobot.py example.domain inspect -http -headers

• Run upload using Mattermost/Slack

  python drrobot.py example.domain upload -matter

usage: drrobot.py [-h] [--proxy PROXY] [--dns DNS] [--verbose]                    [--dbfile DBFILE]                    {gather,inspect,upload,rebuild,dumpdb,output,serve}                    ...    Docker DNS recon tool    positional arguments:    {gather,inspect,upload,rebuild,dumpdb,output,serve}      gather              Run scanners against a specified domain and gather the associated                          systems. You have the option to run using any                          docker_buildfiles/webtools included in your config.      inspect             Run further tools against domain information gathered                          from the gather step. Note: you must either supply a file                          which contains a list of IP/Hostnames, or the targeted                          domain must have a db file in the dbs folder      upload              Upload recon data to Mattermost. Currently only works                          with afolder that contain PNG images.      rebuild             Rebuild the database with additional files/all files                          from the previous runtime      dumpdb              Dump the database of ip, hostname, and banners to a text                          file      output              Generate output in specified format. Contains all                          information from scans (images, headers, hostnames,                          ips)      serve               Serve database file in docker container using django    optional arguments:    -h, --help            show this help message and exit    --proxy PROXY         Proxy server URL to set DOCKER http_proxy too    --dns DNS             DNS server to add to resolv.conf of DOCKER containers    --verbose             Display verbose statements    --dbfile DBFILE       Specify what db file to use for saving data too  

  usage: drrobot.py domain gather [-h] [-aqua] [-sub] [-brute] [-sfinder]                                  [-knock] [-amass] [-recong] [-shodan] [-arin]                                  [-hack] [-dump] [-virus] [--ignore IGNORE]                                  [--headers]    positional arguments:    domain                Domain to run scan against    optional arguments:    -h, --help            Show this help message and exit    -aqua, --Aquatone     AQUATONE is a set of tools for performing                          reconnaissance on domain names    -sub, --Sublist3r     Sublist3r is a python tool designed to enumerate                          subdomains of websites using OSINT    -brute, --Subbrute    SubBrute is a community driven project with the goal                          of creating the fastest, and most accurate subdomain                          enumeration tool.    -sfinder, --Subfinder                          SubFinder is a subdomain discovery tool that discovers                          valid subdomains for websites by using passive online                          sources    -knock, --Knock       Knockpy is a python tool designed to enumerate                          subdomains on a target domain through a wordlist    -amass, --Amass       The OWASP Amass tool suite obtains subdomain names by                          scraping data sources, recursive brute forcing,                          crawling web archives, permuting/altering names and                          reverse DNS sweeping.    -recon, --Reconng     Recon-ng is a full-featured Web Reconnaissance                          framework written in Python. DrRobot utilizes several                          of the recon/hosts-domain modules in this framework.    -shodan, --Shodan     Query SHODAN for publicly facing sites of given domain    -arin, --Arin         Query ARIN for public CIDR ranges. This is better as a                          brute force option as the ranges    -hack, --HackerTarget                          This query will display the forward DNS records                          discovered using the data sets outlined above.    -dump, --Dumpster     Use the limited response of DNSDumpster. Requires API                          access for better results.    -virus, --VirusTotal  Utilize VirusTotal's Observer Subdomain Search    --ignore IGNORE       Space seperated list of subnets to ignore    --headers             If headers should be scraped from ip addresses                          gathered  

usage: main.py inspect [-h] [-httpscreen] [-eye] [--proxy PROXY] [--dns DNS]                         [--file FILE]    positional arguments:    domain                Domain to run scan against    optional arguments:    -h, --help            Show this help message and exit    -httpscreen, --HTTPScreenshot                          Post enumeration tool for screen grabbing websites.                          All images will be downloaded to an output file:                          httpscreenshot.tar and unpacked httpscreenshots    -eye, --Eyewitness    Post enumeration tool for screen grabbing websites.                          All images will be downloaded to outfile:                          Eyewitness.tar and unpacked in Eyewitness    --proxy PROXY         Proxy server URL to set for DOCKER http_proxy    --dns DNS             DNS server for the resolv.conf of DOCKER containers    --file FILE           (NOT WORKING) File with hostnames to run further                          inspection on  

usage: drrobot.py [-h] [--proxy PROXY] [--dns DNS] [--verbose]
                  [--dbfile DBFILE]
                  {gather,inspect,upload,rebuild,dumpdb,output,serve}
                  ...

Docker DNS recon tool

positional arguments:
  {gather,inspect,upload,rebuild,dumpdb,output,serve}
    gather              Run scanners against a specified domain and gather the associated
                        systems. You have the option to run using any
                        docker_buildfiles/webtools included in your config.
    inspect             Run further tools against domain information gathered
                        from the gather step. Note: you must either supply a file
                        which contains a list of IP/Hostnames, or the targeted
                        domain must have a db file in the dbs folder
    upload              Upload recon data to Mattermost. Currently only works
                           with afolder that contain PNG images.
    rebuild             Rebuild the database with additional files/all files
                        from the previous runtime
    dumpdb              Dump the database of ip, hostname, and banners to a text
                        file
    output              Generate output in specified format. Contains all
                        information from scans (images, headers, hostnames,
                        ips)
    serve               Serve database file in docker container using django

optional arguments:
  -h, --help            show this help message and exit
  --proxy PROXY         Proxy server URL to set DOCKER http_proxy too
  --dns DNS             DNS server to add to resolv.conf of DOCKER containers
  --verbose             Display verbose statements
  --dbfile DBFILE       Specify what db file to use for saving data too

usage: drrobot.py domain gather [-h] [-aqua] [-sub] [-brute] [-sfinder]
                                [-knock] [-amass] [-recong] [-shodan] [-arin]
                                [-hack] [-dump] [-virus] [--ignore IGNORE]
                                [--headers]

positional arguments:
  domain                Domain to run scan against

optional arguments:
  -h, --help            Show this help message and exit
  -aqua, --Aquatone     AQUATONE is a set of tools for performing
                        reconnaissance on domain names
  -sub, --Sublist3r     Sublist3r is a python tool designed to enumerate
                        subdomains of websites using OSINT
  -brute, --Subbrute    SubBrute is a community driven project with the goal
                        of creating the fastest, and most accurate subdomain
                        enumeration tool.
  -sfinder, --Subfinder
                           SubFinder is a subdomain discovery tool that discovers
                        valid subdomains for websites by using passive online
                        sources
  -knock, --Knock       Knockpy is a python tool designed to enumerate
                        subdomains on a target domain through a wordlist
  -amass, --Amass       The OWASP Amass tool suite obtains subdomain names by
                        scraping data sources, recursive brute forcing,
                        crawling web archives, permuting/altering names and
                        reverse DNS sweeping.
  -recon, --Reconng     Recon-ng is a full-featured Web Reconnaissance
                        framework written in Python. DrRobot utilizes several
                        of the recon/hosts-domain modules in this framework.
  -shodan, --Shodan     Query SHODAN for publicly facing sites of given domain
  -arin, --Arin         Query ARIN for public CIDR r   anges. This is better as a
                        brute force option as the ranges
  -hack, --HackerTarget
                        This query will display the forward DNS records
                        discovered using the data sets outlined above.
  -dump, --Dumpster     Use the limited response of DNSDumpster. Requires API
                        access for better results.
  -virus, --VirusTotal  Utilize VirusTotal's Observer Subdomain Search
  --ignore IGNORE       Space seperated list of subnets to ignore
  --headers             If headers should be scraped from ip addresses
                        gathered

usage: main.py inspect [-h] [-httpscreen] [-eye] [--proxy PROXY] [--dns DNS]
                       [--file FILE]

positional arguments:
  domain                Domain to run scan against

optional arguments:
  -h, --help            Show this help message and exit
  -httpscreen, --HTTPScreenshot
                        Post enumeration tool for screen grabbing websites.
                        All images will be downloaded to an output file:
                        httpscreenshot.tar and unpacked httpscreenshots
  -eye, --Eyewitness    Post enumeration tool for screen grabbing websites.
                        All images will be downloaded to outfile:
                        Eyewitness.tar and unpacked in Eyewitness
  --proxy PROXY         Proxy server URL to set for DOCKER http_proxy
  --dns DNS             DNS server for the resolv.conf of DOCKER containers
  --file FILE           (NOT WORKING) File with    hostnames to run further
                        inspection on

usage: drrobot.py domain upload [-h] [-matter] [-slack] [--filepath FILEPATH]

positional arguments:
  domain                Domain to run scan against

optional arguments:
  -h, --help            Show this help message and exit
  -matter, --Mattermost Mattermost server to upload findings to
                        Mattermost server
  -slack, --Slack       Slack server
  --filepath FILEPATH   Filepath to the folder containing images to upload.
                        This is relative to the domain specified. By default,
                        this will be the path to the output folder

usage: drrobot.py rebuild [-h] [-f [FILES [FILES ...]]]

optional arguments:
  -h, --help            Show this help message and exit
  -f [FILES [FILES ...]], --files [FILES [FILES ...]]
                        Additional files to supply in addition to the ones in the
                        config file

default : specifies a Docker or Ansible instance. Make sure you adjust configurations according to their usage.

• Docker Configuration Requirements
  
  • Example:
    

    usage: drrobot.py dumpdb [-h]
    
    positional arguments:
      domain                Domain to run scan against
    
    optional arguments:
      -h, --help            Show this help message and exit

  • name: Identifiable name for the program/utility you are using
    
  • default : (Disabled for now)
    
  • mode : DOCKER (uses docker container with this tool when chosen)
    
  • docker_name : What the docker image name will be when running docker images
    
  • network_mode : Network mode to use when creating container. Host uses the host network
    
  • default_conf : Template Dockerfile to build form
    
  • active_conf : Target specific configuration that will be used during runtime
    
  • description : Description of tool (optional)
    
  • src : Where the tool comes from (optional)
    
  • output : Location of output on the docker container. Can be hardcoded into Dockerfiles for preference
    
  • output_folder : Location under the outputs/target folder where output for target will be stored
    
• Ansible Configuration Requirements
  
  • Example

  usage: drrobot.py domain output [-h] [--output OUTPUT] {json,xml}
  
  positional arguments:
    {json,xml}       Generate json file under outputs folder (format)
    domain           Domain to dump output of
  
  optional arguments:
    -h, --help       Show this help message and exit
    --output OUTPUT  Alternative location to create output file

  • name: Identifiable name for the program/utility you are using
  • default : (Disabled for now)
  • mode : ANSIBLE (uses Ansible with this tool when chosen)
  • ansible_arguments : Json configuration for specific informaiton
    • config : playbook to use ($config keyword is replaces for full path to file when issuing ansible playbook command)
    • flags : specifies extra flags to be used with the ansible command (specifically useful for any extra flags you would like to use)
    • extra flags : key does not matter so long as it is different from any other key. These extra flags will all be applied to the ansible file in question
  • description : Description of tool (optional)
  • src : Where the tool comes from (optional)
  • output : Where output will be stored on the external file system
  • infile : (Unique for certain modules) what files this program will use as input to the program. In this case you will notice that it searches /tmp/output for aggregated_protocol_hostnames.txt. This file is supplied from the above extra flags option.
• Web Modules
  
  • Example:

  usage: drrobot.py domain serve [-h]
  
  optional arguments:
    -h, --help  show this help message and exit
  

  • short_name : quick reference name for use in CLI
  • class_name : this must match the name you specify for a given class under the respective module name
    • The reason behind this results from the loading of modules at runtime which requires the use of importlib. This will load the respective class from the classname provided via the CLI options.
  • default : false (Disabled for now)
  • api_call_unused : (Old, may be used later...)
  • description : Description of tool (optional)
• Serve Module:
  
  • Example

  "Sublist3r": {
      "name": "Sublist3r",
      "default" : true,
      "mode" : "DOCKER",
      "docker_name": "sub",
      "network_mode": "host",
      "default_conf": "docker_buildfiles/Dockerfile.Sublist3r.tmp",
      "active_conf": "docker_buildfiles/Dockerfile.Sublist3r",
      "description": "Sublist3r is a python tool designed to enumerate subdomains of websites using OSINT",
      "src": "https://github.com/aboul3la/Sublist3r",
      "output": "/root/sublist3r",
      "output_folder": "sublist3r"
  },

  • command: Command to start server on Docker container (Note: For now only using docker)
  • docker_name : What the docker image name will be when running docker images
  • network_mode : Network mode to use when creating container. Host uses the host network
  • default_conf : Template Dockerfile to build form
  • active_conf : Target specific configuration that will be used during runtime
  • description : Description of tool (optional)
  • ports: Port mapping of localhost to container for docker

1. Add the new tool to the user_config.json
   

       "HTTPScreenshot": {
           "name" : "HTTPScreenshot",
           "short_name" : "http",
           "mode" : "ANSIBLE",
           "ansible_arguments" : {
               "config" : "$config/httpscreenshot_play.yml",
               "flags": "-e '$extra' -i ansible_plays/inventory.yml",
               "extra_flags":{
                   "1" : "variable_host=localhost",
                   "2" : "infile=$infile/aggregated/aggregated_protocol_hostnames.txt",
                   "3" : "outfile=$outfile/httpscreenshots.tar",
                   "4" : "outfolder=$outfile/httpscreenshots",
                   "5" : "variable_user=bitnami"
               }
           },
           "description" : "Post enumeration tool for screen grabbing websites. All images will be downloaded to outfile: httpscreenshot.tar and unpacked httpscreenshots",
           "output" : "/tmp/output",
           "infile" : "/tmp/output/aggregated_protocol_hostnames.txt",
           "   enabled" : false

2. Open src/web_resources.py and make a class with the class_name specified in the previous step. MAKE SURE IT MATCHES EXACTLY
   

           "HackerTarget" :
       {
           "short_name" : "hack",
           "class_name" : "HackerTarget",
           "default" : false,
           "description" : "This query will display the forward DNS records discovered using the data sets outlined above.",
           "api_call_unused" : "https://api.hackertarget.com/hostsearch/?q=example.com",
           "output_file" : "hacker.txt"
       },

1. Add the json to the config file (user if generated).
   

   "Serve" : {
       "name" : "Django",
       "command" : "python manage.py runserver 0.0.0.0:8888",
       "docker_name": "django",
       "network_mode": "host",
       "default_conf": "serve_api/Dockerfile.Django.tmp",
       "active_conf": "serve_api/Dockerfile.Django",
       "description" : "Django container for hosting database",
       "ports" : {
           "8888" : "8888"
        }
   }

   1. Note network_mode is an option specifically for docker containers. It is implementing the --network flag when using docker
2. Under the docker_buildfiles/ folder, create your Dockerfile.NewTool.tmp dockerfile.
   
   1. If you desire adding more options at run time to the Dockerfiles, look at editing src/dockerize
   2. Note: As of right now Dockerfiles must come from the docker_buildfiles folder. Future work includes specifying a remote source for the docker images.

1. Add the json to the config file (user if generated).
   

   {
     "WebTools":
     {
       "NewTool" :
       {
         "short_name": "ntool",
         "class_name": "NewTool",
         "description" : "NewTool description",
         "output_file" : "newtool.txt",
         "api_key" : null,
         "endpoint" : null,
         "username" : null,
         "password" : null
       },

2. As you can see, this has a few items that may seem confusing at first, but will be clarified here:
   
   1. mode: Allows you to specify how you want to deploy a tool you want to use. Currently DOCKER or ANSIBLE are the only available methods to deploy.
      
   2. All options outside of ansible_configuration will be ignored when developing for ANSIBLE.
      
   3. Options under ansible_arguments
      
      1. config: specify which playbook to use
         
      2. flags: which flags to pass to the ansible-playbook command. With the exception of the $extra flag, you can add anything you would like to be done uniquely here.
         
      3. extra_flags : this corresponds to the $extra flag as seen above. This will be used to populate variables that you input into your playbook. You can use this to supply command line arguments when utilizing ansible and Dr. Robot in order to add files and other utilities to your script.
         
         1. variable_host : hostname alias found in the inventory file
         2. variable_user : user to login as on the variable_host machine
         3. infile: file to be used with the tool above. Eyewitness requires hostnames with the format https://some.url, hence aggregated_protocol_hostnames.txt
         4. Note the use of the prefix $infile- these names all match as they are placeholders for the default locations that $infile corresponds to in outputs/target_name/aggregated
         5. If you have a file in another location you can just specify the entire path without any errors occurring.
         6. outfile : The output file location
         7. As with the above infile $outfile in the name is just a key to the location outputs/target_name/
         8. You may specify a hard coded path for other use. Just remember the location for uploading or other processing with Dr. Robot
         9. outfolder : The output folder to unpack/download files too
         10. As with the above infile $outfile in the name is just a key to the location outputs/target_name/
         11. This is a special case for Eyewitness and HttpScreenshot, which you can see in their playbooks. They generate a lot of files and rather than download each individually having them pack up the files as a step in the playbook and then unpacking allows for some integrity.
         12. A quick example below shows how we use the extra_flags to supply the hostname to the playbook for ansible.

         class NewTool(WebTool):
          def __init__(self, **kwargs):
              super().__init__(**kwargs)
              ....
          def do_query(self):
              .... do the query ... 
              store results in
              self.results

"Scanners" : {
 ...
    "NewTool": {
        "name": "NewTool",
        "default" : true, 
        "mode" : DOCKER,
        "docker_name": "ntool",
        "network_mode": "host",
        "default_conf": "docker_buildfiles/Dockerfile.NewTool.tmp",
        "active_conf": "docker_buildfiles/Dockerfile.NewTool",
        "description": "NewTool is an awesome tool for domain enumeration",
        "src": "https://github.com/NewTool",
        "output": "/home/newtool",
        "output_file": "NewTool.txt"
      },
      ...
}

"Enumeration" : {
        "Eyewitness": {
            "name" : "Eyewitness",
            "short_name" : "eye",
            "docker_name" : "eye",
            "mode" : "ANSIBLE",
            "network_mode": "host",
            "default_conf" : "docker_buildfiles/Dockerfile.Eyewitness.tmp",
            "active_conf" : "docker_buildfiles/Dockerfile.Eyewitness",
            "ansible_arguments" : {
                "config" : "$config/eyewitness_play.yml",
                "flags": "-e '$extra' -i ansible_plays/inventory",
                "extra_flags":{
                    "1" : "variable_host=localhost",
                    "2" : "variable_user=root",
                    "3" : "infile=$infile/aggregated_protocol_hostnames.txt",
                    "4" : "outfile=$outfile/Eyewitness.tar",
                    "5" : "outfolder=$outfile/Eyewitness"
                }
            },
            "desc   ription" : "Post enumeration tool for screen grabbing websites. All images will be downloaded to outfile: Eyewitness.tar and unpacked in Eyewitness",
            "output" : "/tmp/output",
            "infile" : "/tmp/output/aggregated/aggregated_protocol_hostnames.txt",
            "enabled" : false
        },
}

---
- hosts: "{{ variable_host|quote }}"
  remote_user: root 

  tasks:
      - name: Apt install git
        become: true
        apt:
            name: git
            force: yes

• Docker required for any of the scanners to run
  
• Python 3.6 required
  
  • Pipenv for versioning of all Python packages. You can use the Pipfile with setup.py requirements as well.
    

    FROM python:3.4
    
    WORKDIR /home
    ENV http_proxy $proxy
    ENV https_proxy $proxy
    ENV DNS $dns
    ENV TARGET $target
    ENV OUTPUT $output
    
    RUN mkdir -p $$OUTPUT
    
    RUN if [ -n "$$DNS" ]; then echo "nameserver $DNS" > /etc/resolv.conf; fi; apt-get install git
    
    RUN if [ -n "$$DNS" ]; then echo "nameserver $DNS" > /etc/resolv.conf; fi; git clone https://github.com/aboul3la/Sublist3r.git /home/sublist
    
    WORKDIR /home/sublist
    
    RUN if [ -n "$$DNS" ]; then echo "nameserver $DNS" > /etc/resolv.conf; fi; pip3 install -r requirements.txt
    
    ENTRYPOINT python3 sublist3r.py --domain $target -o $output/sublist3r.txt

• Ansible if you require the use of external servers.
  
• Python Mattermost Driver [Optional] if using Mattermost you will require this module
  

[example-host]
ip.example.com

eval $(ssh-agent -s)
ssh-add /path/to/sshkey

cd /path/to/drrobot/
pipenv install && pipenv shell
python drrobot.py <command> <flags> target