TheTHE is an environment intended to help analysts and hunters over the early stages of their work in an easier, unified and quicker way. One of the major drawbacks when dealing with a hunting is the collection of information available on a high number of sources, both public and private.
All this information is usually scattered and sometimes even volatile. Perhaps at a certain point there is no information on a particular IOC (Indicator of Compromise), but that situation may change within a few hours and become crucial for the investigation. Based on our experience on Threat Hunting, we have created a free and open source framework to make the early stages of the investigation simpler from:
- Server-client architecture. Investigation may be shared among your team.
- APIkeys are stored in a database and may be shared by a team from a single point.
- Results are cached; so not repeated API calls are used.
- Better feeds your Threat Intelligence Platform. TheTHE allows to better perform a prior investigation of your assets.
- Easy plugins: Whatever you need, it is easily embedded within the system.
- Ideal for SOCs, CERTS or any team.
- Automation of tasks and searches.
- Rapid API processing of multiple tools.
- Unification of information in a single interface, so that screenshots, spreadsheets, text files, etc. are not scattered.
- Enrichment of collected data.
- Periodic monitoring of a given IOC in case new information or related movements appear.
TheTHE has a web interface where the analyst starts its work by entering IOCs that will be sent to a backend, where the system will automatically look up for such resource on the various configured platforms in order to obtain unified information from different sources and access related reports or data existing on them. Furthermore, any change in the resources to be analyzed will be monitored. Everything is executed on a local system, without needing to share information with third parties until such information is not organized, linked, complete and synthesized. This allows that, in case the information must be analyzed later on any other platform (such as a Threat Intelligence Platform), it can be done in the most enriching possible manner.
A complete docker image is scheduled to be released soon, meanwhile you can play with the dev enviroment
Install (development enviroment)
- git clone ...
Creating a Python virtual enviroment
- pip install venv
- python3 -m venv venv
Pull docker images
- docker-compose build
Install node packages
- cd frontend and npm install
Running (development enviroment)
Activate the Python virtual enviroment (from the project root):
source venv/bin/activateBackend:
watchmedo auto-restart -d tasks -p '*.py' -- celery -A tasks.tasks:celery_app worker -l infogunicorn server.main:app --reloadcd frontend
npm run serveSetting up the initial account
You must open a session in the MongoDB (remember is a docker container):
docker exec -it thethe_mongo_1 /bin/bashmongo -u MONGO_INITDB_ROOT_USERNAME -p MONGO_INITDB_ROOT_PASSWORDuse thethedb.users.insert_one({"name": YOUR_NAME, "password": HASH})from passlib.apps import custom_app_context as pwd_context
pwd_context.encrypt("yourpassword")How to create a plugin for THETHE
Plugins affect one or more resources. It may have a frontend counterpart written as a Vue (javascript) component. Plugins must registers itself to describe what they do, etc.
When the plugin is launched it may be queued in a Celery queue for background processing (or not, depending in the kind of tasks, actions, etc.)
A plugin does have the following structure:
Python plugin
A plugin in Python is just for describe the actions, description, name, category, etc. It must inherit from plugin_base/Plugin
The plugin MUST be located in server/plugins/ folder. Below is an exrtract of server/plugins/geoip.py
RESOURCE_TARGET = [ResourceType.IPv4]
# Plugin Metadata {a decription, if target is actively reached and name}
PLUGIN_DESCRIPTION = "Use a GeoIP service to geolocate an IP address"
PLUGIN_IS_ACTIVE = False
PLUGIN_NAME = "geoip"
class Plugin:
description = PLUGIN_DESCRIPTION
is_active = PLUGIN_IS_ACTIVE
name = PLUGIN_NAME
def __init__(self, resource, project_id):
self.project_id = project_id
self.resource = resource
def do(self):
resource_type = self.resource.get_type()
try:
to_task = {
"ip": self.resource.get_data()["address"],
"resource_id": self.resource.get_id_as_string(),
"project_id": self.project_id,
"resource_type": resource_type.value,
"plugin_name": Plugin.name,
}
geoip_task.delay(**to_task)
except Exception as e:
tb1 = traceback.TracebackException.from_exception(e)
print("".join(tb1.format()))Celery task
Finally, the task is introduced as a Celery function which gives us async results. It SHOULD be located into tasks/tasks.py file with the proper Celery decorators
@celery_app.task
def geoip_task(plugin_name, project_id, resource_id, resource_type, ip):
try:
query_result = geoip(ip)
if not query_result:
return
# TODO: See if ResourceType.__str__ can be use for serialization
resource_type = ResourceType(resource_type)
resource = Resources.get(resource_id, resource_type)
resource.set_plugin_results(
plugin_name, project_id, resource_id, resource_type, query_result
)
except Exception as e:
tb1 = traceback.TracebackException.from_exception(e)
print("".join(tb1.format()))Celery subtask
You MAY split the core functionality in a external module located into the tasks/subtasks folder.
def geoip(ip):
try:
URL = f"http://api.ipstack.com/{ip}?access_key={API_KEY}&format=1"
response = urllib.request.urlopen(URL).read()
return json.loads(response)
except Exception as e:
tb1 = traceback.TracebackException.from_exception(e)
print("".join(tb1.format()))
return NoneVue component
Finally, the frontend MIGHT know hot to represent the results as a view. As the system is designed to support dynamic load of plugins all the Vue components are loaded under demand via the DynamicComponent component.
Plugins must have an entry into the frontend/src/components/templates/<plugin name>/index.vue
Here you must deal with the plugins results presentation. As an example (taking GeoIp plugin):
<template>
<v-layout row pt-2 wrap class="subheading">
<v-flex lg5>
<v-flex>
<v-card>
<v-card-title primary-title>
<span class="subheading">Geolocalization</span>
</v-card-title>
<v-divider></v-divider>
<v-card-text>
<v-layout row>
<v-flex lg-6 class="text-xs-left">
<v-layout column>
<v-flex>
<v-label>Continent:</v-label>
</v-flex>
<v-flex>
<v-label>Country:</v-label>
</v-flex>
<v-flex>
<v-label>Region:</v-label>
</v-flex>
<v -flex>
<v-label>City:</v-label>
</v-flex>
<v-flex>
<v-label>Zip:</v-label>
</v-flex>
<v-flex>
<v-label>Latitude:</v-label>
</v-flex>
<v-flex>
<v-label>Longitude:</v-label>
</v-flex>
<v-flex v-if="resource.country_code">
<v-label>Flag:</v-label>
</v-flex>
</v-layout>
</v-flex>
<v-flex lg-6 class="text-xs-right">
<v-layout column>
<v-flex>{{ resource.continent_name }}</v-flex>
<v-flex>{{ resource.co untry_name }}</v-flex>
<v-flex>{{ resource.region_name }}</v-flex>
<v-flex>{{ resource.city }}</v-flex>
<v-flex>{{ resource.zip }}</v-flex>
<v-flex>{{ resource.latitude }}</v-flex>
<v-flex>{{ resource.longitude }}</v-flex>
<v-flex v-if="resource.country_code">
<country-flag :country="resource.country_code"></country-flag>
</v-flex>
</v-layout>
</v-flex>
</v-layout>
</v-card-text>
</v-card>
</v-flex>
</v-flex>
</v-layout>
</template>
<script>
import { make_unique_list } from "../../../utils/utils";
export default {
name: "geoip" ,
props: {
plugin_results: Object
},
data: function() {
return {};
},
computed: {
resource: function() {
let plugin_result = { ...this.plugin_results };
return plugin_result;
}
}
};
</script>