Strelka - Scanning Files At Scale With Python And ZeroMQ

• Codebase is Python 3 (minimum supported version is 3.6)
• Designed for non-interactive, distributed systems (network security monitoring sensors, live response scripts, disk/memory extraction, etc.)
• Supports direct and remote file requests (Amazon S3, Google Cloud Storage, etc.) with optional encryption and authentication
• Uses widely supported networking, messaging, and data libraries/formats (ZeroMQ, protocol buffers, YAML, JSON)
• Built-in scan result logging and log management (compatible with Filebeat/ElasticStack, Splunk, etc.)

• Modern codebase (Python 3.6+)
• More scanners (40+ at release) and file types (60+ at release) than related projects
• Supports direct and remote file requests
• Built-in encryption and authentication for client connections
• Built using libraries and formats that allow cross-platform, cross-language support

• Reduce the total volume of files sent to Strelka
• Use a tracking system to only send unique files to Strelka (networked Redis servers are especially useful for this)
• Use traffic control (tc) to shape connections to Strelka

1. Update packages and install build packages

apt-get update && apt-get install --no-install-recommends automake build-essential curl gcc git libtool make python3-dev python3-pip python3-wheel

2. Install runtime packages

apt-get install --no-install-recommends antiword libarchive-dev libfuzzy-dev libimage-exiftool-perl libmagic-dev libssl-dev python3-setuptools tesseract-ocr unrar upx jq

2. Install pip3 packages
   

   pip3 install beautifulsoup4 boltons boto3 gevent google-cloud-storage html5lib inflection interruptingcow jsbeautifier libarchive-c lxml git+https://github.com/aaronst/macholibre.git olefile oletools pdfminer.six pefile pgpdump3 protobuf pyelftools pygments pyjsparser pylzma git+https://github.com/jshlbrd/pyopenssl.git python-docx git+https://github.com/jshlbrd/python-entropy.git python-keystoneclient python-magic python-swiftclient pyyaml pyzmq rarfile requests rpmfile schedule ssdeep tnefparse

3. Install YARA
   

curl -OL https://github.com/VirusTotal/yara/archive/v3.8.1.tar.gz
tar -zxvf v3.8.1.tar.gz
cd yara-3.8.1/
./bootstrap.sh
./configure --with-crypto --enable-dotnet --enable-magic
make && make install && make check
echo "/usr/local/lib" >> /etc/ld.so.conf
ldconfig

4. Install yara-python

curl -OL https://github.com/VirusTotal/yara-python/archive/v3.8.1.tar.gz  
tar -zxvf v3.8.1.tar.gz  
cd yara-python-3.8.1/  
python3 setup.py build --dynamic-linking  
python3 setup.py install

5. Create Strelka directories
   

   mkdir /var/log/strelka/ && mkdir /opt/strelka/

6. Clone this repository
   

   git clone https://github.com/target/strelka.git /opt/strelka/

7. Compile the Strelka protobuf
   

   cd /opt/strelka/server/ && protoc --python_out=. strelka.proto

8. (Optional) Install the Strelka utilities
   

   cd /opt/strelka/ && python3 setup.py -q build && python3 setup.py -q install && python3 setup.py -q clean --all

1. Clone this repository
   

   git clone https://github.com/target/strelka.git /opt/strelka/

2. Build the container
   

   cd /opt/strelka/ && docker build -t strelka .

$ strelka.py

$ strelka_user_client.py --broker 127.0.0.1:5558 --path <path to the file to scan>
$ cat /var/log/strelka/*.log | jq .

• Capability to run servers in any combination of broker/workers
  • Broker distributes file tasks to workers
  • Workers perform file analysis on tasks
• On-disk scan result logging
  • Configurable log rotation and management
  • Compatible with external log shippers (e.g. Filebeat, Splunk Universal Forwarder, etc.)
• Supports encryption and authentication for connections between clients and brokers
• Self-healing child processes (brokers, workers, log management)

usage: strelka.py [options]

runs Strelka as a distributed cluster.

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           enable debug messages to the console
  -c STRELKA_CFG, --strelka-config STRELKA_CFG
                        path to strelka configuration file
  -l LOGGING_INI, --logging-ini LOGGING_INI
                        path to python logging configuration file

usage: strelka_dirstream.py [options]

sends files from a directory to a Strelka cluster in near real-time.

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           enable debug messages to the console
  -c DIRSTREAM_CFG, --dirstream-config DIRSTREAM_CFG
                        path to dirstream configuration file

• Individual file
• Directory of files
• Remote file (see: remote file requests)

usage: strelka_user_client.py [options]

sends ad-hoc file requests to a Strelka cluster.

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           enable debug messages to the console
  -b BROKER, --broker BROKER
                        network address and network port of the broker (e.g.
                        127.0.0.1:5558)
  -p PATH, --path PATH  path to the file or directory of files to send to the
                        broker
  -l LOCATION, --location LOCATION
                        JSON representation of a location for the cluster to
                        retrieve files from
  -t TIMEOUT, --timeout TIMEOUT
                        amount of time (in seconds) to wait until a file
                        transfer times out
  -bpk BROKER_PUBLIC_KEY, --broker-public-key BROKER_PUBLIC_KEY
                        location of the broker Curve public key certificate
                        (this option enables curve encryption and must be used
                        if the broker has curve enabled)
  -csk CLIENT_SECRET_KEY, --client-secret-key CLIENT_SECRET_KEY
                        location of the client Curve secret key certificate
                        (this option enables curve encryption and must be used
                        if the broker has curve enabled)
  -ug, --use-green      determines if PyZMQ green should be used, which can
                        increase performance at the risk of message loss

usage: generate_curve_certificates.py [options]

generates curve certificates used by brokers and clients.

optional arguments:
  -h, --help            show this help message and exit
  -p PATH, --path PATH  path to store keys in (defaults to current working
                        directory)
  -b, --broker          generate curve certificates for a broker
  -c, --client          generate curve certificates for a client
  -cf CLIENT_FILE, --client-file CLIENT_FILE
                        path to a file containing line-separated list of
                        clients to generate keys for, useful for creating many
                        client keys at once

usage: validate_yara.py [options]

validates YARA rules files.

optional arguments:
  -h, --help            show this help message and exit
  -p PATH, --path PATH  path to directory containing YARA rules
  -e, --error           boolean that determines if warnings should cause
                        errors

• "run_broker": boolean that determines if the server should run a Strelka broker process (defaults to True)
• "run_workers": boolean that determines if the server should run Strelka worker processes (defaults to True)
• "run_logrotate": boolean that determines if the server should run a Strelka log rotation process (defaults to True)
• "worker_count": number of workers to spawn (defaults to 4)
• "shutdown_timeout": amount of time (in seconds) that will elapse before the daemon forcibly kills child processes after they have received a shutdown command (defaults to 45 seconds)

• "broker": network address of the broker (defaults to 127.0.0.1)
• "request_socket_port": network port used by clients to send file requests to the broker (defaults to 5558)
• "task_socket_port": network port used by workers to receive tasks from the broker (defaults to 5559)

• "poller_timeout": amount of time (in milliseconds) that the broker polls for client requests and worker statuses (defaults to 1000 milliseconds)
• "broker_secret_key": location of the broker Curve secret key certificate (enables Curve encryption, requires clients to use Curve, defaults to None)
• "client_public_keys": location of the directory containing client Curve public key certificates (enables Curve encryption and authentication, requires clients to use Curve, defaults to None)
• "prune_frequency": frequency (in seconds) at which the broker prunes dead workers (defaults to 5 seconds)
• "prune_delta": delta (in seconds) that must pass since a worker last checked in with the broker before it is considered dead and is pruned (defaults to 10 seconds)

• "task_socket_reconnect": amount of time (in milliseconds) that the task socket will attempt to reconnect in the event of TCP disconnection, this will have additional jitter applied (defaults to 100ms plus jitter)
• "task_socket_reconnect_max": maximum amount of time (in milliseconds) that the task socket will attempt to reconnect in the event of TCP disconnection, this will have additional jitter applied (defaults to 4000ms plus jitter)
• "poller_timeout": amount of time (in milliseconds) that workers poll for file tasks (defaults to 1000 milliseconds)
• "file_max": number of files a worker will process before shutting down (defaults to 10000)
• "time_to_live": amount of time (in minutes) that a worker will run before shutting down (defaults to 30 minutes)
• "heartbeat_frequency": frequency (in seconds) at which a worker sends a heartbeat to the broker if it has not received any file tasks (defaults to 10 seconds)
• "log_directory": location where worker scan results are logged to (defaults to /var/log/strelka/)
• "log_field_case": field case ("camel" or "snake") of the scan result log file data (defaults to camel)
• "log_bundle_events": boolean that determines if scan results should be bundled in single event as an array or in multiple events (defaults to True)

• "directory": directory to run log rotation on (defaults to /var/log/strelka/)
• "compression_delta": delta (in minutes) that must pass since a log file was last modified before it is compressed (defaults to 15 minutes)
• "deletion_delta": delta (in minutes) that must pass since a compressed log file was last modified before it is deleted (defaults to 360 minutes / 6 hours)

• "remote_timeout": amount of time (in seconds) to wait before timing out individual file retrieval
• "remote_retries": number of times individual file retrieval will be re-attempted in the event of a timeout
• "google_application_credentials": path to the Google Cloud Storage JSON credentials file
• "aws_access_key_id": AWS access key ID
• "aws_secret_access_key": AWS secret access key
• "aws_default_region": default AWS region
• "st_auth_version": OpenStack authentication version (defaults to 3)
• "os_auth_url": OpenStack Keystone authentication URL
• "os_username": OpenStack username
• "os_password": OpenStack password
• "os_cert": OpenStack Keystone certificate
• "os_cacert": OpenStack Keystone CA Certificate
• "os_user_domain_name": OpenStack user domain
• "os_project_name": OpenStack project name
• "os_project_domain_name": OpenStack project domain
• "http_basic_user": HTTP Basic authentication username
• "http_basic_pass": HTTP Basic authentication password
• "http_verify": path to the CA bundle (file or directory) used for SSL verification (defaults to False, no verification)

• "close_timeout": amount of time (in seconds) that a scanner can spend closing itself (defaults to 30 seconds)
• "distribution_timeout": amount of time (in seconds) that a single file can be distributed to all scanners (defaults to 1800 seconds / 30 minutes)
• "scanner_timeout": amount of time (in seconds) that a scanner can spend scanning a file (defaults to 600 seconds / 10 minutes, can be overridden per-scanner)
• "maximum_depth": maximum depth that child files will be processed by scanners
• "taste_mime_db": location of the MIME database used to taste files (defaults to None, system default)
• "taste_yara_rules": location of the directory of YARA files that contains rules used to taste files (defaults to etc/strelka/taste/)

scanners:
  'ScanHeader':
    - positive:
        flavors:
          - '*'
      priority: 5
      options:
        length: 50
  'ScanRar':
    - positive:
        flavors:
          - 'rar_file'
      priority: 5
      options:
        limit: 1000

scanners:
  'ScanRar':
    - positive:
        flavors:
          - 'application/x-rar'
          - 'rar_file'
        filename: '\.rar$'
      priority: 5
      options:
        limit: 1000

scanners:
  'ScanRar':
    - negative:
        filename: '\.rar$'
      positive:
        flavors:
          - 'application/x-rar'
          - 'rar_file'
      priority: 5
      options:
        limit: 1000

scanners:
  'ScanX509':
    - positive:
        flavors:
          - 'x509_der_file'
      priority: 5
      options:
        type: 'der'
    - positive:
        flavors:
          - 'x509_pem_file'
      priority: 5
      options:
        type: 'pem'

• "shutdown_timeout": amount of time (in seconds) that will elapse before the utility forcibly kills child processes after they have received a shutdown command (defaults to 10 seconds)

• "directory": directory that files are sent from (defaults to None)
• "source": application that writes files to the directory, used to control metadata parsing functionality (defaults to None)
• "meta_separator": unique string used to separate pieces of metadata in a filename, used to parse metadata and send it along with the file to the cluster (defaults to "S^E^P")
• "file_mtime_delta": delta (in seconds) that must pass since a file was last modified before it is sent to the cluster (defaults to 5 seconds)
• "delete_files": boolean that determines if files should be deleted after they are sent to the cluster (defaults to False)
• "broker": network address and network port of the broker (defaults to "127.0.0.1:5558")
• "timeout": amount of time (in seconds) to wait for a file to be successfully sent to the broker (defaults to 10)
• "use_green": boolean that determines if PyZMQ green should be used (this can increase performance at the risk of message loss, defaults to True)
• "broker_public_key": location of the broker Curve public key certificate (enables Curve encryption, must be used if the broker has Curve enabled)
• "client_secret_key": location of the client Curve secret key certificate (enables Curve encryption, must be used if the broker has Curve enabled)

• "mime_table": table of strings (Bro source) mapped to a set of strings (Bro mime_type) -- this variable defines which file MIME types Bro extracts and is configurable based on the location Bro identified the file (e.g. extract application/x-dosexec files from SMTP, but not SMB or FTP)
• "filename_re": regex pattern that can extract files based on Bro filename
• "unknown_mime_source": set of strings (Bro source) that determines if files of an unknown MIME type should be extracted based on the location Bro identified the file (e.g. extract unknown files from SMTP, but not SMB or FTP)
• "meta_separator": string used in extracted filenames to separate embedded Bro metadata -- this must match the equivalent value in etc/dirstream/dirstream.yml
• "directory_count_interval": interval used to schedule how often the script checks the file count in the extraction directory
• "directory_count_threshold": int that is used as a trigger to temporarily disable file extraction if the file count in the extraction directory reaches the threshold

• Do not run workers on the same server as a broker
  • This puts the health of the entire cluster at risk if the server becomes over-utilized
• Do not over-allocate workers to CPUs
  • 1 worker per CPU
• Allocate at least 1GB RAM per worker
  • If workers do not have enough RAM, then there will be excessive memory errors
  • Big files (especially compressed files) require more RAM
  • In large clusters, diminishing returns begin above 4GB RAM per worker
• Allocate as much RAM as reasonable to the broker
  • ZMQ messages are stored entirely in memory -- in large deployments with many clients, the broker may use a lot of RAM if the workers cannot keep up with the number of file tasks

• Number of file requests per second
• Type of file requests
  • Remote file requests take longer to process than direct file requests
• Diversity of files requested
  • Binary files take longer to scan than text files
• Number of YARA rules deployed
  • Scanning a file with 50,000 rules takes longer than scanning a file with 50 rules

• Share volumes, not files, with the container
  • Strelka's workers will read configuration files and YARA rules files when they startup -- sharing volumes with the container ensures that updated copies of these files on the localhost are reflected accurately inside the container without needing to restart the container
• Increase stop-timeout
  • By default, Docker will forcibly kill a container if it has not stopped after 10 seconds -- this value should be increased to greater than the shutdown_timeout value in etc/strelka/strelka.yml
• Increase shm-size
  • By default, Docker limits a container's shm size to 64MB -- this can cause errors with Strelka scanners that utilize tempfile
• Set logging options
  • By default, Docker has no log limit for logs output by a container