Skip to main content
This documentation provides a comprehensive, practical reference for developers working with rules for analysis and filters for data extraction, enrichment, and transformation within the UTMStack v11 ecosystem powered by EventProcessor.
Target Audience: This section is designed as a practical reference for developers implementing security detection logic and data transformation pipelines.

​
What is EventProcessor?

EventProcessor is UTMStack v11’s revolutionary security event processing engine that replaced Logstash in the new architecture. Developed by ThreatWinds, it provides:
  • Drastically reduced resource usage compared to Logstash
  • Improved log processing speed
  • Plugin-based architecture for extensibility
  • Real-time correlation before data ingestion
  • Native security analysis capabilities
New in v11: EventProcessor is the core component that processes all security events through a flexible plugin system, enabling UTMStack to scale horizontally and process data in parallel.

​
Architecture Overview

The EventProcessor and UTMStack v11 integration consists of several key components:

EventProcessor Core

Core engine that processes security events, loads plugins, and manages data flow

EventProcessor Plugins

Built-in plugins for input, parsing, and analysis tasks

UTMStack Plugins

Integration plugins for cloud services, correlation, and notifications

go-sdk

Go SDK providing common functionality and interfaces

​
Communication Architecture

EventProcessor uses a plugin architecture where:
  • Plugins are separate processes
  • Communication via gRPC over Unix sockets
  • Allows flexible and extensible system
  • New functionality added without modifying core engine

​
Core Components

​
1. EventProcessor Plugins

Built-in plugins organized by function: Input Plugins
  • Collect or receive logs from external sources
  • Examples: http-input, grpc-input
Parsing Plugins
  • Extract, enrich, and transform logs
  • Examples: add, cast, csv, delete, expand, grok, json, kv, reformat, rename, trim
Analysis Plugins
  • Process logs to detect security attacks
  • Generate alerts based on correlation rules

​
2. UTMStack Plugins

Integration and specialized plugins: Input Plugins
  • aws, azure, gcp, o365, sophos, bitdefender
  • Custom data collection from various sources
Parsing Plugins
  • geolocation - IP geolocation enrichment
Analysis Plugins
  • events - Event processing and analysis
Correlation Plugins
  • alerts - Alert correlation
  • soc-ai - AI-powered analysis
Notification Plugins
  • stats - Statistics and notifications
Sidecar Plugins
  • config - Configuration management
  • Background tasks for system health

​
Rules vs Filters: Understanding the Difference

​
Rules

Rules are YAML files that define how to analyze events to detect security threats.Purpose:
  • Detect security attacks and anomalies
  • Generate alerts when conditions are met
  • Correlate events across time windows
  • Deduplicate similar alerts
Used by: Analysis plugin to generate security alertsExample Use Cases:
  • Brute force attack detection
  • Data exfiltration monitoring
  • Insider threat detection
  • Lateral movement detection

​
Filters

Filters are YAML files that define how to extract and transform raw event data.Purpose:
  • Parse raw log data into structured format
  • Extract relevant fields from logs
  • Normalize field names
  • Enrich data with additional context
  • Transform data types
Used by: Parsing plugin to standardize eventsExample Use Cases:
  • Parse Apache access logs
  • Extract fields from Cisco ASA logs
  • Normalize Windows Event logs
  • Enrich with geolocation data

​
Data Flow Pipeline

​
Processing Stages

1

Data Collection

Input plugins collect raw logs from various sources (agents, syslog, APIs)
2

Parsing & Transformation

Filters extract and transform raw data into standardized format using parsing plugins
3

Event Processing

EventProcessor routes events to appropriate plugins based on data type
4

Security Analysis

Rules analyze events to detect threats and generate alerts
5

Alert Correlation

Correlation plugins detect relationships between alerts
6

Notification & Storage

Alerts are stored and notifications sent to configured systems

​
Key Features

​
Plugin-Based Architecture

Modular Design

Each plugin serves a specific purpose in the processing pipeline

Independent Processes

Plugins run as separate processes, improving stability

gRPC Communication

Fast, efficient communication via gRPC over Unix sockets

Hot Swappable

Add or update plugins without restarting core engine

​
Real-Time Correlation

UTMStack’s Advantage: Correlation happens before data ingestion, reducing workload and improving response times compared to traditional SIEM systems.

​
Horizontal Scalability

With v11’s manager-worker architecture:
  • Add worker nodes to scale processing capacity
  • Distribute plugins across workers
  • Linear performance scaling
  • No single point of failure

​
Developer Tools: go-sdk

The go-sdk provides essential functionality for plugin development:

​
Key Components

Protocol Buffers Definition 
plugins/plugins.proto
Defines messages and services for plugin communication Configuration Management 
plugins/config.go
Provides configuration functionality for plugins Expression Language (CEL) 
plugins/cel.go
Common Expression Language support for rules

​
File Structure

​
Rules Directory Structure

rules/
├── authentication/
│   ├── brute_force.yaml
│   └── failed_login.yaml
├── network/
│   ├── port_scan.yaml
│   └── data_exfiltration.yaml
└── malware/
    └── malicious_file.yaml

​
Filters Directory Structure

filters/
├── apache/
│   └── access_log.yaml
├── cisco/
│   ├── asa.yaml
│   └── firepower.yaml
└── windows/
    └── event_log.yaml

​
Rule and Filter Basics

​
Rule Structure (Simplified)

- id: 1
  dataTypes:
    - windows
  name: Suspicious Login
  where: failed_login_count > 5
  afterEvents:
    - indexPattern: v11-log-*
      with:
        - field: origin.ip
          operator: filter_term
          value: '{{origin.ip}}'
      within: now-1h
      count: 5

​
Filter Structure (Simplified)

pipeline:
  - dataTypes:
      - apache
    steps:
      - json:
          source: raw
      - grok:
          patterns:
            - fieldName: origin.ip
              pattern: '{{.ipv4}}'
          source: log.message
      - cast:
          fields:
            - statusCode
          to: int

​
Development Workflow

1

Understand the Requirement

Identify what security threat to detect (rules) or what data source to parse (filters)
2

Examine Sample Data

Collect and analyze sample events or logs
3

Create YAML Definition

Write rule or filter using appropriate syntax
4

Test Locally

Test with sample data in development environment
5

Deploy

Deploy to UTMStack and monitor performance
6

Refine

Adjust based on real-world results and feedback

​
Best Practices

​
For Rules

  • Start simple and refine based on results
  • Test thoroughly with various event types
  • Use clear descriptions and references
  • Consider performance impact of complex expressions
  • Implement deduplication to prevent alert fatigue

​
For Filters

  • Standardize field names across all filters
  • Remove unnecessary fields to reduce storage
  • Handle edge cases and malformed data gracefully
  • Document complex patterns with comments
  • Test with real production data samples

​
Performance Considerations

​
Optimization Tips

Rules Optimization
  • Limit data types to reduce evaluation overhead
  • Use efficient CEL expressions
  • Limit afterEvents search windows
  • Implement proper deduplication
Filters Optimization
  • Apply steps conditionally when possible
  • Remove fields early in pipeline
  • Use efficient parsing methods (grok patterns)
  • Avoid unnecessary transformations

​
Resource Management

CPU Usage

Complex rules and filters consume more CPU
  • Monitor CPU usage per plugin
  • Optimize expressions and patterns

Memory Usage

Large event volumes require more memory
  • Set appropriate buffer sizes
  • Clean up unnecessary data early

​
Getting Started

Ready to start developing rules and filters?

Implementing Rules

Learn how to create correlation rules to detect security threats

Implementing Filters

Learn how to create filters for data extraction and transformation

Custom Plugins

Develop custom plugins to extend EventProcessor functionality

Best Practices

Learn optimization techniques and common patterns

​
Community and Support

GitHub Repository

View source code and contribute

Rules Repository

Community correlation rules

Discord Community

Get help from the community

Documentation

Browse complete documentation

​
Next Steps

This section provides a practical developer reference. Each guide includes real-world examples, code snippets, and best practices for implementing security detection and data transformation in UTMStack v11.
Continue to the specific guides: