ober/gerbil-es-proxy
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
Elasticsearch logging proxy in Gerbil Scheme.
5 commits 1 branch 0 tags 57 KiB
  • Scheme 92.4%
  • Makefile 7.6%
Find a file
Exact
Jaime Fournier 732f55a19a Fix heap measurement: use field 17 (live objects) instead of field 7/0 
##process-statistics field 7 measures managed heap capacity which grows
monotonically for small-object workloads and never shrinks after GC.
Field 0 is CPU time, not heap at all. Both caused false "memory leak"
reports showing 655GB of apparent growth. Field 17 tracks actual live
object bytes after GC, which correctly stabilizes when no leak exists.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-10 12:53:08 -07:00
.gitignore Add es-proxy v0.2.0: Gerbil ES proxy with stress tests and full test suite 2026-02-10 11:57:47 -07:00
build.ss Compile stress test to native binary for faster execution 2026-02-10 12:18:16 -07:00
es-proxy-stress-test.ss Fix heap measurement: use field 17 (live objects) instead of field 7/0 2026-02-10 12:53:08 -07:00
es-proxy-test.ss Fix heap measurement: use field 17 (live objects) instead of field 7/0 2026-02-10 12:53:08 -07:00
es-proxy.ss Add es-proxy v0.2.0: Gerbil ES proxy with stress tests and full test suite 2026-02-10 11:57:47 -07:00
gerbil.pkg Add es-proxy v0.2.0: Gerbil ES proxy with stress tests and full test suite 2026-02-10 11:57:47 -07:00
Makefile Add patchelf OpenSSL rpath fix for compiled binaries 2026-02-10 12:19:41 -07:00
manifest.ss Fix heap measurement: use field 17 (live objects) instead of field 7/0 2026-02-10 12:53:08 -07:00
mock-es.ss Add es-proxy v0.2.0: Gerbil ES proxy with stress tests and full test suite 2026-02-10 11:57:47 -07:00
README.md first commit 2026-02-10 11:42:33 -07:00

README.md

es-proxy

A transparent Elasticsearch proxy that logs all requests and responses with UUID correlation for auditing. Written in Gerbil Scheme.

Overview

es-proxy sits between Elasticsearch clients and coordinator nodes, logging comprehensive request/response metadata to enable forensic analysis and auditing with minimal performance overhead.

Installation

make build
make install

Usage

es-proxy <remote-host> <remote-port> <local-host> <local-port> [--no-tls] [--cert PATH] [--key PATH]

Example:

es-proxy es-prod-001.example.net 9200 0.0.0.0 9200

With plain HTTP (no TLS):

es-proxy localhost 9200 0.0.0.0 9200 --no-tls

The process runs under systemd as es-proxy.service.

Log Format

Each request/response pair is correlated by UUID:

Request:

TIMESTAMP ID: UUID User: USERNAME Path: /path Query: PARAMS Method: GET Source: IP Agent: USER_AGENT Body: BASE64

Response:

TIMESTAMP ID: UUID User: USERNAME Path: /path Query: PARAMS Method: GET Source: IP Agent: USER_AGENT Status: 200 Duration: 0.005s Size: 1234

Testing

Run all tests:

make test

Run specific test suites:

make test-functional   # Functional tests (mock ES)
make test-memory       # Memory leak detection test
make test-stress       # Extended stress test (5 minutes)

The test suite includes:

  • MockElasticsearch - A mock ES server handling typical endpoints (cluster info, health, search, bulk indexing, cat APIs)
  • Response validation - Validates that mock responses match request parameters (search size/from, bulk item counts, document IDs)
  • Error handling - Tests proper error responses for invalid JSON input
  • Proxy integration - Tests full request/response proxying through es-proxy in plain HTTP mode
  • High-volume concurrency - Validates behavior under 100 concurrent requests from 10 clients
  • Memory leak detection - Tracks heap growth across iterations to detect memory leaks
  • Stress testing - 5-minute sustained load with 50 concurrent clients, varying request sizes, and memory monitoring

Response Validation

The mock Elasticsearch server parses request bodies and generates contextual responses:

  • Search: Returns hits matching requested size and from parameters
  • Bulk: Returns one item per document, preserving _index and _id from request
  • Index document: Returns correct index name and generates/preserves document ID
  • Invalid JSON: Returns 400 status with parse_exception error

Makefile Targets

Run make help for a full list of available targets:

make build        # Build the es-proxy binary
make test         # Run all tests
make test-stress  # Run extended stress test (5 minutes)
make install      # Install binary to /usr/local/bin
make deps         # Install package dependencies
make clean        # Remove build artifacts
make help         # Show all targets

Development

Build from source:

make build

Run tests:

make test

gerbil-es-proxy