Epochly AI Inference: Configuration

Configure the inference components programmatically at construction, plus the small set of environment variables the runtime actually reads.

Overview

The inference accelerator is configured programmatically: each component takes its own configuration object when you construct it (L2CacheConfig, L3CacheConfig, LLMCompanionConfig, RayServeConfig, ... -- see the sections below). In addition, the runtime reads the small set of environment variables listed in Environment Variables.

The InferenceConfig dataclass (with its BatchingConfig, CompilationConfig, and CacheSettings sections) is a declarative, programmatic schema. You can build one directly, from an already-parsed dictionary (InferenceConfig.from_dict(...)), or as a snapshot of the environment (InferenceConfig.from_env()), and pass values from it to the components you construct -- but the runtime does not read an InferenceConfig on its own in the current release.

> Not yet implemented: a unified configuration system -- pyproject.toml

> loading, environment-variable overrides for batching / compilation /

> cache tuning, and a programmatic > environment > file precedence chain --

> is planned but not wired in the current release. pyproject.toml sections

> are not read, and no precedence merging happens between configuration

> sources.

Environment Variables

The runtime reads the environment variables below, plus EPOCHLY_L2_ENCRYPTION_KEY, which is documented with its component in L2 Cache.

Top-Level

> Not yet implemented: environment-variable overrides for batching,

> compilation, cache, and maximum-level tuning are accepted by the

> programmatic InferenceConfig.from_env() snapshot loader (invalid values

> warn and keep defaults) but are not consumed by the runtime: the

> batching / compilation / cache components read their configuration

> objects at construction, not the environment. The table above, the

> Privacy table, and the L2 encryption key are the complete

> set of inference environment variables the runtime reads.

Privacy

> Not yet implemented: environment-variable control of the privacy mode,

> the tenant-isolation toggle, an audit-log file path, an at-rest-encryption

> toggle, and an encryption-key source selector is planned but not wired in

> the current release. The rows above are the complete set of privacy-related

> environment variables the runtime reads. L2 at-rest encryption is enabled

> by providing an encryption key -- programmatically or via

> EPOCHLY_L2_ENCRYPTION_KEY (see L2 Cache).

Privacy Modes

privacy_mode ("ephemeral", "hashes_only", "persisted_encrypted") is accepted and stored by the cache configuration (CacheConfig / CacheSettings), but the current release does not change cache-tier persistence behavior based on it: the managed cache path is the in-memory L1 tier regardless of mode, and the L2 (SQLite) / L3 (Redis) tiers are separate components you construct and wire explicitly. L2 data is encrypted only when an encryption key is provided (see L2 Cache).

> Planned behavior (not yet implemented): mode-driven tier behavior --

> ephemeral / hashes_only suppressing L2/L3 persistence, and

> persisted_encrypted automatically enabling SQLite + AES-256-GCM (L2) and

> Redis + TLS (L3).

Privacy Controls

Privacy controls are configured directly on the components below: redaction patterns on InputRedactor, tenant IDs on TenantIsolation, and buffer capacity on AuditLogger. The PrivacyConfig dataclass exists as a declarative schema, but its persistence-related fields (audit_log_path, encrypt_at_rest, encryption_key_source) are not consumed by the runtime in the current release.

Input Redaction

PII and sensitive data are scrubbed before any data is written to disk or retained in golden stores. Redaction engages wherever inputs are actually retained (golden capture on profiled proxy paths and safety-orchestrator flows). Note that the epochly.wrap() framework route (PyTorch / Transformers / ONNX Runtime models) is a zero-overhead passthrough that retains no call inputs at all, so there is nothing for redaction to intercept on that route. Configure regex patterns to match sensitive data:

from epochly.inference.safety.privacy import InputRedactor
redactor = InputRedactor(
    patterns=[
        r"\b\d{3}-\d{2}-\d{4}\b",           # SSN
        r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",  # Email
        r"\b\d{16}\b",                        # Credit card
    ],
    replacement="[REDACTED]",
)
clean_text = redactor.redact("SSN: 123-45-6789")
# "SSN: [REDACTED]"

Tenant Isolation

Cache keys are namespaced by tenant ID to prevent cross-tenant data leakage:

from epochly.inference.safety.privacy import TenantIsolation
# From constructor
isolation = TenantIsolation(tenant_id="customer_abc")
# From environment (reads EPOCHLY_TENANT_ID)
isolation = TenantIsolation.from_env()
# Namespace a cache key
namespaced = isolation.namespace_key("model:bert:input_hash_abc123")
# "customer_abc:model:bert:input_hash_abc123"

Audit Logging

An in-memory audit log records safety gate decisions, cache accesses, and optimization state changes. Entries are held in a bounded in-process buffer (max_entries, default 10,000; the oldest entries are evicted once the buffer is full) for the lifetime of the process -- they are not persisted to disk.

> Not yet implemented: persistent audit-log files. Export entries via

> get_entries() / get_entries_since() if you need durable audit records.

from epochly.inference.safety.privacy import AuditLogger
audit = AuditLogger(max_entries=10_000)
audit.log_event(
    operation="canary_validation",
    model_id=42,
    optimization_name="torch_compile_bert",
    result="PASS",
    details={"cosine_sim": 0.998, "max_diff": 0.0012},
)
# Query recent entries
entries = audit.get_entries_since(time.time() - 3600)  # Last hour

L2 Cache (SQLite)

The L2 cache is configured separately when creating the cache directly:

from epochly.inference.cache_l2 import L2Cache, L2CacheConfig
l2 = L2Cache(L2CacheConfig(
    db_path="~/.epochly/inference_cache/cache.db",
    ttl_seconds=86400,
    encryption_key=b"32-byte-aes-256-key-here!!!!!!!",  # Optional
))

Encryption Requirements:

• AES-256-GCM requires a 32-byte key
• Key sources, in precedence order: L2CacheConfig.encryption_key (explicit, always wins), then the EPOCHLY_L2_ENCRYPTION_KEY environment variable, read once when the cache is constructed
• The environment value must UTF-8-encode to exactly 32 bytes; empty, whitespace-only, or wrong-size values raise ValueError at construction (data is never silently stored as plaintext when a key was set)
• The key is never logged
• When the cryptography library is absent, data is stored unencrypted and a warning is logged
• Each row uses a unique 12-byte nonce

L3 Cache (Redis)

from epochly.inference.cache_l3 import L3Cache, L3CacheConfig
l3 = L3Cache(L3CacheConfig(
    redis_url="redis://localhost:6379/0",
    key_prefix="tenant_a:",
    ttl_seconds=86400,
    tls_enabled=False,
    socket_timeout=5.0,
))

TLS Support: Use rediss:// URL scheme or set tls_enabled=True.

LLM Companion

from epochly.inference.serving.llm_companion import (
    LLMCompanionAdapter,
    LLMCompanionConfig,
)
companion = LLMCompanionAdapter(
    runtime_url="http://localhost:8000",
    config=LLMCompanionConfig(
        max_concurrent_requests=64,
        cache_enabled=True,
        cache_max_size=10_000,
        default_timeout_seconds=120.0,
        api_path="/v1/completions",
    ),
)

Ray Serve Wrapper

from epochly.inference.serving.ray_serve_wrapper import (
    EpochlyRayServeWrapper,
    RayServeConfig,
    epochly_serve,
)
# Via wrapper
wrapper = EpochlyRayServeWrapper(RayServeConfig(
    enable_telemetry=True,
    enable_priority_routing=True,
))
# Via decorator
@epochly_serve(config=RayServeConfig())
def predict(model, data):
    return model(data)

Security Validator

from epochly.inference.security import SecurityValidator
validator = SecurityValidator(max_input_size_bytes=10 * 1024 * 1024)
report = validator.run_all_checks(
    cache=inference_cache,
    privacy_mode="ephemeral",
    encryption_key_present=False,
    redaction_patterns_configured=False,
    config={"canary_enabled": True},
)

Compilation Safety Monitor

from epochly.inference.compilation.safety_monitor import TorchCompileSafetyMonitor
monitor = TorchCompileSafetyMonitor(
    max_graph_breaks=5,
    memory_growth_threshold_mb=500.0,
)
# Pre-compile check
result = monitor.pre_compile_check(model, sample_input)
# Post-compile health
health = monitor.check_health(pre_mb=1000, post_mb=1050)
# Output validity
validity = monitor.check_output_validity(model_output)

Prometheus Endpoint

Expose a Prometheus-compatible /metrics endpoint using the PrometheusExporter:

from fastapi import FastAPI
from fastapi.responses import PlainTextResponse
from epochly.inference.metrics.prometheus_exporter import PrometheusExporter
from epochly.inference.metrics.inference_metrics import InferenceMetrics
from epochly.inference.metrics.cost_estimator import CostEstimator
app = FastAPI()
inference_metrics = InferenceMetrics()
cost_estimator = CostEstimator(gpu_name="A100_80GB")
@app.get("/metrics")
def metrics():
    return PlainTextResponse(
        content=PrometheusExporter.export(
            inference_metrics,
            cost=cost_estimator,
            model_labels={
                id(model): {
                    "model_name": "bert-base",
                    "model_version": "v2.1",
                    "endpoint": "/predict",
                }
            },
        ),
        media_type="text/plain; version=0.0.4",
    )

The exporter generates metrics in the standard Prometheus exposition format, including counters, gauges, and histograms.