AI Inference

Epochly AI Inference: Architecture

The inference accelerator: framework detection, the small automatic wrap surface, and opt-in serving, safety, and metrics components you compose yourself.

How the inference accelerator is shaped: framework detection, the small automatic wrap surface, and the opt-in serving, safety, and metrics components you compose yourself.

System Overview

The Epochly AI Inference Accelerator is a set of inference serving, observability, and safety components for Python services. The automatic surface is deliberately small today: epochly.wrap() returns a zero-overhead passthrough proxy for framework models (PyTorch / Transformers / ONNX Runtime) and applies measured memoization on the generic route. Everything else on this page -- micro-batching, compilation, caching, safety validation, A/B testing, the model registry, and metrics export -- ships as components you construct and wire explicitly; nothing orchestrates them automatically.

> Not yet implemented: an optimization orchestrator that routes wrapped

> models through batching / compilation / caching behind the safety gates.

> That wiring is planned for a future release; this page marks each planned

> integration where it appears.

The Automatic Surface: epochly.wrap()

epochly.wrap(obj)
    |
    +-- framework model (torch.nn.Module / transformers Pipeline /
    |   PreTrainedModel / onnxruntime.InferenceSession)
    |     -> InferenceProxy passthrough: calls delegate directly to the
    |        model with zero added per-call overhead; no profiling,
    |        counting, or optimization on this route today
    |
    +-- generic object (.predict/.transform/.generate/.encode/__call__)
    |     -> GenericModelAdapter + InferenceWrapOptimizer: measured
    |        memoization of deterministic repeat calls (only measured
    |        cache-hit speedups are recorded)
    |
    +-- plain function / method
          -> Level 2 JIT decorator route (structured unavailable reason
             when the route cannot be realized)

EPOCHLY_INFERENCE_ENABLED=0 disables the whole surface: wrap() returns its input unchanged (see Configuration).

Component Architecture

    Customer's Python Service
    (FastAPI / Ray Serve / custom)
              |
    +=========+====================+  <-- Data Plane (in-process)
    |                               |
    | Serving Adapters              |  Request-level interception
    |  EpochlyInferenceMiddleware   |  ASGI: timing, admission control
    |  EpochlyRayServeWrapper       |  Deployment wrapper (v1 preview)
    |  LLMCompanionAdapter          |  vLLM/TGI: admission, caching
    |  ABModelComparison            |  A/B split/shadow traffic routing
    |  GradualRollout               |  Staged ramping w/ health checks
    |                               |
    | Framework Adapters            |  Model-level interception (wrap)
    |  PyTorchAdapter               |  InferenceProxy(Module.__call__)
    |  TransformersAdapter          |  InferenceProxy(pipeline.__call__)
    |  OnnxAdapter                  |  InferenceProxy(session.run)
    |                               |
    +=========+====================+
    Standalone component library (constructed and wired by you; no
    automatic orchestration exists today):
    ModelRegistryClient        Model loading, versioning, lineage
    SafetyOrchestrator         gate_optimization() + golden capture
    DynamicMicroBatcher        Async micro-batching (opt-in)
    ModelCompiler              torch.compile with pre-check (opt-in)
    InferenceCache / L2Cache / L3Cache   Independent cache tiers
    ValidatorRegistry          Custom + built-in workload validators
    InferenceMetrics           Manual metrics collector
    CostEstimator              Cost attribution from recorded GPU-seconds
    PrometheusExporter         --> /metrics exposition text (you host it)
    InferenceOTelExporter      --> OpenTelemetry instrument dict

Data Plane

In-process components that intercept and instrument inference traffic:

Serving Adapters: ASGI middleware, Ray Serve wrappers, LLM companion
A/B Testing: Split and shadow mode traffic routing between model variants, with Welch's t-test statistical significance and gradual rollout
Framework Adapters: Model-specific proxies for PyTorch, HuggingFace, ONNX
Inference Proxies: Wraps individual model instances at the serving boundary

Model Registry

Version-tracked model loading from multiple backends (local, HuggingFace Hub, MLflow). Provides:

Version tracking: SHA-256 content hashing with automatic cache invalidation signaling
Lineage tracking: Provenance metadata (data version, code hash, hyperparameters, training metrics)
Staged promotions: Lifecycle management (DRAFT -> STAGING -> PRODUCTION -> ARCHIVED) with optional approval callbacks and TOCTOU-safe atomic transitions
Rollback: Revert to previous production version with full version history

Configuration

Components are configured per instance at construction (MicroBatcherConfig, CompilationConfig, CacheConfig, LLMCompanionConfig, ...). See the Configuration reference for the delivered surface.

> Not yet implemented: a policy engine with per-endpoint configuration

> (batching window, precision mode, concurrency cap, cache TTL, fallback

> runtime, rollout percentage). No such subsystem exists in the current

> release; configuration is per-component and programmatic.

Validation Harness

Safety controls: golden datasets, canary validation, circuit breakers, drift monitoring, fallback chains, validator registry, hysteresis control. These ship as composable components -- see Safety Architecture for what is and is not wired.

Metrics Export

InferenceMetrics is a manual collector: you record requests and inferences into it and export via PrometheusExporter (exposition text for a /metrics endpoint you host) or InferenceOTelExporter (an OpenTelemetry instrument dict).

> Not yet implemented: automatic export of inference metrics to the

> Lens dashboard. Inference metrics do not flow to Lens from this codebase

> today; Prometheus/OpenTelemetry export that you wire yourself is the

> delivered path.

Module Structure

src/epochly/inference/
    __init__.py              # Lazy init, zero cost if no ML framework
    detector.py              # Framework and model detection
    profiler.py              # Inference-specific profiling + golden capture
    optimizer.py             # PrePostOptimizer: opt-in pre/post-processing
                             #   (tokenizer caching, CPU parallelization)
    wrap_optimizer.py        # InferenceWrapOptimizer: generic-route
                             #   measured memoization used by epochly.wrap()
    cache.py                 # L1 in-memory cache
    cache_l2.py              # L2 SQLite WAL + AES-256-GCM cache
    cache_l3.py              # L3 Redis distributed cache
    config.py                # InferenceConfig dataclass
    context.py               # RequestContext, BatchKey
    security.py              # SecurityValidator: manual security checks
                             #   (not invoked automatically at startup)
    progression.py           # Level-progression criteria (no runtime
                             #   consumer in the current release)
    batch/
        dynamic_micro_batcher.py  # Async micro-batching with keyed sub-queues
        batch_optimizer.py
        request_queue.py
    compilation/
        torch_compiler.py    # torch.compile with pre-check and cache
        safety_monitor.py    # Graph break, memory, NaN monitoring
    frameworks/
        base_adapter.py
        inference_proxy.py
        pytorch_adapter.py
        transformers_adapter.py
        onnx_adapter.py
        generic_adapter.py   # Structural route for .predict/.transform/
                             #   .generate/.encode/__call__ surfaces
    registry/
        model_registry.py    # ModelRegistryClient, ModelStage, ModelLineage
    serving/
        ab_testing.py        # ABModelComparison, MultiModelComparison, GradualRollout
        fastapi_middleware.py
        fastapi_dependency.py
        llm_companion.py     # vLLM/TGI control layer
        ray_serve_wrapper.py # Ray Serve wrapper (v1 preview)
    metrics/
        inference_metrics.py
        cost_estimator.py
        otel_exporter.py         # OpenTelemetry instrument mapping
        prometheus_exporter.py   # Prometheus exposition format export
    safety/
        canary_validator.py
        circuit_breaker.py
        drift_monitor.py         # EWMA-based online drift detection
        fallback_chain.py
        golden_store.py
        hysteresis.py            # Anti-flapping state transition control
        privacy.py               # InputRedactor, TenantIsolation, AuditLogger
        safety_orchestrator.py
        validator_registry.py    # Custom workload validator plugin registry

Enhancement Levels

The level taxonomy is the product's roadmap vocabulary. What ships today: framework detection is live (import hooks), and the only live automatic optimization is the generic-route memoization applied by epochly.wrap(). The L1/L2a/L2b building blocks ship as standalone, manually constructed components.

Level	Name	Status in the current release
L0	Profiling	Framework detection live via import hooks. The `wrap()` framework route is a zero-overhead passthrough and does not profile calls today; `InferenceProfiler` is a component you construct and attach directly.
L1	Pre/Post Optimization	`PrePostOptimizer` (tokenizer caching, CPU parallelization) ships as an opt-in component; not wired into `wrap()`.
L2a	Micro-Batching	`DynamicMicroBatcher` ships as an opt-in async component; not wired into `wrap()`.
L2b	Compilation	`ModelCompiler` (torch.compile with pre-check and golden validation hooks) ships as an opt-in component; no product path invokes it.
L3+	Verified Optimize	Not implemented (planned: quantization, ONNX export).

> Not yet implemented: automatic level progression. Wiring these

> components into epochly.wrap() behind an optimization orchestrator is

> planned for a future release; today you construct and drive each

> component directly. progression.py defines the progression criteria,

> but nothing consumes them yet.

Cache Architecture

Three independently constructible cache tiers with a latency/capacity tradeoff:

L1 (In-Memory LRU)     L2 (SQLite WAL)        L3 (Redis)
- ~1us access           - ~1ms access           - ~5ms access
- 10K entries           - 1GB on disk           - Distributed
- Process-local         - AES-256-GCM           - TLS + key prefix
- Thread-safe           - TTL enforcement        - TTL via EXPIRE

Each tier is a standalone class -- InferenceCache (L1), L2Cache, L3Cache -- that you construct and call get/put on directly. The tiers are not chained: a miss in one tier does not consult another, and nothing wires any tier into epochly.wrap().

> Not yet implemented: tiered lookup (L1 -> L2 -> L3) with promotion of

> hits into faster tiers, and automatic cache integration on the wrap()

> path. Both are planned; in the current release each tier is manual and

> independent.

Safety Architecture

The safety stack ships as composable components. No automatic optimization currently flows through them: the only live optimization (the wrap() generic-route memoization) does not pass through this pipeline, and the components gate an optimization only where you wire them in (for example, passing a SafetyOrchestrator to ModelCompiler.compile_async).

The gate sequence the components implement, in the order a wired deployment would compose them:

Golden output capture via the SafetyOrchestrator golden callback on an InferenceProfiler you attach (epochly.wrap() does not attach a profiler today)
Pre-compile check via torch._dynamo.explain()
Canary validation with workload-specific validators (built-in or custom via ValidatorRegistry)
Circuit breaker per optimization
Drift monitoring via EWMA-based DriftMonitor with shadow comparisons
Hysteresis control to prevent state flapping (asymmetric enable/disable thresholds)
Fallback chain for graceful degradation
Alert hooks via ValidatorRegistry alert callbacks on FAIL reports

> Not yet implemented: running optimizations through this pipeline

> automatically. That requires the future optimization orchestrator; in the

> current release the pipeline is assembled per deployment by the user.

Validator Registry

The ValidatorRegistry provides a plugin interface for custom workload validators. Built-in validators cover standard workload types (embedding, classifier, reranker, generation, encoder). Enterprise users can register domain-specific validators (financial accuracy, medical terminology, etc.) that override built-in validators for the same workload type.

Key properties:

Custom validators override built-in validators; unregistering restores the original
Supports both synchronous and asynchronous validators
Alert callbacks fire on FAIL results for integration with PagerDuty, Slack, etc.
InputSchemaValidator provides pre-inference input data validation (dtype, shape, range)

Drift Monitoring

The DriftMonitor samples live traffic using shadow execution, computes workload-specific comparison metrics via the same validators as canary validation, and applies EWMA smoothing for streaming drift detection. When drift exceeds thresholds, the circuit breaker trips and the optimization is disabled.

Hysteresis Control

The HysteresisController prevents optimization state oscillation:

Disable: Immediate on circuit breaker trip
Re-enable: Requires minimum off-duration (default 300s) AND N consecutive canary PASS results (not MARGINAL)
Minimum on-duration: Prevents disabling too quickly before meaningful data is collected (default 60s)

Model Registry Architecture

ModelRegistryClient
    |
    +-- Backend Loaders
    |     +-- local:       File read + SHA-256 hash
    |     +-- huggingface: transformers.AutoModel.from_pretrained
    |     +-- mlflow:      mlflow.pytorch.load_model
    |
    +-- Version Tracking
    |     +-- SHA-256 content hashing
    |     +-- on_version_change callback for cache invalidation
    |
    +-- Lineage Tracking (ModelLineage)
    |     +-- data_version, code_hash, hyperparameters
    |     +-- training_metrics, parent_model
    |
    +-- Staged Promotions (ModelStage)
    |     +-- DRAFT -> STAGING -> PRODUCTION -> ARCHIVED
    |     +-- Optional approval callbacks
    |     +-- TOCTOU-safe atomic verify+apply
    |     +-- Single PRODUCTION version per model (auto-archive)
    |
    +-- Rollback
          +-- Production version history (deque, max 100)
          +-- Archive current, restore previous

A/B Testing Architecture

ABModelComparison
    |
    +-- Split Mode: Route to A or B per request
    |     +-- Random selection based on traffic_split probability
    |     +-- Per-model latency and error metrics
    |
    +-- Shadow Mode: Run both, return A
    |     +-- B runs after A (no latency impact on critical path)
    |     +-- Both results recorded for comparison
    |
    +-- Statistical Analysis
          +-- Welch's t-test (scipy or manual fallback)
          +-- Confidence intervals
          +-- Sample size tracking
MultiModelComparison (A/B/n)
    +-- Weighted traffic distribution across N models
    +-- Cumulative weight routing for efficient selection
GradualRollout
    +-- Stepped traffic ramping: 1% -> 5% -> 10% -> 25% -> 50% -> 100%
    +-- Health check gates at each step
    +-- Minimum dwell time enforcement per step

Metrics Architecture

InferenceMetrics (core collector)
    |
    +-- PrometheusExporter.export()  --> /metrics HTTP endpoint
    |     +-- Counters, gauges, histograms
    |     +-- Per-model labels (model_name, model_version, endpoint)
    |     +-- Exemplars for trace correlation
    |     +-- Cost metrics (baseline vs optimized)
    |
    +-- InferenceOTelExporter.export()  --> OTel SDK
    |     +-- Spec Section 8.1 instrument names
    |     +-- Safety metrics (canary, circuit breaker, drift, hysteresis)
    |
    +-- CostEstimator
          +-- Per-1k request cost (baseline vs optimized)
          +-- Projected hourly/monthly savings

Security Model

SecurityValidator provides security checks that you run explicitly -- for example from your own service-startup path -- via run_all_checks(...):

Cache tenant isolation
Privacy control configuration (mode, redaction patterns, audit logging)
Safety bypass resistance
Input size limits

Nothing invokes these checks automatically: the current release has no startup hook that runs them, so call run_all_checks yourself if you want startup validation.

Privacy controls:

InputRedactor: Regex PII scrubbing before any data is stored
TenantIsolation: Namespace cache keys by tenant/deployment ID
AuditLogger: Append-only structured log of safety decisions