Activation Thresholds

Epochly uses empirically-validated thresholds to determine when to activate each optimization level. This document explains these thresholds and how to tune them.

Overview

Activation thresholds control when Epochly enables different optimization strategies. These thresholds are based on benchmarks across thousands of workloads to ensure optimizations only apply when they provide real benefit.

Why Thresholds Matter

Optimization has overhead:

• Thread pool creation takes ~10ms
• JIT compilation takes ~100ms for first compile
• Worker spawning takes ~500ms
• GPU initialization takes ~200ms

If the workload doesn't exceed threshold minimums, the overhead exceeds the benefit.

Default Thresholds

Level 1: Threading Thresholds

Level 2: JIT Thresholds

Level 3: Multicore Thresholds

Level 4: GPU Thresholds

Configuring Thresholds

Via Environment Variables

# Level 2: JIT threshold
export EPOCHLY_JIT_HOT_PATH_THRESHOLD=500
# Level 4: GPU threshold
export EPOCHLY_GPU_WORKLOAD_THRESHOLD=5000000  # 5MB

Via TOML Configuration

[epochly.thresholds]
hot_path_threshold = 500
min_data_size = 50000
gpu_workload_threshold = 5000000

Via Runtime API

import epochly
# Set JIT threshold
epochly.configure(jit_hot_path_threshold=500)
# Set GPU threshold
epochly.configure(gpu_workload_threshold=5_000_000)

Threshold Tuning Guide

When to Lower Thresholds

Lower thresholds when:

• Your workload is known to be parallelizable
• You have dedicated compute resources
• Latency is less important than throughput
• You're willing to accept higher startup overhead

When to Raise Thresholds

Raise thresholds when:

• Running many small operations
• Low latency is critical
• Memory is constrained
• Running on shared infrastructure

Empirical Threshold Data

Level 1 Threading Benchmarks

Threading becomes beneficial when I/O wait time exceeds threading overhead:

Level 2 JIT Benchmarks

JIT compilation provides benefit after sufficient calls to amortize compile time:

Level 3 Multicore Benchmarks

Parallelization overhead depends on data size and transfer costs:

Level 4 GPU Benchmarks

GPU benefits depend heavily on data transfer costs:

Monitoring Threshold Effectiveness

Check Activation Statistics

import epochly
# Get threshold statistics
stats = epochly.get_threshold_stats()
print(f"JIT activations: {stats['jit_activations']}")
print(f"JIT skips (below threshold): {stats['jit_skips']}")
print(f"GPU activations: {stats['gpu_activations']}")
print(f"GPU skips (below threshold): {stats['gpu_skips']}")

Analyze Threshold Decisions

import epochly
# Enable threshold logging
epochly.configure(log_threshold_decisions=True)
# Run workload
result = my_function(data)
# Check decisions
decisions = epochly.get_threshold_decisions()
for d in decisions:
    print(f"{d.level}: {d.decision} - {d.reason}")

Best Practices

1. Start with Defaults

Epochly's defaults are based on extensive benchmarking. Only change them if you have measured evidence that different values work better for your workload.

2. Profile Before Tuning

import epochly
# Run profiling
with epochly.profile_context() as profile:
    result = my_function(data)
# Analyze results
if profile.suggests_lower_threshold('jit'):
    print("Consider lowering JIT threshold")

3. Test Both Directions

When tuning, test both higher and lower values:

import time
import epochly
for threshold in [100, 500, 1000, 2000, 5000]:
    epochly.configure(jit_hot_path_threshold=threshold)
    
    start = time.perf_counter()
    for _ in range(1000):
        my_function(data)
    elapsed = time.perf_counter() - start
    
    print(f"Threshold {threshold}: {elapsed:.3f}s")

4. Consider Workload Variability

If your workload varies significantly, use more conservative (higher) thresholds to avoid overhead on small operations.

5. Monitor in Production

Enable threshold telemetry to understand real-world activation patterns:

export EPOCHLY_TELEMETRY=true
export EPOCHLY_LOG_THRESHOLDS=true

Advanced: Custom Threshold Functions

For advanced use cases, you can provide custom threshold functions:

import epochly
def custom_gpu_threshold(data_size, operation_type):
    """Custom function to decide GPU offload."""
    if operation_type == 'matmul':
        return data_size > 5_000_000  # 5MB for matmul
    elif operation_type == 'fft':
        return data_size > 1_000_000  # 1MB for FFT
    else:
        return data_size > 10_000_000  # 10MB default
epochly.configure(gpu_threshold_function=custom_gpu_threshold)

Troubleshooting

Optimization Not Activating

1. Check current thresholds: epochly.get_config()
2. Check workload size matches threshold requirements
3. Enable debug logging: export EPOCHLY_DEBUG=true

Optimization Causing Slowdown

1. Workload may be below threshold - raise thresholds
2. Check for excessive overhead with epochly.profile_context()
3. Consider disabling specific levels: epochly.set_level(2)

Inconsistent Performance

1. Workload may be near threshold boundary
2. Consider raising threshold to avoid activation jitter
3. Use manual level control for critical paths