Emergency Controls

Production safety mechanisms for quickly disabling Epochly optimization.

Overview

Epochly provides multiple emergency controls to quickly disable optimization in production environments when issues arise. These safety mechanisms allow you to immediately return to baseline Python performance without code changes or redeployment.

Key Safety Features:

Instant disable via environment variables
Multiple levels of control granularity
No application restart required for some methods
Zero overhead when emergency disabled
Cannot be accidentally overridden

Emergency Disable

EPOCHLY_EMERGENCY_DISABLE=1

Highest priority emergency stop - Use when immediate action is required:

export EPOCHLY_EMERGENCY_DISABLE=1
# Restart application
systemctl restart myapp

Characteristics:

Immediate effect: Takes effect on next process start
Highest priority: Cannot be overridden by any other setting
Complete bypass: Completely bypasses all Epochly initialization
Zero overhead: Epochly code never executes
Use for: Production regressions, critical issues, emergency rollback

EPOCHLY_DISABLE=1

Standard disable - Use for controlled disabling:

export EPOCHLY_DISABLE=1
# Restart application
systemctl restart myapp

Characteristics:

Prevents initialization: Stops new Epochly instances from starting
Lower priority: Can be overridden by explicit epochly.configure()
Minimal overhead: Small initialization check only
Use for: Testing, debugging, controlled rollback

Comparison Table

Feature	EPOCHLY_EMERGENCY_DISABLE=1	EPOCHLY_DISABLE=1
Priority	Highest - cannot override	Medium - can override in code
Performance	Zero overhead	Minimal overhead
Use Case	Critical production issues	Testing/debugging
Override	Cannot be overridden	Can override with `configure()`
Effect	Complete bypass	Prevents new initialization
Restart Required	Yes	Yes

CLI Emergency Stop

Quick Stop Commands

# Emergency stop (sets EPOCHLY_EMERGENCY_DISABLE=1)
epochly-deploy emergency-stop
# Standard stop (sets EPOCHLY_DISABLE=1)
epochly stop

Recovery Procedures

After Emergency Disable

# 1. Remove emergency disable
unset EPOCHLY_EMERGENCY_DISABLE
# 2. Restart application
systemctl restart myapp
# 3. Verify status
curl http://localhost:8080/health/epochly

After Standard Disable

# 1. Remove disable
unset EPOCHLY_DISABLE
# 2. Restart application
systemctl restart myapp
# 3. Check Epochly status
python -c "import epochly; print(epochly.get_status())"

Gradual Re-enablement

# Step 1: Enable monitoring only (no optimization)
export EPOCHLY_MODE=monitoring
export EPOCHLY_LEVEL=0
systemctl restart myapp
# Step 2: After monitoring looks good, enable conservative
export EPOCHLY_MODE=conservative
export EPOCHLY_LEVEL=2
systemctl restart myapp
# Step 3: After testing, enable balanced
export EPOCHLY_MODE=balanced
export EPOCHLY_LEVEL=2
systemctl restart myapp

Mode-Based Controls

Monitoring Only (No Optimization)

# Disable optimization but keep monitoring
export EPOCHLY_MODE=monitoring
export EPOCHLY_LEVEL=0

This collects performance metrics without applying any optimization.

Cap at Specific Level

# Limit to threading only (no JIT or multicore)
export EPOCHLY_LEVEL=1
# Limit to threading + JIT (no multicore)
export EPOCHLY_LEVEL=2

Feature-Specific Disables

Disable JIT Compilation

export EPOCHLY_JIT_ENABLED=0

Disables Level 2 JIT compilation while keeping threading (Level 1).

Disable Multicore/ProcessPool

export EPOCHLY_DISABLE_PROCESSPOOL_FALLBACK=1

Prevents Level 3 from using ProcessPool (falls back to Level 2).

Disable GPU Acceleration

export EPOCHLY_GPU_ENABLED=0

Disables Level 4 GPU acceleration (falls back to Level 3).

Disable Auto-Initialization

export EPOCHLY_DISABLE_AUTO_INIT=1

Prevents Epochly from auto-initializing via sitecustomize.

Disable Import Interception

export EPOCHLY_DISABLE_INTERCEPTION=1

Disables automatic import interception and optimization.

Programmatic Controls

Setting Environment Variables in Code

import os
# Emergency disable before importing epochly
os.environ['EPOCHLY_EMERGENCY_DISABLE'] = '1'
import epochly
# Epochly will not initialize

Check Emergency State

import epochly
status = epochly.get_status()
print(f"Emergency Disabled: {status.get('emergency_disable', False)}")
print(f"Regular Disabled: {status.get('regular_disable', False)}")
print(f"Enabled: {status.get('enabled', False)}")

Emergency Disable Function

import epochly
# Programmatically trigger emergency disable
epochly.emergency_disable()
# Check status
assert not epochly.get_status()['enabled']

Context Manager

import epochly
# Temporarily disable Epochly
with epochly.epochly_disabled_context():
    # Epochly is disabled in this block
    result = compute_baseline(data)
# Epochly re-enabled after context

DeploymentController Class

from epochly import DeploymentController
controller = DeploymentController()
# Emergency disable
controller.emergency_disable()
# Check state
status = controller.get_status()
print(f"Emergency mode: {status['emergency_disabled']}")
# Re-enable
controller.enable()

When to Use Emergency Controls

Decision Flowchart

Issue Detected
    |
    v
Critical/Production?
    |
    +-- Yes --> Use EPOCHLY_EMERGENCY_DISABLE=1
    |           (Immediate, cannot override)
    |
    +-- No --> Need to test/debug?
                |
                +-- Yes --> Use EPOCHLY_DISABLE=1
                |           (Standard disable)
                |
                +-- No --> Want metrics only?
                            |
                            +-- Yes --> Use EPOCHLY_MODE=monitoring
                            |
                            +-- No --> Cap specific feature?
                                        |
                                        Use feature-specific disable

Use EPOCHLY_EMERGENCY_DISABLE When:

Production regression detected: Performance worse than baseline
Unexpected crashes: Application crashes potentially related to Epochly
Debugging critical issue: Need to rule out Epochly immediately
Immediate baseline needed: Must return to pure Python immediately
Security concern: Suspected security issue requires immediate action

Use EPOCHLY_DISABLE When:

Preventing activation: Want to prevent Epochly from starting in new processes
CI/CD pipeline: Disable for testing or specific build stages
Development debugging: Testing without Epochly during development
Gradual rollout: Controlling which environments have Epochly active

Use EPOCHLY_MODE=monitoring When:

Want metrics without optimization: Collect performance data for analysis
Profiling baseline: Measuring current performance before optimization
Comparison testing: A/B testing optimized vs baseline performance

Monitoring Emergency State

Python API

import epochly
status = epochly.get_status()
# Check various disable states
if status.get('emergency_disable'):
    print("⚠️  EMERGENCY DISABLE ACTIVE")
elif status.get('regular_disable'):
    print("ℹ️  Standard disable active")
elif not status.get('enabled'):
    print("Epochly not enabled")
else:
    print(f"✓ Epochly active at Level {status['enhancement_level']}")

CLI Status Check

epochly status
# Output:
# Status: Emergency Disabled
# EPOCHLY_EMERGENCY_DISABLE: 1
# Recommendation: Remove emergency disable to re-enable

Bash/Shell Check

# Check emergency disable
if [ "$EPOCHLY_EMERGENCY_DISABLE" = "1" ]; then
    echo "Emergency disable active"
fi
# Check all Epochly environment variables
env | grep EPOCHLY

Production Runbook

Scenario 1: Performance Regression Detected

# 1. Immediate response - Emergency disable
export EPOCHLY_EMERGENCY_DISABLE=1
systemctl restart myapp
# 2. Verify baseline performance
# Monitor metrics for 10-15 minutes
# 3. Capture diagnostics
epochly doctor > /tmp/epochly-diagnostics.txt
# 4. If baseline performance acceptable
#    Issue is with Epochly configuration
# 5. Begin gradual re-enablement
unset EPOCHLY_EMERGENCY_DISABLE
export EPOCHLY_MODE=monitoring
export EPOCHLY_LEVEL=0
systemctl restart myapp

Scenario 2: Unexpected Errors

# 1. Capture current state before disabling
python -c "import epochly; print(epochly.get_status())" > /tmp/epochly-state.json
epochly doctor > /tmp/epochly-diagnostics.txt
# 2. Check error logs
tail -100 /var/log/myapp.log > /tmp/error-context.log
# 3. Emergency disable
export EPOCHLY_EMERGENCY_DISABLE=1
systemctl restart myapp
# 4. Monitor for error resolution
# If errors stop: Issue is Epochly-related
# If errors continue: Not Epochly-related
# 5. Report issue with diagnostics if Epochly-related

Scenario 3: Gradual Re-enablement

After emergency disable, gradually restore optimization:

# Phase 1: Monitoring only (0-24 hours)
unset EPOCHLY_EMERGENCY_DISABLE
export EPOCHLY_MODE=monitoring
export EPOCHLY_LEVEL=0
systemctl restart myapp
# Monitor: No optimization, collect baseline metrics
# Phase 2: Conservative mode (24-48 hours)
export EPOCHLY_MODE=conservative
export EPOCHLY_LEVEL=2
systemctl restart myapp
# Monitor: Safe optimizations only
# Phase 3: Balanced mode (48+ hours)
export EPOCHLY_MODE=balanced
export EPOCHLY_LEVEL=2
systemctl restart myapp
# Monitor: Full production optimization

Docker/Kubernetes Emergency Controls

Docker Run

# Emergency disable on Docker run
docker run -e EPOCHLY_EMERGENCY_DISABLE=1 myapp
# Standard disable
docker run -e EPOCHLY_DISABLE=1 myapp
# Monitoring only
docker run -e EPOCHLY_MODE=monitoring -e EPOCHLY_LEVEL=0 myapp

Docker Compose

version: '3.8'
services:
  app:
    image: myapp
    environment:
      # Emergency disable
      - EPOCHLY_EMERGENCY_DISABLE=1
      
      # Or standard disable
      # - EPOCHLY_DISABLE=1
      
      # Or monitoring only
      # - EPOCHLY_MODE=monitoring
      # - EPOCHLY_LEVEL=0

Kubernetes ConfigMap

Create emergency disable ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: epochly-emergency-config
  namespace: production
data:
  EPOCHLY_EMERGENCY_DISABLE: "1"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  template:
    spec:
      containers:
      - name: app
        envFrom:
        - configMapRef:
            name: epochly-emergency-config

Apply Emergency Disable to Kubernetes

# 1. Create emergency ConfigMap
kubectl create configmap epochly-emergency-config   --from-literal=EPOCHLY_EMERGENCY_DISABLE=1   -n production
# 2. Update deployment to use emergency config
kubectl patch deployment app -n production   --patch '{"spec":{"template":{"spec":{"containers":[{"name":"app","envFrom":[{"configMapRef":{"name":"epochly-emergency-config"}}]}]}}}}'
# 3. Rollout restart
kubectl rollout restart deployment/app -n production
# 4. Monitor rollout
kubectl rollout status deployment/app -n production
# 5. Verify emergency disable active
kubectl exec -it deployment/app -n production --   python -c "import os; print('Emergency:', os.environ.get('EPOCHLY_EMERGENCY_DISABLE'))"

Remove Emergency Disable in Kubernetes

# 1. Delete emergency ConfigMap
kubectl delete configmap epochly-emergency-config -n production
# 2. Revert to normal config
kubectl patch deployment app -n production   --patch '{"spec":{"template":{"spec":{"containers":[{"name":"app","envFrom":[{"configMapRef":{"name":"epochly-config"}}]}]}}}}'
# 3. Rollout restart
kubectl rollout restart deployment/app -n production

Best Practices

Document procedures: Keep emergency disable procedures in runbooks
Test in staging: Practice emergency disable procedures before production incidents
Monitor metrics: Set up alerts for performance regressions
Have rollback plan: Always have EPOCHLY_EMERGENCY_DISABLE=1 ready
Gradual re-enablement: Never jump directly back to aggressive optimization
Capture diagnostics: Run epochly doctor before and after disabling
Communication: Alert team when emergency controls are activated
Root cause analysis: Investigate why emergency disable was needed