d/dx: Tech Stack¶

Getting Started¶

Prerequisites¶

Python 3.11+ (managed by uv — see below)
C++20 compliant compiler (gcc/clang)
CMake 3.25+
uv (curl -LsSf https://astral.sh/uv/install.sh | sh)

Installation¶

uv python install 3.11    # optional, any 3.11+ works — download a uv-managed Python (see note below)
uv sync                   # install everything

This creates a virtual environment at .venv/, installs all workspace packages in editable mode, and compiles the C++ engine via scikit-build-core.

Why uv python install? uv pulls packages from PyPI, which distributes pre-built wheels tagged for specific macOS versions. System/conda Pythons (especially on older Macs) may report an outdated macOS version, causing uv to build packages from source instead of using wheels. uv python install gives you a correctly-tagged Python that avoids this. Newer Macs (Apple Silicon) are unlikely to hit this issue.

This creates a virtual environment at .venv/ (override with UV_PROJECT_ENVIRONMENT), installs all workspace packages in editable mode, and compiles the C++ engine via scikit-build-core.

When to re-run `uv sync`¶

You don’t need to think about this most of the time. If something breaks after a git pull or branch switch — ImportError, missing module, C++ crash — just run uv sync and it will fix itself.

For day-to-day Python development, you just edit and run. Editable installs pick up changes immediately.

Run uv sync --package engine if you edited C++ source or bindings.cpp (Python-only changes never need a rebuild).

Managing dependencies¶

Use uv add to install packages — this installs the package and records it in the correct pyproject.toml in one step:

uv add --package forecasting requests    # adds to forecasting/pyproject.toml
uv add --package cloud boto3             # adds to cloud/pyproject.toml
uv remove --package forecasting requests # removes it

uv add pytest                            # adds to root pyproject.toml (shared across all packages)
uv add --dev ruff                        # adds as a dev dependency (not included in published wheels)

uv sync installs everything (root dependencies, all workspace member dependencies, and dev dependencies) into one venv.

Do not use pip install or conda install. Those install into the venv without updating pyproject.toml, so the dependency won’t be there for anyone else after uv sync.

Running scripts¶

Activate the virtual environment once and use python directly — no need to prefix every command with uv run:

source .venv/bin/activate
python scripts/run_backtest.py --config path/to/config.yaml

Or use uv run without activating:

uv run python scripts/run_backtest.py --config path/to/config.yaml

Run a Backtest (Local):

make backtest YAML=path/to/my_config.yaml

Run a Distributed Backtest (Modal):

make backtest-parallel YAML=path/to/my_config.yaml

IDE Setup¶

Point your IDE’s Python interpreter to .venv/bin/python in the repo root. All workspace imports will resolve correctly for autocompletion, go-to-definition, and linting.

Documentation¶

Build the docs:
```
make docs
```

View: Open docs/build/html/index.html in your browser:

open -a "Google Chrome" docs/build/html/index.html

Documentation Stack¶

The project uses a unified documentation system:

Sphinx: Main documentation generator (Python-first).
Doxygen: Parses C++ source code to XML.
Breathe & Exhale: Bridges Doxygen XML into Sphinx, allowing seamless C++/Python documentation in a single site.

0. Our thesis - the autonomous quant firm¶

Most AI trading attempts fail because they focus on slow frequencies with low signal-to-noise ratios. Embarcadero targets High-Frequency Trading (HFT) where:

High Signal-to-Noise: Rewards are dense and immediate.
Machine Intuition: Success comes from learning patterns directly from raw data, not human financial theory.
Scale over Headcount: We replace human Quantitative Researchers (QRs) with self-improving LLM agents.

We don’t have LLMs trade directly (which is slow); we have Frontier Models (~GPT-5) write the C++ and Python code for smaller trading models (~GPT-2) or simpler models, like linear regression. This bypasses inference latency and leverages the model’s peak capability: code generation.

System Architecture: The Bilevel Loop¶

The system is bifurcated into two entities operating on different timescales:

The Outer Loop (The Agent): A reasoning agent that acts as a Virtual QR. It generates hypotheses, writes C++ features, and manages the research process.
The Inner Loop (The Engine): A high-performance C++ core that executes the order book, processing 10M+ events per second to evaluate the agent’s code.

1. System design¶

This system is a Hybrid HFT Backtesting Engine designed to process Level 3 (Market-By-Order) data with extreme throughput while retaining the flexibility of Python for research.

Why hybrid Python & C++?¶

Python (Orchestration & Data): Python excels at data loading (Polars/Pandas/Numpy), vectorised operations (Target Generation), and configuration management.
C++ (Hot Path): C++ is required for the event loop. Processing 10+ million events tick-by-tick, maintaining the state of a Limit Order Book (LOB), and computing recursive features cannot be done efficiently in pure Python loops.

Performance Key: Zero-Copy¶

The system uses pybind11 to pass raw pointers from Python-allocated Numpy arrays directly to the C++ engine. No data is copied during the transition.

Python loads data -> numpy.ndarray (contiguous memory).
C++ receives int64_t*, int8_t* pointers.
C++ iterates over pointers -> writes results to output pointer.

2. The Engine: Performance & Safety¶

To enable an agent to “play” in the HFT environment, our engine must be both performant and sandboxed.

2.1 Input Format¶

Source: Databento (.dbn.zst compressed).
Schema: Market-By-Order (MBO).
Columns: ts_recv (uint64), price (int64), size (uint32), action (char), side (char), order_id (uint64), flags (uint8).

2.2 The Loader (`forecasting/data/loader.py`)¶

The DataLoader class is critical for performance. It converts the structured dictionary returned by Databento into contiguous numpy arrays.

Critical Step: np.ascontiguousarray().
- Why? Standard slicing in numpy creates strided views. C++ expects dense, packed memory. If we pass a strided array, the system would segfault or read garbage.
- The Loader ensures memory alignment before the C++ engine ever sees the data.

2.3 Zero-Copy Data Pipeline¶

The system uses pybind11 to pass raw pointers from Python-allocated Numpy arrays directly to the C++ engine. No data is copied during the transition.

Memory Alignment: The DataLoader ensures arrays are contiguous via np.ascontiguousarray().
Throughput: By avoiding data copies, the agent can run thousands of backtests per hour to iterate on alpha ideas.

2.4 Feature System & Metaprogramming¶

The engine uses a Visitor Pattern for features. New features are auto-discovered at compile-time via a registration script.

Agent Interaction: When an agent writes a new .hpp file, our generate_registry.py script automatically exposes it to the Python environment without manual human intervention.

2.5 The Sandbox (Agent Safety)¶

Allowing an LLM to write C++ is dangerous. Our agent_interface acts as a Gatekeeper:

Static Analysis: Rejects code containing dangerous system calls like system(), fork(), or unauthorized network headers.
Micro-Testing: Compiles the agent’s feature in a standalone environment and runs it against a mock LOB.
Feedback: If it segfaults, the error is fed back to the agent as a “Compiler Error” or “Runtime Exception,” allowing the agent to self-correct its code.

3. The C++ Engine Internals (`forecasting/engine/core`)¶

The header files in forecasting/engine/core/ define the simulation.

3.1 `OrderBook.hpp`¶

This class maintains the state of the Book. It uses a Dual-Index approach to support O(1) order modification (typical of MBO data) and efficient Depth access.

L3 (Order) Index: std::unordered_map<uint64_t, Order> orders
- Maps Order ID -> {Price, Size, Side, BirthTick}.
- Allows handling MODIFY and CANCEL messages instantly without searching price levels.
L2 (Price) Index: std::map<int64_t, int64_t> bids/asks
- Maps Price -> Total Volume.
- Kept sorted by price (std::map is a Red-Black tree).
- Allows efficient iteration for get_depth_volume(k).

3.2 Feature System (`Feature.hpp`, `FeatureRegistry.hpp`)¶

Features are implemented using the Visitor Pattern.

Interface: class Feature { virtual void compute(const OrderBook&, double* out) = 0; }
Execution:
1. Engine initializes a vector of Feature*.
2. For every event t:
  - OrderBook.process(event)
  - For each feature: feature->compute(book, &results[t])

3.3 Extensibility¶

To add a new C++ feature:

Create forecasting/engine/features/MyNewFeature.hpp.
Inherit from Feature.
Implement compute().
Register it in FeatureRegistry.hpp (or use the generation script).

3.4 Metaprogramming (Feature Registration)¶

The system uses a compile-time registration script to avoid manually updating the engine every time a new feature is added.

Script: forecasting/engine/generate_registry.py
Workflow:
1. Scans forecasting/engine/features/*.hpp.
2. Parses special Doxygen-style tags: * @Feature: name(param_default).
3. Auto-generates forecasting/engine/core/FeatureRegistry.hpp.

Example Annotation (C++ Header):

// In features/MyTrend.hpp
/**
 * @Feature: trend
 */
class MyTrend : public Feature { ... }

Generated Factory Code:

if (name == "trend")
    return std::make_unique<MyTrend>(name, params);

Benefit: Developers only need to add a file in features/ and run build.sh. The build process automatically detects the new feature and exposes it to Python.

4. The Python Wrapper & Target Generation¶

The C++ Engine is wrapped by forecasting/engine/wrapper.py. This is not just a passthrough; it contains logic that is better suited for Python.

4.1 Logic Split¶

Features (C++): Anything that depends on the current state of the book (e.g., Depth Imbalance, OFI) is done in C++.
Targets (Python): Calculation of future returns (e.g., “Return 1s ahead”) is done here.
- Why? Calculating Price[t + 1s] involves looking ahead. In C++, this requires buffering or two-pass processing. In Python/Pandas, this is a simple pd.merge_asof (vectorised binary search) which is extremely fast and readable.

4.2 Wrapper Flow¶

Filter Config: Separate “Features” (sent to C++) from “Targets”.
Sanitize: Ensure C++ only receives float/int parameters (no strings).
Run C++: Call _engine_cpp.run_simulation().
Compute Targets: Use target.py helpers to merge future prices back to current timestamps.

5. Configuration Reference¶

Experiments are defined in YAML. This allows reproducible research with as little unnecessary compiling as possible.

YAML Schema¶

id: "experiment_name"
data_paths:                   # Supports multiple files for distributed/batch running
  - "/data/day1.dbn.zst"
  - "/data/day2.dbn.zst"

target_s: [10, 60]            # Target Horizons in Seconds

features:
  - id: "depth_imbalance"     # C++ Feature ID
    params:
      depth_levels: 5

6. Agentic Feature Engineering¶

The forecasting engine includes a dedicated workflow for AI Agents (like Gemini) to safely write and add C++ features to the engine.

6.1 Design Philosophy: Safety First¶

Safety is paramount when allowing AI to write C++ code. We implement a rigorous Sandboxing Pipeline:

Isolation: Agents write to a temporary staging/ directory, never the core engine.
Static Analysis: A “Gatekeeper” script regex-scans code to ban dangerous system calls (system(), fork(), network headers) and ensures strict inheritance.
Micro-Testing: Candidate code is compiled into a standalone binary and run against a mock OrderBook. If it segfaults, it crashes the test harness, not the production engine available to the main Python process.

6.2 Workflow¶

Context Generation: Run agent_interface/prepare_agent_prompt.py to generate an API summary and few-shot examples for the model.
Code Generation: The LLM writes a candidate .hpp file to agent_interface/staging/.
Assessment: The assess_feature.py script validates the candidate (Static Checks + Micro-Tests).
Promotion: If all checks pass, the feature is moved to forecasting/engine/cpp/features/ and automatically registered.

6.3 Interactive Demo¶

Run the included notebook to see the Agent Loop in action:

jupyter notebook tests/notebooks/test_agent_loop.ipynb

8. Workspace Structure¶

The repo is a uv workspace. Each top-level directory is an independent Python package with its own pyproject.toml. Cross-package imports just work — no sys.path hacking.

ddx/
├── pyproject.toml          # Workspace root (coordinates all packages)
├── engine/                 # C++ engine + pybind11 bridge
│   ├── pyproject.toml
│   ├── CMakeLists.txt
│   ├── src/                # C++ source (Engine, OrderBook, bindings)
│   ├── include/            # C++ headers
│   ├── features/           # Feature implementations (.hpp)
│   └── wrapper/            # Python wrapper around _engine_cpp
├── forecasting/            # Data loading, simulation, metrics
│   ├── pyproject.toml
│   ├── data/               # Loaders, normalization
│   ├── simulation/         # BacktestHarness, AnalysisHarness
│   └── metrics/            # R2SufficientStats, compute_r2
├── evolution/              # Feature evolution, grading
│   └── pyproject.toml
├── agent_interface/        # LLM agent tools
│   └── pyproject.toml
├── cloud/                  # Distributed execution — Modal, Ray
│   └── pyproject.toml
├── common/                 # Shared logging infrastructure
│   └── pyproject.toml
├── setup/                  # Env config utilities
│   └── pyproject.toml
├── scripts/                # CLI scripts
│   └── pyproject.toml
└── tests/                  # Integration tests

Dependency direction: Python packages → engine → C++ (never the reverse). C++ developers can build and test with CMake alone — no Python tooling required.