Native integration

Hydra Forge treats native code as a first-class component of a scientific Python library.

Native compilation, exposure, and verification are handled through explicit, inspectable boundaries rather than ad-hoc build logic embedded in Python code.

This document describes how native code is structured, built, exposed, and verified within a Hydra Forge library.

Native integration is fully deterministic: source, compilation, exposure, and documentation are all defined explicitly and verified mechanically.

Design goals

Native integration is designed around four goals:

1. Predictability - consistent behavior across machines
2. Transparency - explicit sources, artifacts, and interfaces
3. Parity - enforced agreement between Python and native layers
4. Longevity - maintainability over time

Native code as a dedicated layer

All native code lives under a single, well-defined location:

src/<package_name>/_native/

This directory represents the entire native surface of the library. Build orchestration is centralized; no native logic is scattered across Python modules.

There is no alternative or implicit native entry point.

src/hydra_dx/_native/
├── src/
│   ├── cpp/
│   └── rust/
├── compiled/
│   └── exposed/
│       └── exposed_api.json
├── native_build_map.json
├── native_categories.json
├── native_compilation.json
├── version.json

Native source of truth: _native/src/

The _native/src/ directory contains the authoritative source code for all native components.

_native/src/
├── cpp/
└── rust/

Key principles

• Source code is never generated implicitly
• Multiple native languages can coexist safely
• Language-specific conventions are respected
• Code is organized by intent, not by build system

Compiled artifacts: _native/compiled/

Compiled native extensions are placed in:

_native/compiled/

Source code and build artifacts are intentionally separated. Artifacts are never modified in place by Python code.

Build orchestration and mapping

Native builds are governed by an explicit build map:

_native/native_build_map.json

This file records declared native modules, their sources, artifacts, and Python exposure.

During rebuild and verification, Hydra Forge validates (and blocks on) consistency between sources, artifacts, names, and exposed interfaces.

Each native C++ unit is automatically assigned a compiler_manifest.json after the first build.

• Generated during the initial forge smithery run
• Provides an explicit location for per-unit compiler configuration
• May be left unchanged or refined as needed
• Remains part of the deterministic and versioned build process

This ensures that customization is always explicit and reproducible, while requiring no manual setup for standard builds.

Python-facing interface: portal/

Native code is exposed to Python exclusively through the generated portal namespace.

src/<package_name>/portal/

The portal represents the entire Python-facing native API surface of the library. It is not discovered dynamically at runtime and does not rely on import-time inspection.

• Python code never imports native binaries directly
• All native functionality is accessed through explicit, generated import paths
• The exposed structure defines what is public and what is not
• Import paths are stable and versioned

The contents of portal/ are generated mechanically from compiled native artifacts and explicit exposure manifests, forming a deterministic, inspectable contract between native code and Python.

If a symbol is not present in the exposed tree, it cannot be imported or documented.

Exposure generation and the exposed tree

Native exposure is a deliberate, repeatable generation step, not a runtime discovery mechanism.

Hydra Forge materializes an explicit exposure tree under:

src/<package_name>/_native/compiled/exposed/

This tree defines the complete set of native modules and symbols that are importable through portal.

• Compiled native artifacts are reconciled against explicit exposure manifests
• Exposure rules are applied explicitly
• Import paths are generated deterministically
• A manifest of the exposed API is written and verified

Exposure generation can be re-run at any time after modifying native_categories.json, without recompiling native code.

All exposure occurs during controlled build or verification steps, never at import time.

Language-specific binding details remain internal to the engine and never leak into the public Python-facing interface.

Native API definition

The native API is materialized into exposed_api.json, an auto-generated file that reflects the current state of compiled native code.

• The file is generated automatically during native build and exposure steps
• Function signatures and structure are extracted from compiled bindings
• Docstrings are captured verbatim from the native source via the compiled module
• No runtime inspection is required for API construction or documentation

exposed_api.json is not authored or modified by developers. It is regenerated by Hydra Forge (via forge smithery) whenever native code is compiled or updated.

This file acts as a persistent, deterministic snapshot of the native API, ensuring that exposure, import paths, and documentation remain aligned without relying on runtime introspection.

These docstrings are treated as the authoritative source of truth when present for native API documentation. They are not re-authored, paraphrased, or mirrored in Python code.

Functions may be exposed without docstrings during development. In such cases, documentation is present but minimal or degraded. This is intentional and supports rapid prototyping.

Documentation parity

Native documentation is part of the public contract, but it is not a prerequisite for experimentation.

• Native functions may exist and be exposed without docstrings
• Docstrings are captured verbatim from compiled bindings during exposure generation
• Documentation visibility always matches runtime availability

Functions without docstrings remain fully usable at runtime, but appear with minimal or degraded documentation. This behavior is intentional.

Hydra Forge is designed to support rapid native prototyping. You can iterate, expose, test, and benchmark native code without being forced into premature documentation work.

At the same time, the system makes documentation quality observable. With the provided exposure blueprints and binding patterns, there is no structural barrier to adding or improving native docstrings. Only a conscious engineering choice.

This allows native code to move from experimental to production-grade without changing exposure paths, import structure, or documentation mechanics.

The best way to evaluate this mechanism is not through examples on this page, but by inspecting the generated library documentation directly.

Every exposed native symbol, its import path, runtime behavior, and extracted documentation can be reviewed in the generated API reference.

This documentation is produced from exposure manifests and validated inventory data. There is no parallel, hand-written documentation layer.

Verification and safety guarantees

Native integration is verified during generation and rebuild.

• Artifact presence and correctness
• Interface visibility
• Documentation presence and alignment with runtime availability
• Inventory consistency
• Test execution

Failure is explicit and blocking; partially integrated native code is never delivered.

Why inline native builds are avoided

Hydra Forge intentionally avoids:

• Inline setup.py compilation
• Import-time builds
• Implicit rebuilds
• Environment-dependent heuristics

These patterns are replaced with explicit orchestration and verification.

Summary

Hydra Forge integrates native code by making boundaries explicit and verifiable.