RL Rollouts with Your Own Trainer

• BYOT rollouts — the most hands-on, for teams already running their own trainer who want Fireworks purely for high-scale rollout inference.
• Training API — the middle ground, for teams who want to own their training logic without managing infrastructure.
• Managed RFT — the turnkey path, for teams who just want a tuned model without running the loop themselves.

​

Where this guide fits

• Disaggregated: Your trainer and rollout cluster can run in different regions or clouds; deployments can span multiple regions to pool capacity.
• Full-parameter or LoRA: Hot-load full-parameter checkpoints for large models, or hot-load a LoRA adapter on top of a frozen base model—both run on Fireworks inference shapes. See LoRA rollouts.
• Fast checkpoint transfer: Lossless compressed incremental snapshots (arc_v2, typically 20×+ compression) over standard object storage—no special RDMA networking between trainer and inference.
• Async / off-policy friendly: Background download during rollouts; configurable swap semantics similar in spirit to PipelineRL—see checkpoint-swap behavior.

​

Supported models

​

Placeholders

​

Prerequisites

1. Fireworks account and API key — create a key and set export FIREWORKS_API_KEY="<key>".
2. Account ID — In the dashboard, open your account settings or any resource URL; the account slug is the segment after /accounts/ (for example accounts/<account_id>/...).
3. Feature enablement — Request external-bucket hot-load for RL rollouts on account <account_id>, including your bucket provider (S3, GCS/gs://, or NEBIUS).
4. Object storage read access for Fireworks — Fireworks needs read-only access to the bucket prefix you will pass as --hot-load-bucket-url. At enablement, Fireworks shares the IAM principal to grant access. Typical setup:
   • Amazon S3: Grant the Fireworks principal s3:GetObject (and s3:ListBucket on the prefix) on s3://<your_bucket>/<your_upload_path>/*.
   • Google Cloud Storage: Grant roles/storage.objectViewer on the bucket or prefix to the Fireworks service account provided at onboarding.
   • Nebius / MinIO: Equivalent read-only credentials or access key scoped to the upload prefix.
5. firectl installed — See firectl.
6. Base model and deployment shape — An RL-capable shape for your model (GPU count, precision). If you omit --deployment-shape, firectl prompts you to pick one interactively.

​

Architecture

​

End-to-end loop

1. Create a hot-load deployment.
2. Upload and hot-load an initial full snapshot.
3. Run rollouts against that snapshot.
4. For each training step: upload and hot-load the next incremental snapshot (see Incremental snapshots).
5. Run rollouts again.
6. Every 20th or 30th step, publish a full snapshot instead of an incremental one. If the incremental chain fails, fall back to a full snapshot.

​

Quickstart checklist

​

1. Create a hot-load deployment

firectl create deployment accounts/<account_id>/models/<model_id> \
  --deployment-shape <shape_name> \
  --deployment-id <deployment_id> \
  --enable-hot-load \
  --hot-load-bucket-type S3 \
  --hot-load-bucket-url s3://<your_bucket>/<your_upload_path> \
  --hot-load-transition-type ASYNC \
  --region US_OHIO_1

​

2. Upload and hot-load an initial full snapshot

​

Snapshot layout

s3://<your_bucket>/<your_upload_path>/<checkpoint_id>/
├── config.json                   # HF model config (must match the base model)
├── tokenizer.json                # tokenizer files (same as the base model)
├── tokenizer_config.json
├── model.safetensors.index.json  # weight name → shard file map
├── model.weight.spec.json        # weight name → {shape, dtype} for every tensor
├── model-00000.safetensors       # weights, one layer per file
├── model-00001.safetensors
└── ...

s3://<your_bucket>/<account_id>/<account_id>-<deployment_id>/version_001/

• identity / <checkpoint_id> — Any opaque string (for example version_001 or step_00100).
• Format — Same layout as the base model on HuggingFace, plus the two manifest files described below. No tensor-parallel sharding in uploaded files.
• File size — Split weights into multiple .safetensors files, each under about 5 GB. One layer per file is required (a single shard must not mix weights from more than one layer) and also minimizes load time.

​

Required files (full snapshot)

{
  "weight_map": {
    "model.layers.0.self_attn.q_proj.weight": "model-00001.safetensors",
    "model.layers.0.self_attn.k_proj.weight": "model-00001.safetensors"
  }
}

{
  "tensor_map": {
    "model.layers.0.self_attn.q_proj.weight": {
      "shape": [2048, 2048],
      "dtype": "float16"
    }
  }
}

​

Incremental snapshots

​

Quantized snapshots

• Add the matching quantization_config block to config.json.
• Make sure model.weight.spec.json (tensor_map) describes the quantized tensors (shape + dtype); the snapshot is rejected if a quantization_config is present but the spec has no dequantizable tensors or is missing the metadata needed to dequantize them.
• Keep the quantization recipe consistent across every snapshot in a run so the incremental chain stays valid (dtypes must not change between a snapshot and its parent).

​

Signal and poll

​

Hot-load API

​

Signal snapshot ready

{ "identity": "version_001" }

curl -X POST https://api.fireworks.ai/hot_load/v1/models/hot_load \
  -H "Authorization: Bearer <fireworks_api_key>" \
  -H "fireworks-model: accounts/<account_id>/models/<model_id>" \
  -H "fireworks-deployment: accounts/<account_id>/deployments/<deployment_id>" \
  -H "Content-Type: application/json" \
  -d '{ "identity": "version_001" }'

import os
import requests

API_KEY = os.environ["FIREWORKS_API_KEY"]
ACCOUNT = "<account_id>"
MODEL = f"accounts/{ACCOUNT}/models/<model_id>"
DEPLOYMENT = f"accounts/{ACCOUNT}/deployments/<deployment_id>"
HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "fireworks-model": MODEL,
    "fireworks-deployment": DEPLOYMENT,
    "Content-Type": "application/json",
}

resp = requests.post(
    "https://api.fireworks.ai/hot_load/v1/models/hot_load",
    headers=HEADERS,
    json={"identity": "version_001"},
    timeout=60,
)
resp.raise_for_status()

​

Poll load status

curl https://api.fireworks.ai/hot_load/v1/models/hot_load \
  -H "Authorization: Bearer <fireworks_api_key>" \
  -H "fireworks-model: accounts/<account_id>/models/<model_id>" \
  -H "fireworks-deployment: accounts/<account_id>/deployments/<deployment_id>"

status = requests.get(
    "https://api.fireworks.ai/hot_load/v1/models/hot_load",
    headers=HEADERS,
    timeout=30,
).json()

replicas = status.get("replicas", [])
ready = (
    replicas
    and all(r.get("readiness") for r in replicas)
    and all(r.get("current_snapshot_identity") == "version_001" for r in replicas)
)

​

When to start rollouts

• Default (on-policy): Wait until all replicas report readiness on the new identity.
• Off-policy / higher utilization: You may start sending rollouts when a subset of replicas is ready—inspect each entry in replicas in the GET response. Stale-policy rollouts are expected; use async transition mode and monitor policy version in streaming responses (see Policy version in responses).

​

3. Run rollouts

curl https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Authorization: Bearer <fireworks_api_key>" \
  -H "fireworks-model: accounts/<account_id>/models/<model_id>" \
  -H "fireworks-deployment: accounts/<account_id>/deployments/<deployment_id>" \
  -H "x-multi-turn-session-id: <trajectory_id>" \
  -H "x-session-affinity: <trajectory_id>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "accounts/<account_id>/models/<model_id>",
    "messages": [{"role": "user", "content": "..."}]
  }'

​

Steady-state training loop

1. Intermediate steps — Build and upload an incremental snapshot (arc_v2), signal with incremental_snapshot_metadata, poll until ready, then run rollouts.
2. Every 20th or 30th step — Publish a new full snapshot for faster recovery and chain reset.
3. On failure — Fall back to a full snapshot; see Ledger & debugging.

curl -X POST https://api.fireworks.ai/hot_load/v1/models/hot_load \
  -H "Authorization: Bearer <fireworks_api_key>" \
  -H "fireworks-model: accounts/<account_id>/models/<model_id>" \
  -H "fireworks-deployment: accounts/<account_id>/deployments/<deployment_id>" \
  -H "Content-Type: application/json" \
  -d '{
    "identity": "version_002",
    "incremental_snapshot_metadata": {
      "previous_snapshot_identity": "version_001",
      "compression_format": "arc_v2",
      "checksum_format": "alder32"
    }
  }'

​

LoRA rollouts

​

Auto-detection

​

LoRA snapshot layout

s3://<your_bucket>/<your_upload_path>/<checkpoint_id>/
├── adapter_config.json
└── adapter_model.safetensors

• adapter_config.json — PEFT adapter config. Its presence is what marks the snapshot as LoRA; it must reference the same base model the deployment serves.
• adapter_model.safetensors — adapter weights. Sharded adapter_model-*.safetensors and the legacy adapter_model.bin are also accepted.

​

Signal and poll

curl -X POST https://api.fireworks.ai/hot_load/v1/models/hot_load \
  -H "Authorization: Bearer <fireworks_api_key>" \
  -H "fireworks-model: accounts/<account_id>/models/<model_id>" \
  -H "fireworks-deployment: accounts/<account_id>/deployments/<deployment_id>" \
  -H "Content-Type: application/json" \
  -d '{ "identity": "adapter_step_001" }'

​

No incremental chain

​

Numerics alignment

• Match quantization / precision between trainer checkpoints and the deployment shape (work with Fireworks if you need a custom shape).
• Measure logprob divergence between trainer forward passes and rollout inference on the same tokens.
• For MoE models, use Router Replay (R3) during rollouts—see MoE Router Replay.

​

Next steps