Hive Hive
Sign in

feat(inference): proxy embedding requests

GitHub issue · Closed

Metadata
Source
tuist/hive #106
Updated
Jul 5, 2026
Domains
Hive
Details

What changed

Hive’s model gateway now accepts embedding requests through POST /inference/v1/embeddings.

The implementation reuses the existing profile and token model:

  • Validates the requested stable model name against the token-bound profile.
  • Rewrites that model to the configured upstream embedding model before proxying.
  • Forwards the upstream response without exposing provider credentials to clients.
  • Records each request with an explicit embedding operation on inference usage rows.
  • Estimates embedding cost from input tokens, with output tokens recorded as zero.
  • Documents the new endpoint and embedding pricing behavior for self-hosted operators.

Why

Atlas currently has to point document embedding traffic directly at the upstream provider because Hive only relayed chat completions. That bypass makes model changes and cost attribution harder to manage centrally.

Routing embeddings through Hive gives operators the same stable profile name, provider credential hiding, token attribution, and cost tracking that chat completion clients already get.

Approach

I kept embeddings as another gateway operation rather than introducing a separate embedding subsystem. The new controller action follows the same authentication, model authorization, provider lookup, and response forwarding path as chat completions, but uses the upstream /embeddings path and non-streaming response handling.

Usage rows now store an operation value so chat completion and embedding traffic can be distinguished later without changing the profile and token model.

Impact

Existing chat completion clients continue to use the same endpoint and profile configuration. Existing usage rows receive chat_completion as the default operation through the migration.

Operators can create an embedding profile, point OpenAI-compatible embedding clients at Hive’s /inference/v1 base address, and use the Hive profile token as the bearer credential. Embedding profiles should set the input token price for the upstream model so Hive can estimate spend from provider-reported usage.

Validation

  • mix format
  • mix test test/hive/inference_test.exs test/hive_web/controllers/inference_controller_test.exs
  • mix test test/hive_web/live/ops_live/inference_test.exs
  • mix compile --warnings-as-errors
  • mix test returned 801 passing tests.
  • Ran a one-off local proxy check that sent a Hive embeddings request through the controller to a local OpenAI-compatible upstream over the real Req client, then verified model rewriting, upstream authorization, response forwarding, and costed usage recording.
Flights

Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.

New Flights are paused Configure model inference, GitHub, and a sandbox provider to start another Flight. Existing results remain available below.
No Flights yet

Start a Flight and preserve its objective, outcome, and session here.

Comments
GA
github-actions[bot] Jul 3, 2026

Blick review didn’t run

The blick review step failed before producing a manifest, so there’s no review to post on this PR. This usually means the agent (opencode) couldn’t start — common causes are an expired or suspended model API key, a missing secret, or the workflow timing out.

See the workflow run for details: https://github.com/tuist/hive/actions/runs/28669725485

Commit: 02ee16d0d9ce4c9c447eb630492a28b5b420247e

GA
github-actions[bot] Jul 3, 2026

Blick review didn’t run

The blick review step failed before producing a manifest, so there’s no review to post on this PR. This usually means the agent (opencode) couldn’t start — common causes are an expired or suspended model API key, a missing secret, or the workflow timing out.

See the workflow run for details: https://github.com/tuist/hive/actions/runs/28670282685

Commit: a57bb1ce6bc3e84104a2acf0247fd75004d1c32c