Fireworks AI for Agent Observability

Integration version1.0.0

To find out if this integration is available in your organization, see your Datadog Integrations page or ask your organization administrator.

To initiate an exception request to enable this integration for your organization, email support@ddog-gov.com.

Overview

Fireworks AI is an inference platform for open source and fine-tuned models. With the Fireworks AI integration, you can use Fireworks AI models in Datadog Agent Observability with your own Fireworks AI API key:

  • Custom LLM-as-a-judge evaluators: Use a Fireworks AI model as the judge that evaluates your application’s spans, traces, or sessions.
  • Playground: Test prompts, tool calls, and structured outputs against Fireworks AI models.

You can select a model from the list that Fireworks AI returns for your API key, or enter a model name, such as the name of a deployed fine-tuned model.

Datadog sends these requests to Fireworks AI with your API key, so Fireworks AI bills the usage to your account.

Setup

Installation

No installation steps are required.

Configuration

Create a Fireworks AI API key

  1. In the Fireworks AI dashboard, open the API keys page.
  2. Click Create API key and copy the key.

Add the Fireworks AI integration

Adding an account requires the Integrations Manage permission.

  1. In Datadog, open the Fireworks AI for Agent Observability integration tile. You can also find the tile in Agent Observability > Settings > Integrations.
  2. On the Configure tab, click Add new.
  3. Enter an account name and the API key from the previous step.
  4. Click the check mark to save the account.

Note: Datadog does not validate the API key when you save the account. To replace a key, edit the account and save the new key.

Validation

  1. In the Playground, open Model configuration from the toolbar.
  2. Under Provider, select Fireworks AI. Then select your account under Account and a model under Model, and click Done.
  3. Enter a prompt and click Run.

A response from the model confirms that the API key works.

Use Fireworks AI models

Custom LLM-as-a-judge evaluators

When you create a custom LLM-as-a-judge evaluator from the Evaluators page:

  1. Next to Model, click the model button to open Model configuration.
  2. Under Account, select Fireworks AI > your account. If no Fireworks AI account is listed, select + Add new account > + Fireworks AI to add one.
  3. Under Model, select a model, or enter a model name such as accounts/<ACCOUNT_ID>/models/<MODEL_ID> and select Use model ID. For a model on a dedicated deployment, use the name format from the Fireworks AI deployments guide.
  4. Click Confirm.

Each evaluator stores its own provider, account, and model.

Playground

After you select a Fireworks AI model in the Playground, click Get code to export the prompt as code. The code reads the API key from the FIREWORKS_API_KEY environment variable.

Limitations

  • Among evaluators, only custom LLM-as-a-judge evaluators can use Fireworks AI. Managed evaluators do not.
  • Fireworks AI judges evaluate text only. Datadog does not send images or audio from spans to Fireworks AI. For providers that support them, see Image and audio evaluators.
  • The model list might not include every model, such as a model on a dedicated deployment. If a model is missing, enter its name.
  • Fireworks AI is not available for HIPAA-enabled organizations.

Data Collected

Metrics

This integration does not collect metrics.

Service Checks

This integration does not include service checks.

Events

This integration does not collect events.

Support

Need help? Contact Datadog support.