# `Aludel.Evals`
[🔗](https://github.com/ccarvalho-eng/aludel/blob/main/lib/aludel/evals.ex#L1)

Context for managing evaluation suites, test cases, quality policies, and runs.

# `change_suite`

```elixir
@spec change_suite(Aludel.Evals.Suite.t(), map()) :: Ecto.Changeset.t()
```

Returns a changeset for tracking suite changes.

# `change_test_case`

```elixir
@spec change_test_case(Aludel.Evals.TestCase.t(), map()) :: Ecto.Changeset.t()
```

Returns a changeset for tracking test case changes.

# `create_suite`

```elixir
@spec create_suite(map()) ::
  {:ok, Aludel.Evals.Suite.t()} | {:error, Ecto.Changeset.t()}
```

Creates a new suite.

# `create_suite_policy`

```elixir
@spec create_suite_policy(Aludel.Evals.Suite.t(), map()) ::
  {:ok, Aludel.Evals.SuitePolicy.t()} | {:error, Ecto.Changeset.t()}
```

Creates the next immutable quality policy version for a suite.

Version assignment locks the parent suite row so concurrent writers cannot
create the same suite-local version.

# `create_suite_run`

```elixir
@spec create_suite_run(map()) ::
  {:ok, Aludel.Evals.SuiteRun.t()} | {:error, Ecto.Changeset.t()}
```

Creates a new suite run.

# `create_test_case`

```elixir
@spec create_test_case(map()) ::
  {:ok, Aludel.Evals.TestCase.t()} | {:error, Ecto.Changeset.t()}
```

Creates a new test case.

# `create_test_case_document`

```elixir
@spec create_test_case_document(map()) ::
  {:ok, Aludel.Evals.TestCaseDocument.t()} | {:error, Ecto.Changeset.t()}
```

Creates a test case document.

# `delete_suite`

```elixir
@spec delete_suite(Aludel.Evals.Suite.t()) ::
  {:ok, Aludel.Evals.Suite.t()} | {:error, Ecto.Changeset.t()}
```

Deletes a suite.

# `delete_suite_run`

```elixir
@spec delete_suite_run(Aludel.Evals.SuiteRun.t()) ::
  {:ok, Aludel.Evals.SuiteRun.t()} | {:error, Ecto.Changeset.t()}
```

Deletes a suite run.

# `delete_test_case`

```elixir
@spec delete_test_case(Aludel.Evals.TestCase.t()) ::
  {:ok, Aludel.Evals.TestCase.t()} | {:error, Ecto.Changeset.t()}
```

Deletes a test case.

# `delete_test_case_document`

```elixir
@spec delete_test_case_document(Aludel.Evals.TestCaseDocument.t()) ::
  {:ok, Aludel.Evals.TestCaseDocument.t()} | {:error, Ecto.Changeset.t()}
```

Deletes a test case document.

# `execute_suite`

```elixir
@spec execute_suite(
  Aludel.Evals.Suite.t(),
  Aludel.Prompts.PromptVersion.t(),
  Aludel.Providers.Provider.t(),
  keyword()
) :: {:ok, Aludel.Evals.SuiteRun.t()} | {:error, term()}
```

Executes a test suite against a prompt version and provider.

Runs all test cases for the suite, evaluating their assertions
against the LLM output and creating a suite run with results. When the suite
has a quality policy, execution snapshots the latest immutable version before
any model requests and persists its evaluation with the run.

## Parameters
  - suite: The test suite to execute
  - prompt_version: The prompt version to use
  - provider: The LLM provider to call
  - opts: Sampling options accepted by `Aludel.Evals.Sampling.new/1`

## Returns
  - `{:ok, suite_run}` with execution results
  - `{:error, reason}` if execution fails

# `get_suite!`

```elixir
@spec get_suite!(binary()) :: Aludel.Evals.Suite.t()
```

Gets a suite by ID, raising if not found.

# `get_suite_run!`

```elixir
@spec get_suite_run!(binary()) :: Aludel.Evals.SuiteRun.t()
```

Gets a suite run by ID, raising if not found.

# `get_suite_run_for_export!`

```elixir
@spec get_suite_run_for_export!(binary()) :: Aludel.Evals.SuiteRun.t()
```

Gets a suite run by ID for export.

Preloads the suite, prompt version, and provider so the export
payload can be built without leaking Repo access to the web layer.

# `get_suite_with_prompt!`

```elixir
@spec get_suite_with_prompt!(binary()) :: Aludel.Evals.Suite.t()
```

Gets a suite by ID with prompt preloaded, raising if not found.

# `get_suite_with_test_cases!`

```elixir
@spec get_suite_with_test_cases!(binary()) :: Aludel.Evals.Suite.t()
```

Gets a suite with all test cases preloaded.

# `get_suite_with_test_cases_and_prompt!`

```elixir
@spec get_suite_with_test_cases_and_prompt!(binary()) :: Aludel.Evals.Suite.t()
```

Gets a suite with test cases and prompt preloaded.

# `get_test_case!`

```elixir
@spec get_test_case!(binary()) :: Aludel.Evals.TestCase.t()
```

Gets a test case by ID, raising if not found.

# `get_test_case_document!`

```elixir
@spec get_test_case_document!(binary()) :: Aludel.Evals.TestCaseDocument.t()
```

Gets a single test case document.

Raises `Ecto.NoResultsError` if the document does not exist.

# `get_test_case_with_documents!`

```elixir
@spec get_test_case_with_documents!(binary()) :: Aludel.Evals.TestCase.t()
```

Gets a test case with documents preloaded.

# `import_test_cases`

```elixir
@spec import_test_cases(Aludel.Evals.Suite.t(), [map()]) ::
  {:ok, [Aludel.Evals.TestCase.t()]}
  | {:error, %{row: pos_integer(), changeset: Ecto.Changeset.t()}}
```

Imports test case attributes into an existing suite as one transaction.

The suite association is set from the supplied suite rather than accepted
from imported attributes. If any row fails validation, no test cases are
persisted and the failing row index is returned with its changeset.

# `latest_suite_policy`

```elixir
@spec latest_suite_policy(Aludel.Evals.Suite.t()) ::
  Aludel.Evals.SuitePolicy.t() | nil
```

Returns the latest quality policy for a suite, or `nil` when none exists.

# `launch_suite_execution`

```elixir
@spec launch_suite_execution(pid(), binary(), binary(), binary(), keyword()) ::
  {:ok, reference()} | {:error, term()}
```

Launches suite execution in a supervised task and reports completion
back to the given recipient process. Sampling options are forwarded to the
execution task.

# `list_suite_policies`

```elixir
@spec list_suite_policies(Aludel.Evals.Suite.t()) :: [Aludel.Evals.SuitePolicy.t()]
```

Lists immutable quality policy versions for a suite, newest first.

# `list_suite_runs`

```elixir
@spec list_suite_runs() :: [Aludel.Evals.SuiteRun.t()]
```

Lists all suite runs in the system.

# `list_suite_runs_for_suite`

```elixir
@spec list_suite_runs_for_suite(binary()) :: [Aludel.Evals.SuiteRun.t()]
```

Gets suite runs for a specific suite.

# `list_suite_runs_for_suite_with_associations`

```elixir
@spec list_suite_runs_for_suite_with_associations(binary()) :: [
  Aludel.Evals.SuiteRun.t()
]
```

Gets suite runs for a specific suite with prompt_version and provider
preloaded.

# `list_suites`

```elixir
@spec list_suites() :: [Aludel.Evals.Suite.t()]
```

Lists all suites in the system.

# `list_suites_for_prompt`

```elixir
@spec list_suites_for_prompt(binary()) :: [Aludel.Evals.Suite.t()]
```

Lists evaluation suites for one prompt, ordered by name.

# `list_suites_with_prompt`

```elixir
@spec list_suites_with_prompt() :: [Aludel.Evals.Suite.t()]
```

Lists all suites with their associated prompt preloaded.

# `list_test_cases`

```elixir
@spec list_test_cases() :: [Aludel.Evals.TestCase.t()]
```

Lists all test cases in the system.

# `pass_rates_by_prompt`

```elixir
@spec pass_rates_by_prompt() :: [map()]
```

Calculates pass rates grouped by prompt.

Returns a list of maps with prompt info and pass rate statistics.

# `reload_suite_run_with_associations`

```elixir
@spec reload_suite_run_with_associations(Aludel.Evals.SuiteRun.t()) ::
  Aludel.Evals.SuiteRun.t()
```

Reloads a suite run with associations preloaded.

# `retry_suite_run_test_case`

```elixir
@spec retry_suite_run_test_case(Aludel.Evals.SuiteRun.t(), binary()) ::
  {:ok, Aludel.Evals.SuiteRun.t()} | {:error, term()}
```

Retries a single test case result within an existing suite run.

The existing embedded result is replaced in-place and the suite run
aggregates are recalculated from the updated result set. Sampled results
repeat their complete persisted sampling configuration. The run is refreshed
before execution, and overlapping retries reject the stale write instead of
replacing a newer result.

# `update_suite`

```elixir
@spec update_suite(Aludel.Evals.Suite.t(), map()) ::
  {:ok, Aludel.Evals.Suite.t()} | {:error, Ecto.Changeset.t()}
```

Updates an existing suite.

# `update_test_case`

```elixir
@spec update_test_case(Aludel.Evals.TestCase.t(), map()) ::
  {:ok, Aludel.Evals.TestCase.t()} | {:error, Ecto.Changeset.t()}
```

Updates an existing test case.

---

*Consult [api-reference.md](api-reference.md) for complete listing*