PHPackages padosoft/eval-harness-ai-bridge - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. padosoft/eval-harness-ai-bridge ActiveLibrary padosoft/eval-harness-ai-bridge =============================== Bridge between padosoft/eval-harness and the laravel/ai SDK: turn an AgentResponse into a scoreable trajectory, evaluate multi-turn conversations, and assert an agent against a golden dataset from Pest or PHPUnit. 00[1 PRs](https://github.com/padosoft/eval-harness-ai-bridge/pulls)PHPCI passing Since Aug 24Pushed todayCompare [ Source](https://github.com/padosoft/eval-harness-ai-bridge)[ Packagist](https://packagist.org/packages/padosoft/eval-harness-ai-bridge)[ RSS](/packages/padosoft-eval-harness-ai-bridge/feed)WikiDiscussions main Synced today READMEChangelog (1)DependenciesVersions (2)Used By (0) eval-harness-ai-bridge ====================== [](#eval-harness-ai-bridge) **Evaluate a `laravel/ai` agent on *what it did*, not just what it said.** The bridge between [padosoft/eval-harness](https://github.com/padosoft/eval-harness) and the official Laravel AI SDK. [![Packagist](https://camo.githubusercontent.com/b0ece22361ffd0502a5bcc5a96b2c523cb85d2ae620ae553b4bedc60e5edad76/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f7061646f736f66742f6576616c2d6861726e6573732d61692d6272696467652e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/padosoft/eval-harness-ai-bridge) [![CI](https://camo.githubusercontent.com/a3a887ff0f778a07e9f46a3cd498237394ae939855b797b9b59da01ce2027548/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f616374696f6e732f776f726b666c6f772f7374617475732f7061646f736f66742f6576616c2d6861726e6573732d61692d6272696467652f63692e796d6c3f6272616e63683d6d61696e267374796c653d666c61742d737175617265)](https://github.com/padosoft/eval-harness-ai-bridge/actions) [![PHP](https://camo.githubusercontent.com/79c145400ad0925e7c5a684d6e543477c75fe700d1b9ef2934bcd246ef21f40f/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f7068702d253345253344382e332d3737376262333f7374796c653d666c61742d737175617265)](https://camo.githubusercontent.com/79c145400ad0925e7c5a684d6e543477c75fe700d1b9ef2934bcd246ef21f40f/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f7068702d253345253344382e332d3737376262333f7374796c653d666c61742d737175617265) [![Laravel](https://camo.githubusercontent.com/cbef5a2edd6eb43b4547ed337ee084403e443037560cbc25099582d9c7530263/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c61726176656c2d313225323025374325323031332d6666326432303f7374796c653d666c61742d737175617265)](https://camo.githubusercontent.com/cbef5a2edd6eb43b4547ed337ee084403e443037560cbc25099582d9c7530263/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c61726176656c2d313225323025374325323031332d6666326432303f7374796c653d666c61742d737175617265) [![License](https://camo.githubusercontent.com/80745dbe33afd9a55d8f337beab685539ad38daf38fc030831c0c4f0d93fdcfc/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d4170616368652d2d322e302d626c75653f7374796c653d666c61742d737175617265)](https://camo.githubusercontent.com/80745dbe33afd9a55d8f337beab685539ad38daf38fc030831c0c4f0d93fdcfc/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d4170616368652d2d322e302d626c75653f7374796c653d666c61742d737175617265) --- The bug you cannot see ---------------------- [](#the-bug-you-cannot-see) Your support agent replies: > *"Your order ships Tuesday."* It is correct. Every metric you have scores it **1.0** — exact match, ROUGE, embeddings, even an LLM judge grading it against the golden answer. It never called the order lookup. It guessed, from the shape of the question and whatever was in its context, and it happened to land. It will land often enough to look healthy, and the day it does not, it will be wrong with total confidence — to a customer, about their order. **No amount of scoring the text will ever tell you this.** The final answer is precisely the part a broken agent can still get right by accident. What this package does ---------------------- [](#what-this-package-does) `padosoft/eval-harness` can already score *how* an answer was produced: `tool-called`, `tool-called-with`, `tool-call-order`, `steps-below`, `no-pending-approvals`, `approval-gated`. It scores them against a plain **`Trajectory`** DTO that knows nothing about any SDK. This package is the twelve lines that fill that DTO from a `laravel/ai` response — plus the two other things you need before an agent eval is real: **multi-turn conversations**, and an assertion you can put in your **test suite**. ``` use Padosoft\EvalHarnessAiBridge\Runners\AgentSampleRunner; $eval->dataset('support.agent') ->loadFromYaml(database_path('evals/support.yaml')) ->withMetrics(['llm-as-judge', 'tool-called', 'no-pending-approvals']) ->register(); $report = $eval->run('support.agent', new AgentSampleRunner( fn (array $input) => Ai::agent(SupportAgent::class)->prompt($input['question']), )); ``` That is the whole integration. The runner calls your agent, returns the text for the text metrics, and records the trajectory for the trajectory metrics. ``` - id: ship-date input: { question: 'When does order 44192 ship?' } expected_output: 'Tuesday' metadata: trajectory: tools: [lookup_order] # it must actually look it up arguments: lookup_order: { id: 44192 } # and look up the right order ``` The guess now scores **0.0** on `tool-called`, and your build goes red on the bug that was invisible yesterday. --- Why the DTO is not the SDK's response object -------------------------------------------- [](#why-the-dto-is-not-the-sdks-response-object) This is the design decision the whole package exists to protect. The obvious implementation is to have the harness accept a `Laravel\Ai\Responses\AgentResponse` directly. It is less code, and it is the shape competing tools ship. It also means **your eval suite is only as portable as your SDK choice**: move to a custom orchestrator, to MCP, to `laravel-flow` saga steps, or simply to the next major version of the SDK, and every agent assertion you wrote has to be rewritten. So the harness scores a DTO, and the translation lives here — in **one file** whose blast radius an architecture test keeps honest: ``` public function test_the_harness_itself_never_references_the_ai_sdk(): void public function test_only_the_adapter_and_the_runner_know_about_laravel_ai(): void ``` A boundary that is only described in a README erodes. These fail when it does. --- What gets translated -------------------- [](#what-gets-translated) `laravel/ai``Trajectory``$response->toolCalls``toolCalls[]` — name and arguments`$response->toolResults`the `result` / `error` on the **matching** call`$response->steps``steps`last `Step::$finishReason``finishReason``$response->pendingApprovals``pendingApprovals`approved tool results`approvals[]`usage, provider, model`metadata`Four details that are not obvious, and each has a test: - **Results join calls by id, never by position.** Parallel tools return out of order, and a pending call has no result at all. Matching by index silently attaches one call's outcome to another's. - **A denied call is recorded as failed, with no result.** The tool never ran. *"Did it look the order up?"* must not be satisfied by a rejection. - **A run stopped on an approval reports `pending_approval`, not `stop`.** Text that says *"I have submitted that"* while an approval is pending reads as success and is not. - **Usage travels in the metadata, in the shape eval-harness's cost ledger reads** — so agent spend appears next to judge spend instead of being quietly treated as free. --- Multi-turn conversations ------------------------ [](#multi-turn-conversations) The failure that costs the most in a real assistant is not a wrong first answer. It is the model that answers turn one perfectly and **forgets it by turn three**. A single-turn dataset cannot express that. ``` name: support.multi-turn conversations: - id: refund-then-address tags: [returns] turns: - user: 'I want to return the boots I bought last week.' expect: 'asks for the order number' - user: 'It is 44192.' expect: 'confirms the 30-day window' - user: 'And can you send it to my work address instead?' expect: 'uses the address from the order, does not re-ask for the order number' tags: [context-retention] ``` ``` use Padosoft\EvalHarnessAiBridge\Datasets\ConversationDataset; $eval->dataset('support.multi-turn') ->withSamples(ConversationDataset::fromFile(database_path('evals/support-conversations.yaml'))) ->withMetrics(['llm-as-judge']) ->register(); ``` **Each turn becomes its own row**, carrying every turn before it as `input.history`. That shape earns three things: - the report names **the turn that broke**, not "conversation 4 failed" — a pass rate over turns tells you *where* an assistant loses the thread, which is the number that leads to a fix; - **every existing metric works unchanged**, because a turn is a row; - turns are independently addressable, so the regression gate joins **turn 3** across runs by content hash like any other row. The history holds the *dataset's* expected answers, never the model's own previous output. An eval whose turn-3 input depends on what the model said at turn 2 measures something different on every run and cannot be compared to itself. --- Evals in your test suite ------------------------ [](#evals-in-your-test-suite) A golden dataset that only runs in a nightly job is a dataset nobody watches. Put it on the same red/green as everything else: **Pest** ``` it('holds its ground on support questions', function () { expect(fn (array $input) => Ai::agent(SupportAgent::class)->prompt($input['question'])) ->toPassEval('support.agent', minMacroF1: 0.85); }); ``` **PHPUnit** ``` use Padosoft\EvalHarnessAiBridge\Testing\AssertsEvals; final class SupportAgentTest extends TestCase { use AssertsEvals; public function test_the_support_agent_holds_its_ground(): void { $this->assertPassesEval( 'support.agent', fn (array $input) => Ai::agent(SupportAgent::class)->prompt($input['question']), minMacroF1: 0.85, ); } } ``` An eval is not a unit test: it is statistical, slow, and it costs money. So this is deliberately a **threshold** assertion — a suite that demands 100% on an LLM pipeline is a suite that gets muted within a fortnight — and a failure names the rows: ``` Dataset 'support.agent' failed: macro-F1 0.7143 is below the required 0.8500. 3 failing row(s), worst first: - refund-window (score 0.1000, pass rate 0%) - work-address (score 0.3000, pass rate 33%) - stock-check (score 0.6000, pass rate 67%) Run `php artisan eval-harness:brief ` for the full diagnosis. ``` `macro-F1 0.71 < 0.85` tells nobody what to open. Three more things the assertion does: - **`--repetitions` and `--budget-usd` are available from the test.** An LLM pipeline that answers correctly two times in three is not a pipeline that scores 1.0, and one execution cannot tell those apart. - **A run halted on its budget can never pass**, whatever the numbers say. The rows that never executed are disproportionately the ones that would have failed. - **One run can feed several assertions** (`assertEvalReportPasses`). Running an eval three times to check three thresholds is three bills. --- Installation ------------ [](#installation) ``` composer require --dev padosoft/eval-harness-ai-bridge ``` Requires PHP 8.3+, Laravel 12 or 13, `padosoft/eval-harness` ^1.6, and `laravel/ai`. There is **no service provider and nothing to configure**. A package whose job is to connect two other packages should not become a third thing to configure: everything here is a class you construct where you use it. --- API --- [](#api) ClassWhat it is for`Trajectories\AgentResponseTrajectory::fromResponse()`translate any `laravel/ai` response into a `Trajectory``Runners\AgentSampleRunner`run an agent as the system-under-test and record its trajectory`Datasets\ConversationDataset::fromFile()`multi-turn conversations as dataset rows`Testing\AssertsEvals``assertPassesEval()` / `assertEvalReportPasses()` for PHPUnit`Testing\EvalAssertion`the run-and-judge primitive both surfaces sit on`expect(...)->toPassEval()`the Pest expectation, auto-registered when Pest is presentAlready recording trajectories yourself? Use the adapter alone: ``` $trajectory = AgentResponseTrajectory::fromResponse($response); app(TrajectoryRecorder::class)->record($sample->id, $trajectory); ``` --- Part of the Padosoft AI suite ----------------------------- [](#part-of-the-padosoft-ai-suite) PackageWhat it does**[padosoft/eval-harness](https://github.com/padosoft/eval-harness)** (+ `-admin`)Golden datasets, RAG metrics, repeated sampling with real statistics, baselines and per-row regression gates, run briefings, cost budgets, adversarial testing[padosoft/laravel-evidence-risk-review](https://github.com/padosoft/laravel-evidence-risk-review)Evidence tiers and risk sweeps — also a **zero-token** eval metric[padosoft/laravel-pii-redactor](https://github.com/padosoft/laravel-pii-redactor)Field-level PII detection and masking inside the app boundary[padosoft/laravel-flow](https://github.com/padosoft/laravel-flow) (+ `-ai`, `-admin`)Saga engine with approval gates and replay for AI workflows[padosoft/laravel-ai-finops](https://github.com/padosoft/laravel-ai-finops)Budgets, quotas and chargeback for AI spend[padosoft/laravel-iam-agents](https://github.com/padosoft/laravel-iam-agents)Delegated access: an agent acts for a user without ever holding their token--- Contributing ------------ [](#contributing) Read `CONTRIBUTING.md` and `docs/LESSON.md` before opening a PR. Every change ships with `composer validate --strict`, Pint, PHPStan level 6 and PHPUnit green. License ------- [](#license) Apache-2.0. See [LICENSE](LICENSE). ### Health Score 21 — LowBetter than 17% of packages Maintenance65 Regular maintenance activity Popularity0 Limited adoption so far Community8 Small or concentrated contributor base Maturity13 Early-stage or recently created project Bus Factor1 Top contributor holds 66.7% of commits — single point of failure How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Community Maintainers ![](https://avatars.githubusercontent.com/u/10467699?v=4)[Lorenzo](/maintainers/lopadova)[@lopadova](https://github.com/lopadova) --- Top Contributors [![lopadova](https://avatars.githubusercontent.com/u/10467699?v=4)](https://github.com/lopadova "lopadova (2 commits)")[![claude](https://avatars.githubusercontent.com/u/81847?v=4)](https://github.com/claude "claude (1 commits)") --- Tags aiai-agentsai-toolsharnessharness-ai ### Embed Badge ![Health badge](/badges/padosoft-eval-harness-ai-bridge/health.svg) ``` [![Health](https://phpackages.com/badges/padosoft-eval-harness-ai-bridge/health.svg)](https://phpackages.com/packages/padosoft-eval-harness-ai-bridge) ``` PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)