dsh-eval

dsh-eval

English | 中文

Agent evaluation over headless dsh profiles: a benchmark YAML document, one headless dsh subprocess per case x trial, trace harvesting from persisted session logs, automatic metric folding, and JSON/markdown run reports.

The package is a bundle and a command-line app. Install it into a profile and run benchmarks with:

sh
dsh plugin --profile eval add dsh-eval
dsh eval run benchmark.yaml
dsh eval report eval-run.json
dsh eval compare eval-v1.json eval-v2.json

dsh eval is a launcher alias for --profile eval; the eval profile composes dsh-base plus this bundle. Each trial spawns the benchmark-configured dsh command (default dsh) against the benchmark-configured profile (default headless) with a private temp workspace, an isolat