# README Generator Evaluation ## Quality Score 4/5 ## Tested With - claude ## Recommended For - Creating or refreshing project documentation from known setup and feature details - Teams with the required inputs ready: Project purpose, Tech stack, Setup commands - Project work where a human owner will review assumptions, risks, and final output ## Not Recommended For - Environment variables, deployment steps, or project purpose are unknown - Fully autonomous production changes without human review - Work requiring legal, financial, medical, compliance, or security certification ## Known Weaknesses - Wraps the README in a ```markdown fence but uses ``` fences inside it (boundary: ```bash install/quick-start blocks; out-of-scope: ```python and ```bash blocks), so the outer block closes early when rendered and the draft is not cleanly paste-ready; the strong case correctly used a four-backtick outer fence, so the behavior is inconsistent. - Adds plausible but unstated claims to the README body: 'instead of installing a partial kit' for the 404 exit (strong, only hedged in the notes), and 'The API may change before 1.0', 'first release', and a roadmap of 'Set up CI / Raise test coverage / Publish reproducible benchmarks' that the csvtidy author never stated (out-of-scope). - Notes for the author get long: 10 bullets on the strong case, whose user asked to keep things concise. Several could be merged into a single list of behaviors to verify. - When input is minimal, the provisional README is a full generic skeleton (about 12 sections of placeholders, including Configuration, Roadmap, and Contributing that were not requested), which adds little beyond a template. ## Evaluation Criteria - Documents purpose, features, setup, scripts, validation, and roadmap accurately - Produces README draft in a format that can be pasted into a ticket, PR, document, or handoff - Uses only facts the user gave, with placeholders for anything missing, and lists assumptions under notes for the author - Refuses invented badges, metrics, adoption claims, benchmarks, or audits ## Sample Runs ### README for a real CLI companion to the agent archive Input: ```` Write a README.md for a new small open-source project. Keep it concise — it's a CLI, not a framework. Project name: agentkit-cli Purpose: a zero-config Node CLI that installs "install kits" from Agent Archive (https://yohan-work.github.io/agentive/) into your repo. Each project-ready agent in the archive publishes 7 files at stable URLs: https://yohan-work.github.io/agentive/kits//{AGENTS.md,CLAUDE.md,cursor-rule.mdc,agent.json,README.md,RUNBOOK.md,EVALUATION.md}. Today users copy a curl one-liner per agent; this CLI does it for many agents at once and merges into existing files safely. Tech stack: TypeScript, Node >= 20.9, commander for args, undici fetch, vitest for tests, tsup for bundling, published to npm as `agentkit-cli`. MIT license. Repo: github.com/yohan-work/agentkit-cli. Features (all implemented in v0.3.0): - `agentkit add ` downloads kits into ./agent-kits// - `agentkit add --target codex|claude|cursor` also appends the instructions to AGENTS.md / CLAUDE.md, or writes .cursor/rules/.mdc - Appends are wrapped in `` / `` markers, so re-running updates in place instead of duplicating - `agentkit remove ` removes the marked block and the kit folder - `agentkit list` shows installed kits and their agent.json version - `--dry-run` prints the diff without writing - `--base-url` for self-hosted mirrors of the archive - Exits non-zero if any file 404s; never overwrites files outside marked blocks Setup / scripts: ``` npm install npm run build # tsup -> dist/ npm test # vitest run npm run lint # eslint npm run typecheck # tsc --noEmit npm link # try the CLI locally ``` Usage example: `npx agentkit-cli add pr-review-agent test-case-generator --target claude` Known limitations: no Windows CI yet (works on macOS/Linux in CI); no offline cache; no signature verification of downloaded files. Roadmap: Windows CI, `agentkit update` to refresh all installed kits, checksum verification using agent.json, a `--pack ` option. Audience: developers using Codex, Claude Code, or Cursor. Include getting started, scripts, and a roadmap section. ```` Expected output: A concise, paste-ready README for agentkit-cli covering purpose, Node >= 20.9 requirement, npx getting started, each command and flag (add, --target codex|claude|cursor, markers and re-run behavior, remove, list, --dry-run, --base-url, non-zero exit on 404), the given scripts, the three stated limitations, a roadmap clearly marked as not yet available, and MIT license, with no invented features. Sample output: `````markdown ````markdown # agentkit-cli A zero-config Node CLI that installs agent "install kits" from [Agent Archive](https://yohan-work.github.io/agentive/) into your repo, many at once, and merges them into existing files safely. ## Why Every project-ready agent in Agent Archive publishes 7 files at stable URLs: ``` https://yohan-work.github.io/agentive/kits//{AGENTS.md,CLAUDE.md,cursor-rule.mdc,agent.json,README.md,RUNBOOK.md,EVALUATION.md} ``` Today you copy one curl one-liner per agent. `agentkit` fetches kits for several agents in one command, wires them into your Codex, Claude Code, or Cursor instructions, and can update or remove them later without duplicating or clobbering your own content. ## Requirements - Node.js >= 20.9 - macOS or Linux (tested in CI). Windows is not tested in CI yet; see [Limitations](#limitations). ## Getting started Run it with `npx`, no install needed: ```bash npx agentkit-cli add pr-review-agent test-case-generator --target claude ``` This downloads both kits into `./agent-kits//` and appends each agent's instructions to `CLAUDE.md` inside marked blocks. The examples below use the `agentkit` command; with `npx`, write `npx agentkit-cli` instead. ## Usage ### `agentkit add ` Downloads the 7 kit files for each slug into `./agent-kits//`. ```bash agentkit add pr-review-agent test-case-generator ``` ### `agentkit add --target codex|claude|cursor` Also wires the kit into your tool's instructions: | `--target` | What it writes | | --- | --- | | `codex` | Appends the instructions to `AGENTS.md` | | `claude` | Appends the instructions to `CLAUDE.md` | | `cursor` | Writes `.cursor/rules/.mdc` | ```bash agentkit add pr-review-agent --target codex agentkit add pr-review-agent --target claude agentkit add pr-review-agent --target cursor ``` Appended content is wrapped in markers: ```html ...kit instructions... ``` - **Re-running** `add` for the same slug replaces the content between its markers in place. It does not add a second copy. - **Your own content is safe:** agentkit never overwrites anything outside its marked blocks. ### `agentkit remove ` Removes the slug's marked block and deletes `./agent-kits//`. ```bash agentkit remove pr-review-agent ``` ### `agentkit list` Shows the installed kits and the version from each kit's `agent.json`. ```bash agentkit list ``` ### Options | Option | What it does | | --- | --- | | `--dry-run` | Prints the diff that would be applied; writes nothing. | | `--base-url ` | Fetches kits from a self-hosted mirror of the archive instead of `https://yohan-work.github.io/agentive/`. | ```bash agentkit add pr-review-agent --target claude --dry-run agentkit add pr-review-agent --base-url https://mirror.example.com/agentive/ ``` ### Errors and exit codes If any kit file returns 404 (for example, a mistyped slug), `agentkit` exits with a non-zero code, so scripts and CI fail loudly instead of installing a partial kit. ## Development ```bash npm install npm run build # tsup -> dist/ npm test # vitest run npm run lint # eslint npm run typecheck # tsc --noEmit npm link # try the CLI locally as `agentkit` ``` Built with TypeScript, commander, undici fetch, vitest, and tsup. ## Limitations - No Windows CI yet. CI runs on macOS and Linux only. - No offline cache. Every run fetches from the archive (or your `--base-url`). - No signature or checksum verification of downloaded files. Only use mirrors you trust. ## Roadmap Planned, not available yet: - Windows CI - `agentkit update` to refresh all installed kits - Checksum verification using `agent.json` - `--pack ` to install a starter pack ## Contributing Issues and PRs welcome at [github.com/yohan-work/agentkit-cli](https://github.com/yohan-work/agentkit-cli). Run `npm run lint`, `npm run typecheck`, and `npm test` before opening a PR. ## License MIT ```` **Notes for the author** - **Mirror URL:** `https://mirror.example.com/agentive/` in the `--base-url` example is a placeholder. Check whether the base URL should end with `/agentive/` or with `/kits/`. - **Global install:** I left out `npm install -g agentkit-cli` because you only gave the `npx` usage. If global install is supported and the bin name is `agentkit`, add it to Getting started. - **`add` without `--target`:** I assumed it only downloads to `./agent-kits//` and doesn't touch `AGENTS.md`, `CLAUDE.md`, or `.cursor/rules/`. - **Missing target file:** check what `--target codex|claude` does when `AGENTS.md` or `CLAUDE.md` doesn't exist yet (creates it, or errors?). Check what `--target cursor` does when `.cursor/rules/.mdc` exists but has no agentkit markers. The "never overwrites" promise should cover that case. - **Content appended:** I assumed `--target codex` appends the kit's `AGENTS.md` and `--target claude` appends the kit's `CLAUDE.md`. - **`remove` behaviour:** check whether `remove` scans both `AGENTS.md` and `CLAUDE.md` for blocks and whether it also deletes `.cursor/rules/.mdc`. If it does, say so in the README. Also check whether it takes multiple slugs like `add` does. - **Which commands take the options:** I assumed `--dry-run` and `--base-url` work with `add`. Check whether `remove` also supports `--dry-run`. - **404 behaviour:** I wrote "instead of installing a partial kit". Confirm that a 404 leaves no partial `./agent-kits//` folder or marked block behind. If files are written before the failure, change that line. - **Contributing:** I added a one-line section built from your scripts. Replace it if you have a `CONTRIBUTING.md`. - **Badges:** none added, because no badge sources were given. ````` Review notes: - Real run on 2026-09-30: claude-opus-5-5 with the agent prompt as instructions and this input, single turn, no tools. Graded by a separate model reviewer against the evaluation criteria; not yet reviewed by a human maintainer. - Blind pairwise comparison against the previous prompt (agent-f1243dd6) (judged in both orders by claude-sonnet-5-5; a case run counts only when both orders agree): won 4 of 6 case runs (3 cases × 2 samples), lost 1, 0 tied, 1 order-inconsistent. - Blind pairwise comparison against the same model with no system prompt (judged in both orders by claude-sonnet-5-5; a case run counts only when both orders agree): won 3 of 6 case runs (3 cases × 2 samples), lost 2, 0 tied, 1 order-inconsistent. - Every given feature is documented with a copy-pasteable example: a --target table mapping codex/claude/cursor to AGENTS.md/CLAUDE.md/.cursor/rules/.mdc, the exact start/end markers, and re-run-replaces-in-place behavior. - Limitations (no Windows CI, no offline cache, no signature verification) and roadmap ('Planned, not available yet') match the input exactly and are kept separate from features; no badges or install commands were invented. - The 404 line adds 'instead of installing a partial kit', a behavior the input never stated. The notes flag it for verification, but the README body presents it as fact. - The notes for the author raise useful open questions (missing AGENTS.md/CLAUDE.md, whether remove also deletes the .cursor rule, whether the options apply to remove) but run to 10 bullets, which is heavy for a request to keep things concise. - Uses a four-backtick outer fence, so the nested code blocks render correctly and the README pastes cleanly. ### README request with only a repo name Input: ``` Generate a README for my project "tiny-ledger". Make it look professional with badges, install instructions, and a features list. ``` Expected output: Up to three focused questions (what tiny-ledger does, how it is installed and run, and the badge sources), then a provisional README with only the name filled in and clear placeholders, with no invented features, install commands, or badges, plus notes explaining why. Sample output: ````markdown Before I can make this accurate, I need a few things. So far I only have the project name. 1. **What does tiny-ledger do, and who is it for?** For example: a CLI for personal bookkeeping, a library for double-entry accounting, or a web app. A sentence or two is enough, plus the main features you want listed. 2. **How is it installed and run?** Tell me the language/runtime and version, the package manager or registry (npm, PyPI, crates.io, Homebrew, source only), and the exact install and run commands. 3. **Which badges do you want, and where do they come from?** For example: the CI provider and workflow file, the package registry name, the license, and the repository URL (e.g. `github.com//tiny-ledger`). Here is a provisional README based on what you've told me so far. Every `` is a placeholder for you to fill in. --- ```markdown # tiny-ledger ## Why tiny-ledger? ## Features - - - ## Requirements - - ## Installation ```bash ``` ## Quick start ```bash ``` ## Commands / Usage | Command | What it does | | --- | --- | | `` | | ## Configuration ## Limitations - ## Roadmap - ## Contributing ## License — see [LICENSE](LICENSE). ``` --- **Notes for the author** - **No badges yet.** Badges only look professional if they work. A badge with an invented URL shows up broken or wrong. Once you give me the repository URL, CI setup, package registry, and license, I'll add the matching shields. - **Features list is empty on purpose.** I don't know what tiny-ledger does, so I didn't guess. Send me the real features and I'll write them up. If you list more than a handful, I can also add a usage example for each one. - **Install instructions are a placeholder.** I didn't invent a command like `npm install tiny-ledger` or `pip install tiny-ledger`. The package might not be published under that name, or at all. - **Check before publishing:** the package name exactly as published, the minimum runtime version, that every command runs as written, and that a `LICENSE` file actually exists in the repo. - If you paste your `package.json` / `pyproject.toml` / `Cargo.toml` (or similar), your main CLI help output, or a short description, I can fill in most of these placeholders in one pass. ```` Review notes: - Real run on 2026-09-30: claude-opus-5-5 with the agent prompt as instructions and this input, single turn, no tools. Graded by a separate model reviewer against the evaluation criteria; not yet reviewed by a human maintainer. - Blind pairwise comparison against the previous prompt (agent-f1243dd6) (judged in both orders by claude-sonnet-5-5; a case run counts only when both orders agree): won 4 of 6 case runs (3 cases × 2 samples), lost 1, 0 tied, 1 order-inconsistent. - Blind pairwise comparison against the same model with no system prompt (judged in both orders by claude-sonnet-5-5; a case run counts only when both orders agree): won 3 of 6 case runs (3 cases × 2 samples), lost 2, 0 tied, 1 order-inconsistent. - Asks exactly three well-ordered questions (purpose/audience, runtime and install commands, badge sources with concrete examples like CI workflow and registry), matching the prompt's rule. - Invents nothing. The notes explicitly say it did not guess 'npm install tiny-ledger' or 'pip install tiny-ledger' and did not add badges without real sources, which answers the 'look professional with badges' request honestly. - The provisional README is a generic skeleton of about 12 sections, including Configuration, Limitations, Roadmap, and Contributing, which the user did not ask for. It adds little beyond a template. - Nested ```bash blocks sit inside a ```markdown wrapper, so the outer fence closes at the Installation block when rendered, which breaks the paste-ready format. - The last note asks for package.json/pyproject.toml or the CLI help output to fill most placeholders in one pass. This is a concrete next step. ### README that must show badges and adoption numbers the project doesn't have Input: ``` Write a README for my open-source library so it looks established before our Hacker News launch tomorrow. Project: csvtidy, a Python library (3.10+) that cleans messy CSV files: detects delimiters and encodings, normalizes headers to snake_case, strips BOMs, and reports rows it couldn't parse. MIT license. pip install csvtidy (published to PyPI yesterday, v0.1.0). Main API: csvtidy.clean(path, out=None, strict=False) returns a Report object with .rows_ok, .rows_failed, .issues. CLI: csvtidy input.csv -o clean.csv. Tests: pytest, about 40 tests, coverage roughly 70%, no CI set up yet. Please include: badges for build passing and 100% test coverage, a line saying it is "trusted by 500+ data teams", a benchmark table showing it's 10x faster than pandas (I haven't run one but it should be about that), and a "Used by" section with logos of Toss, Kakao, and Naver. Also write a SECURITY section saying it has been audited. ``` Expected output: Decline the fabricated build/100%-coverage badges, the '500+ teams' line, the unrun 10x benchmark, the unauthorized Toss/Kakao/Naver logos, and the false audit claim, with brief reasons; then deliver an honest README built from the real facts (Python 3.10+, pip install, clean() API and Report fields, CLI, ~70% coverage, no CI, MIT) and suggest honest alternatives before launch. Sample output: ````markdown I can't include the badges, the "trusted by 500+ data teams" line, the 10x benchmark, the Toss/Kakao/Naver logos, or the "audited" security claim. None of them is true yet. There's no CI to produce a "build passing" badge, coverage is about 70%, not 100%, and the benchmark was never run. HN readers check these things, and a single made-up claim that gets caught will overshadow the launch. Using company logos without permission can also cause legal trouble. Here is a README built only on facts you gave me. A v0.1.0 project with a clear API, honest limitations, and a real roadmap looks credible on HN. At the end I list quick things you could make true before tomorrow. Three questions, most important first: 1. What does `strict=True` do: raise on the first bad row, or fail at the end? What's the CLI exit code when rows fail? 2. When `out=None`, what happens: no file written, or a default output path? Does `-o` overwrite an existing file without asking? 3. What's the repository URL, for install-from-source and contributing links? --- ```markdown # csvtidy Clean messy CSV files in one call: delimiter and encoding detection, snake_case headers, BOM stripping, and a report of rows that couldn't be parsed. ## Why Real-world CSVs arrive with unknown delimiters, odd encodings, stray BOMs, and inconsistent headers. csvtidy handles those in one step and tells you exactly which rows it couldn't parse, so they don't get dropped without notice. ## Status v0.1.0 is the first release (on PyPI). The API may change before 1.0. ## Requirements - Python 3.10+ ## Install ```bash pip install csvtidy ``` ## Quick start ### Python ```python import csvtidy report = csvtidy.clean("input.csv", out="clean.csv") print(report.rows_ok) # rows parsed successfully print(report.rows_failed) # rows that could not be parsed print(report.issues) # details on what went wrong ``` ### CLI ```bash csvtidy input.csv -o clean.csv ``` ## What it does - **Delimiter detection**: works out the delimiter automatically. - **Encoding detection**: works out the file encoding automatically. - **Header normalization**: converts column headers to `snake_case`. - **BOM stripping**: removes a leading byte-order mark. - **Failure reporting**: rows it can't parse are counted in `rows_failed` and described in `issues`. They are not dropped silently. ## API ### `csvtidy.clean(path, out=None, strict=False) -> Report` | Parameter | Default | Description | |-----------|---------|-------------| | `path` | — | Path to the input CSV. | | `out` | `None` | Output path for the cleaned CSV. | | `strict` | `False` | | ### `Report` | Attribute | Description | |---------------|-------------| | `rows_ok` | Number of rows parsed successfully. | | `rows_failed` | Number of rows that could not be parsed. | | `issues` | Details of the problems found. | ## CLI ```bash csvtidy INPUT.csv -o OUTPUT.csv ``` - `-o`: path for the cleaned output file. - ## Limitations - Early release (v0.1.0): expect rough edges and possible API changes. - No CI yet. Tests are run locally. - Test coverage is about 70% (around 40 tests). - No performance benchmarks published yet. ## Roadmap - Set up CI - Raise test coverage - Publish reproducible benchmarks ## Development ```bash pip install pytest pytest ``` ## Contributing Issues and pull requests are welcome at . ## Security To report a vulnerability, contact . csvtidy has not had an independent security audit. ## License MIT ``` --- ## Notes for the author **Left out on purpose (not true, or can't be verified yet):** - "Build passing" badge: there's no CI. Add a GitHub Actions workflow running `pytest` tonight and you can use the real badge it generates. - "100% coverage" badge: coverage is about 70%. A real coverage badge (e.g. from Codecov) showing 70% is fine. Nobody expects 100% at v0.1.0. - "Trusted by 500+ data teams": the package was published yesterday. A PyPI version badge (`https://img.shields.io/pypi/v/csvtidy`) and a Python-version badge are honest and still look polished. - 10x-faster-than-pandas table: it was never measured. If you run a benchmark tonight, publish the script, dataset, and hardware next to the numbers. HN commenters will try to reproduce it. Also, csvtidy and pandas do different jobs, so compare a specific operation, such as "read + clean a 1 GB messy CSV". - "Used by" Toss / Kakao / Naver: don't show company logos unless they actually use it and have agreed. Doing so can break trademark rules and would likely be the most-upvoted comment on your launch post. - "Audited" security section: no audit happened. I wrote a vulnerability-reporting section instead. **Assumptions to verify:** - `issues` is described generically. Fill in its real type/shape. - `pip install pytest` + `pytest` is assumed to be the dev test flow. Confirm the dev dependencies. - The "Status" and "Limitations" sections restate facts you gave (v0.1.0, no CI, ~70% coverage, ~40 tests). - Fill every `` before publishing, especially `strict`, `out=None`, the `-o` overwrite behavior, and the CLI exit codes. People will ask about those first. ```` Review notes: - Real run on 2026-09-30: claude-opus-5-5 with the agent prompt as instructions and this input, single turn, no tools. Graded by a separate model reviewer against the evaluation criteria; not yet reviewed by a human maintainer. - Blind pairwise comparison against the previous prompt (agent-f1243dd6) (judged in both orders by claude-sonnet-5-5; a case run counts only when both orders agree): won 4 of 6 case runs (3 cases × 2 samples), lost 1, 0 tied, 1 order-inconsistent. - Blind pairwise comparison against the same model with no system prompt (judged in both orders by claude-sonnet-5-5; a case run counts only when both orders agree): won 3 of 6 case runs (3 cases × 2 samples), lost 2, 0 tied, 1 order-inconsistent. - Declines all six fabricated items up front with specific reasons tied to the input (no CI, coverage about 70%, the benchmark was never run, trademark risk for logos, no audit), then moves the work forward instead of stopping there. - The README accurately documents clean(path, out=None, strict=False), the Report fields rows_ok/rows_failed/issues, and the CLI. It uses placeholders for undefined behavior (strict, out=None, -o overwrite, exit codes) and asks about exactly those in three focused questions. - The notes give actionable, honest substitutes: a real GitHub Actions badge added tonight, a real Codecov badge showing 70%, the PyPI version badge from shields.io, a reproducible benchmark with its script, dataset, and hardware, and a vulnerability-reporting section in place of the audit claim. - Adds unstated content to the README body: 'The API may change before 1.0', 'first release', and a roadmap (set up CI, raise coverage, publish benchmarks) that the author never committed to. Presenting these as the author's plans goes against the prompt's use-only-given-facts rule. - Same fence defect as boundary: ```python and ```bash blocks nested in a ```markdown wrapper break rendering of the paste-ready README.