Code
Write an eval suite
Runnable tasks that measure a capability, with a scored report at the end.
Fill it in
The specific thing you suspect is weak.
How many cases.
How a run is scored.
Your prompt
Build an eval suite of 15 tasks, graded by A mix. What to measure: [your what to measure] Write tasks that can FAIL. The most common mistake in an eval suite is tasks the subject passes trivially, which produces a 95% score that measures nothing. Aim for a spread: if everything passes, the suite is too easy and you have learned nothing. Each task needs an unambiguous expected outcome and a grader that cannot be argued with. Where a rubric is unavoidable, write the rubric as checks rather than adjectives. Run it, report the score with per-task results, and be specific about what the failures have in common. That pattern is the actual output; the number is just the headline.
Use Write an eval suiteOpens with everything above already filled in.
Why this works
The usual failure is tasks the subject passes trivially, producing a 95% that measures nothing. This aims for a spread, writes graders that cannot be argued with, runs the suite, and reports what the failures have in common. That pattern is the output; the score is just the headline.
Questions
- Can it evaluate any model?
- The suite is runnable code, so point it at whatever you like.