Rule Evals

Evaluation of whether explicit rules improve AI code generation output.

Methodology

• Model: Claude Opus 4.8 via Claude Code
• Approach: Compare baseline (no rule) vs extracted rule (concise "Rule for AI agents" version)
• Metric: Does providing the rule improve output compared to model's inherent knowledge?
• Code: eval suite on GitHub
• Model comparison: per-rule results across Opus 4.5, 4.6, 4.7, 4.8, Sonnet 4.6 and Sonnet 5

Results

Rule	ESLint	--fix	Eval Improved
Vue SFC Structure
template-on-the-top	✅	✅	✅
script-setup-with-typescript	✅	-	✅
group-script-by-logical-concerns	-	-	✅
multi-word-component-names	✅	-	💪
Props & State
define-props-with-typescript	✅	-	✅
define-emits-with-typescript	✅	-	✅
destructure-props	✅	-	✅
same-name-prop-shorthand	✅	✅	✅
no-mutating-props	✅	-	✅
no-side-effects-in-computed	✅	-	✅
prefer-definemodel	-	-	💪
prefer-ref-over-reactive	-	-	💪
prefer-vueuse	-	-	✅
Template Directives
keyed-v-for	✅	-	💪
use-of-instead-of-in-for-v-for	✅	✅	✅
Styles
scoped-styles	✅	-	💪
Composables
composables-setup-context	-	-	💪
composables-naming	-	-	💪
composables-return-refs	-	-	💪
composables-cleanup	-	-	💪
composables-vs-utils	-	-	💪
composables-organize-by-concern	-	-	✅
composables-input-flexibility	-	-	💪
composables-single-responsibility	-	-	💪
composables-async-patterns	-	-	💪
composables-thin-composables	-	-	✅
composables-progressive-extraction	-	-	💪

✅ in Eval Improved = Rule measurably improves output over baseline 💪 in Eval Improved = Model already applies this pattern without explicit instruction