Test Assertions
Not all outputs are created equal — and that's why assertions matter.
In Genum, assertions are the mechanism for comparing the expected output of a prompt with its actual output during testing. They are essential to automated regression testing and behavioral validation.
Genum supports multiple assertion modes depending on the complexity and reliability of the comparison you need.
Assertion Modes
Manual Assertion
Use this when automated checks are insufficient.
- Output is manually reviewed by a human.
- Suitable for edge cases, subjective outputs, or exploratory validation.
- Does not participate in regression automation.
Strict Assertion
Performs an exact match between expected_output and last_output.
- Character-by-character equality
- Fails if even one byte differs
- Best for deterministic prompts and structured responses
AI Assertion
Leverages a Genum internal LLM-based evaluator to semantically compare outputs.
- Uses a configurable comparison prompt
- Allows specification of:
- Fields to include/exclude
- Match types (e.g. semantic, exact, ignore)
- Rules for similarity thresholds or tolerances
- Highly useful for:
- JSON field comparison
- Fuzzy natural language alignment
- Domain-specific evaluations
Example use cases:
- Ignore volatile fields like timestamps or IDs
- Compare meaning, not just wording
- Validate structured responses with varying surface forms
Assertion Configuration
AI Assertions are customizable via prompt-level settings or test-case overrides:
- Define field-by-field rules
- Choose strict vs semantic comparison
- Declare expected mismatches or tolerances
This enables:
- Fine-grained regression control
- Flexible handling of evolving prompts
- Reliable CI-level feedback on behavior drift
Assertions let you declare what correctness means in your AI system.
Manual for humans, strict for machines, and AI-based for everything in between.
Use the right assertion — and test with clarity. ✅