ZynoviqA benchmark built to be cited, not just run.
A Zynoviq benchmark is a fixed, versioned set of certified professional cases with a published method. It is run the same way every time, so results compare across systems and over time. Vendors license it to show evidence; buyers ask for it before they buy.
What exists today, and what is planned
| Benchmark | What it covers | Status | More |
|---|---|---|---|
| India GST AI Benchmark — input tax credit | Eligibility, blocked credits and provisos, GSTR-2B conditions, reversals, time limits | In production — available January 2027 | Benchmark details → |
| India GST AI Report | Public models on the benchmark; method disclosed; held-out cases kept private | Edition 1 planned for March 2027 | The report → |
| SAP finance-agent benchmark | GST inside SAP finance documents and postings | Planned | GST in SAP finance agents → |
| Payroll under the labour codes | Wage definitions, statutory deductions and labour-code compliance | Planned | Payroll domain → |
| Synthetic company test world | A realistic synthetic Indian company (ledgers, vendors, invoices, payroll) with known-correct outcomes, into which an agent is dropped and marked on every action it takes | Planned | For agent companies → |
| Company law | Compliance questions, filings and board processes | Planned | Company-law domain → |
| Procurement | Procurement workflows and compliance | Planned — later | Procurement domain → |
Only the GST input-tax-credit benchmark is in production today. Everything marked planned is a target, not a claim that the benchmark exists. Every domain and its status.
Three ways to use it
| Form | Who | What you get | More |
|---|---|---|---|
| Vendor licence | Software vendors, SAP partners, tax-tech firms, platforms | Run the benchmark across all your customers' deployments; cite results under agreed terms; receive a refreshed version after rule changes | For software vendors → |
| Private run | AI labs, agent companies, enterprise AI teams | We run the benchmark on your system in your test environment; scored privately, never published | Build an Evaluation Mission → |
| Buyer comparison | Enterprises, PSUs, government departments | Candidate products run on the same benchmark in the buyer's own licensed instance; vendors are offered a pre-test so they consent; nothing is published without consent | For buyers of AI → |
Each form is quoted: an annual licence for vendors, a project fee for a private run, a licensed instance for a buyer comparison. Packages and how to start · Licence terms.
The published number, and the private test
Two things share one benchmark and must never be confused. The published number is what a vendor cites, under agreed terms, or what the India GST AI Report shows for publicly available models. The private test is a run on your own system, in your own test environment, reported only to you. The cases are the same; the held-out answer keys are never released in either form. Nothing about a private run is published, and a vendor's published number is never produced without the vendor's consent.
Why it can be trusted
- Fixed and versioned. The same certified cases, marked the same way, on every run; every version carries a valid-as-of date.
- Two independent solvers. Every case is solved separately by two qualified professionals before anyone compares answers.
- Adjudicated. Disagreements are settled by a senior practitioner and recorded, with any dissent kept on the case.
- Tied to the law. Each expected answer cites its section, notification or circular, with the date up to which it is valid.
- Held-out, never sold as training data. Benchmark cases come from a separate pool from anything we sell for training. Training and test separation.
- Duplicate detection across pools, so a benchmark case cannot quietly reappear in a dataset.
- Version recorded on every result, so a score always points to the exact case set it was measured on.
AI never sets, changes or certifies professional ground truth. People write, solve, review and certify every answer. The AI boundary.
Kept current
Professional rules change, and a benchmark that does not change with them stops measuring reality. When a rule changes, qualified professionals revise the affected cases through the full quality chain and a new version is issued with a new valid-as-of date. Older versions stay on record so past results remain comparable, and failures found in evaluation can become new cases in a later version. Evidence and currency · Regulatory Regression Service.
Licensed packs
A benchmark is also delivered as a pack your team can run in its own tools. Every pack contains the same components:
| Component | What it gives you |
|---|---|
| Scenarios | Realistic synthetic cases covering the capability, including the hard ones |
| Expected outcomes and actions | Certified by qualified professionals |
| Expert rationale | Why each answer is correct, with the rule or practice behind it |
| Decision criteria | The conditions that decide each case |
| Edge cases | Where the usual answer stops being right |
| Evaluation rubrics | How to score partial answers and how serious each error is |
| Regression cases | A stable set to re-run after every change |
| Adversarial cases | Professional red-team scenarios |
Bespoke or productised
| Model | What it is | What it means for you |
|---|---|---|
| Bespoke | Built for one customer's AI system, workflow or configuration | Fitted to your system; your information is kept separate from reusable material unless you explicitly authorise otherwise |
| Productised | Built once and licensed to multiple customers | Shared foundation, lower cost per customer, maintained and refreshed by us |
Reusable across customers, or adapted to yours
- Packs anchored in law, such as GST, rest on rules that are the same for everyone. The same certified cases can be licensed to many customers as they are, and refreshed centrally when the law changes.
- Packs for configured business processes, such as order management inside an ERP, depend on how each organisation has set up its system and written its policy. A productised pack provides the foundation; the expected outcomes are adapted to your documented configuration and policy.
Expert case subscription
A pack is a snapshot. An expert case subscription keeps it fresh: a fixed number of newly certified cases each month or quarter, as set in your contract, focused on rule changes, new exceptions and the failure patterns your AI has shown, versioned with a valid-as-of date on every answer. Live in GST; other domains on request where we have qualified professionals. Quoted per subscription.
The catalogue lists every pack with its current status. Formats, delivery and licences · Data rights.
SAP is a trademark of its owner. Zynoviq is not affiliated with or endorsed by SAP.
How a benchmark compounds
- 01ServicesA mission for one AI problem.
- 02Reusable assetsCertified cases you keep.
- 03BenchmarkFixed, versioned, refreshed.
- 04Recurring evaluationRe-run on every release and rule change.
- 05A reference pointAn aspiration to earn, not a claim.
Each pass through evaluation, regression and red teaming adds new cases, so a benchmark becomes more useful the longer it is used.
What a benchmark is not
A benchmark result shows how an AI system performed on a fixed set of certified cases, at a point in time. It is evidence for your own decisions: a release gate you define, a comparison between candidates, a trend across versions. It is not a certification, approval or assurance of your AI, and a good score does not mean the system is safe everywhere; it means it was right on these cases, on this date. We certify our answers, not your system. Evidence, not certificates · What we do not do.
Benchmarks — FAQ
Will our benchmark results be published?+
No. A private run is reported only to you. Vendor-licence results are cited by the vendor under agreed terms. Public comparisons cover publicly available AI models only, in the India GST AI Report.
Is a good benchmark score a certification of our AI?+
No. A result is evidence of how your AI performed on a fixed set of certified cases at a point in time. It is not a certification, approval or assurance of your system. We certify our answers, not your system.
Can we run the benchmark in our own evaluation tools?+
Yes. Benchmark packs can be licensed for use in your own harness: questions only, or the full pack under strict licence terms. Licence types · How tests run.
What happens to a benchmark when the law changes?+
Every case carries a valid-as-of date. When a rule changes, affected cases are re-reviewed through the full quality chain and a new version is issued; older versions stay on record so past results remain comparable. Evidence and currency.
Can you build a benchmark for our domain or workflow?+
Yes, through an Evaluation Mission. Benchmarks anchored in law are reusable across customers; benchmarks for configured business processes are adapted to your documented configuration and policy. Build an Evaluation Mission.