Digital Engineering/AI Product Assurance/Assessment Methodology
The Noseberry Product Assurance Framework: Assessment Methodology
The full method behind every Noseberry assessment: what we look at, what evidence counts, how a status is decided, how findings are classified, and how you challenge a finding you think is wrong. It is written to be read, argued with, and run by someone else.
Why we publish this
An assessment score nobody can check is a marketing number. If we call your Application Security dimension Critical and you cannot see what evidence produced that word, what would have produced a different one, or who decided, you have bought an opinion with a logo attached.
This page is the specification the Noseberry Product Assurance Framework runs on: the dimensions and sub-criteria, the evidence each accepts, what the status labels mean, how findings are classified, how the score is constructed, who may review, and how you argue with us.
It is written so a sceptical engineer could take the same codebase, follow the same rules and land close to the same conclusions. That is the test we hold it to.
Evidence and how much each source counts
Every finding traces to at least one evidence source, and the report names it. Sources are not equal, and the framework is explicit about which carries a conclusion.
Primary evidence
Can support a finding alone
Static analysis from Semgrep, CodeQL and SonarQube, tuned to the stack rather than run at defaults. Dependency and licence data from the resolved tree, including transitive packages, advisories and licence terms. Infrastructure configuration, declared in code and observed through read-only console access. Engineer review notes: a named reviewer's recorded reading of the system, with file and line references.
Corroborating evidence
Strengthens a finding, never establishes one
Repository history: change frequency by file, authorship concentration, and the sequence in which a control appeared and stopped appearing. History tells you where to look and how a defect arrived, not that it exists.
Contextual evidence
Shapes severity, never creates a finding
Interviews about what the product is meant to do, what has already broken, what constraints exist. They change how a finding is classified and how its impact is written. Nothing enters the register on an interview alone.
The rule that matters
A tool result is a signal, not a finding. Automated output is triaged by a named engineer, and whatever survives is confirmed against the code before it is written down. An alert that proves to be unreachable code, a control applied elsewhere, or a mismatched rule is closed during triage.
Coverage is stated in every report: repositories, branches and commit hashes, environments, exclusions.
The six dimensions
Each dimension carries a status, Critical, Needs Attention or Ready, with sub-criteria that roll up to it. The same definitions apply in every engagement, audit or cleanup alike.
Code Health
Definition. Whether the codebase can be safely changed by someone who did not write it.
Sub-criteria. Structural coherence and duplication; complexity distribution; naming and boundary discipline; test coverage and quality; change safety for a new engineer.
Evidence. Static analysis metrics, duplication and complexity measurement, test results, engineer review notes, change-frequency data.
Out of scope. Style, formatting, framework choice and idiom. We assess changeability, not taste.
Application Security
Definition. Whether security controls exist, are applied consistently, and hold at the boundaries where they are relied upon.
Sub-criteria. Authentication and sessions; authorisation consistency; input handling and output encoding; secrets and configuration; dependency and licence provenance; data exposure paths.
Evidence. Static analysis and secrets detection, dependency advisories, engineer review of the authorisation model at every entry point, infrastructure configuration. Assessed against OWASP ASVS and the OWASP Top 10, with secure-SDLC practice referenced to NIST SSDF. We identify gaps relevant to these standards and do not certify compliance against them.
Out of scope. Runtime exploitation, network testing, social engineering, physical security, third-party services you consume.
Product Architecture
Definition. Whether the system's structure matches the product it now has to be, not the prototype it started as.
Sub-criteria. Data model integrity; module and service boundaries; ownership of records and state; integration and contract design; fit to current scope.
Evidence. Engineer review of schema and module structure, inter-module dependency graphs, integration contracts, repository history showing where boundaries eroded.
Out of scope. Product strategy, market fit and pricing. We assess structure against stated intent, not the intent.
Scalability & Performance
Definition. The point at which the current design stops holding, and what it does when it gets there.
Sub-criteria. Query behaviour at realistic data volume; indexing and access patterns; caching and invalidation; concurrency and connections; background and scheduled work.
Evidence. Query plan analysis at representative volumes, schema and index review, engineer review of concurrency and queue design, cloud metrics where provided.
Out of scope. Load testing, benchmark execution and capacity certification. We identify limits by inspection and modelling; a measured figure requires load testing, scoped separately.
Cloud & Delivery
Definition. Whether the software can be deployed, observed, recovered and paid for without one specific person present.
Sub-criteria. Environment separation; infrastructure as code; deployment and rollback; observability and alerting; backup and restore; cost behaviour as usage grows.
Evidence. Infrastructure configuration in code and console, pipeline definitions, deployment history from repository and CI records, monitoring configuration, billing data.
Out of scope. Disaster recovery exercises, availability commitments, tooling procurement, vendor contracts. We assess capability, not uptime.
Engineering Governance
Definition. How change is proposed, reviewed, tested and released, and what is written down.
Sub-criteria. Change control and review; test and release gates; governance of AI tooling in development; documentation and decision records; key-person dependency.
Evidence. Repository history of review and merge practice, CI configuration, documentation as it exists, interviews about actual rather than intended practice.
Out of scope. HR practice, team structure, individual performance, hiring, organisational design. We assess the process, never the people in it.
How findings are classified
Every finding gets one class. The tests below are applied in order; the first returning yes wins.
Class
Decision test
Expected response
Is data, money, availability or a legal obligation exposed now, under conditions that already exist?
Fix before the next release.
Will this cause material damage under a condition expected within roughly two quarters: growth, a new customer tier, a key departure, a disclosed advisory?
Scheduled into the current quarter.
Is a real cost being paid continuously, mostly in engineering time and change risk, compounding if left?
Planned work, sequenced around delivery.
Is there a clear benefit with no deadline and no compounding cost?
Do it when the surrounding code is open anyway.
Two rules apply to every entry. First, a finding must be located: file, line, resource or configuration item, plus the commit hash it was observed at.
Second, every finding carries a business-impact statement in plain language: which customer, which obligation, which cost, on what timescale. A finding written only in engineering terms cannot be prioritised by the person holding the budget, and prioritisation is the purpose of the register.
The scoring model
Specified, not activated
Numeric scores are not published anywhere, not on this site and not in any deliverable. Version 1.0 defines the model; it does not switch it on. Bands need calibrating against a real population before a number means anything, so the first twenty assessments under this version are being scored retrospectively. Scoring activates at version 1.1, with the calibration data published alongside it. Until then, reports use status labels.
Sub-criteria
Each is scored against a written band descriptor: absent, present but unreliable, partial, consistent with gaps, and consistent and evidenced. Descriptors are dimension-specific and printed in the report appendix.
Roll-up
A dimension's score is the mean of its sub-criteria, with each sub-criterion weighted equally within the dimension.
Findings depress the score
High findings and Medium findings each carry a defined deduction; Optimisation findings deduct nothing. Any Critical finding caps the dimension near the bottom of the range regardless of the underlying average, because a dimension containing active exposure cannot be healthy on average.
Composite
The weighted mean of the six dimension scores. By default Application Security and Code Health carry the most weight, followed by Product Architecture, then Scalability & Performance, Cloud & Delivery and Engineering Governance.
Weights are engagement-context sensitive, and we say so
Due diligence raises Product Architecture and Engineering Governance, because the buyer is purchasing changeability. A pre-security-review engagement raises Application Security; a scaling engagement raises Scalability & Performance and Cloud & Delivery. Any deviation is agreed at scoping and printed on the report's first page as a table, so results stay comparable at default weights.
Status mapping
Each dimension maps to one of three status labels: Critical, Needs Attention or Ready. A Critical finding forces a Critical status irrespective of the arithmetic.
Who assesses, and who checks them
The two-person rule
No finding reaches a client report on one engineer's judgement. Every assessment has an author, who reads the system and writes the findings, and an independent reviewer, who checks each against its stated evidence and assigned class. Both are named in the report.
Reviewer qualification
A reviewer must have shipped and operated production systems, must have carried responsibility for a live system outside working hours, and must not have authored the findings under review. For security findings, the reviewer must have worked on the control area rather than read about it.
Independence
No reviewer's compensation is linked to the sale of remediation work arising from their assessment. That is the incentive which corrupts assessment practice across this industry, and removing it is cheap.
Disagreement
A reviewer cannot silently overwrite the author. Disagreement is recorded and escalated to a third engineer outside the engagement, whose decision settles the class. Where the dispute is substantive rather than about class, and escalation does not settle it, both positions are published as a recorded dissent rather than a false consensus.
How to challenge a finding
You will sometimes know something we do not: a control applied elsewhere, a decommissioned code path, a constraint making a route unreachable. The process exists for that.
Submit it in writing
Send it to the named reviewer on your report, quoting the finding reference and stating what you believe is wrong: the evidence, the classification, the impact statement, or all three. There is no form and no charge.
We respond within five working days
In writing, with one of four outcomes: upheld, with the reasoning restated; reclassified; amended, where the evidence or impact statement was imprecise; or withdrawn, where the finding does not stand.
Every outcome is recorded
The register carries a change log showing what was challenged, by whom, and what happened, withdrawals included. A methodology whose author quietly deletes its mistakes is not a methodology. Challenges are accepted for twelve months.
What this methodology does not establish
It is not a penetration test.
We assess controls by inspection, not exploitation. We do not attack running systems, chain vulnerabilities or prove exploitability. A clean Application Security status is not evidence that an attacker would fail.
It is not a certification.
Noseberry is not a certification body and this is not an audit in the accounting sense. Findings can be mapped to the control areas a SOC 2 or ISO 27001 process examines, and that mapping supports readiness for those processes without substituting for them.
It is not a line-by-line guarantee.
Automated analysis covers the whole codebase. Human review is targeted, weighted towards privileged code paths, high-change-frequency files, data handling and integration boundaries, and the proportion reviewed by hand is stated in every report. An assessment cannot prove the absence of defects, and any methodology claiming otherwise is not being honest.
It is not a substitute for a security programme.
A point-in-time assessment tells you where you stand on a date. It does not monitor, patch, respond to incidents or watch dependencies. That is what Continuous Product Assurance exists to cover. Findings are recorded against specific commit hashes, and a codebase changing daily starts diverging from its report immediately.
It does not assess business viability.
Not the market, not the product decisions, not the commercial model, not your contracts. Licence obligations are identified as findings; interpreting them is a lawyer's job.
Version history
| Version | Date | Change |
|---|---|---|
| 1.0 | 29 July 2026 | First public release. Dimensions, sub-criteria, evidence weighting, classification tests, challenge process and scoring model defined; scoring specified, not activated. |
| 0.9 | 14 May 2026 | Internal. Evidence weighting introduced; interviews demoted to contextual evidence only. |
| 0.8 | 3 March 2026 | Internal. First consolidated draft of the six dimensions. |
Every report states the version it was produced under. Material changes to dimension definitions, classification tests or weightings increment the major version; clarifications increment the minor. Superseded versions stay available on request, so old reports can be read against their own rules.
Frequently asked questions
Because the alternative does not work. A status label is only worth something if the person receiving it can interrogate how it was produced, and an enterprise security team or an investor's technical adviser will ask. Publishing removes the argument about whether we are marking our own homework and replaces it with an argument about evidence, which is a much better argument to have. The advantage in this work is the quality of the engineers applying the method, not the secrecy of the method.
Yes, and some teams do. Everything needed for a self-assessment is on this page: the dimensions, the sub-criteria, the evidence rules, the classification tests and the scoring arithmetic. What you cannot easily replicate internally is independence. The two-person rule requires a reviewer with no stake in the answer, and inside a small team the person best qualified to review is usually the person who wrote the code. Self-assessment finds the obvious problems reliably. It is weakest exactly where teams are already blind.
Because a number implies a precision we have not yet earned. The model is specified above, but the bands need calibrating against a real population of assessments before a 68 means something defensibly different from a 74. We are scoring the first twenty engagements under version 1.0 retrospectively, and scoring activates at version 1.1 with that calibration data published alongside it. Until then, every report carries the three status labels per dimension. A defensible label beats an impressive number nobody outside our office can check.
Two controls. No reviewer's compensation is linked to remediation work arising from their assessment, which removes the direct incentive. And the classification tests above decide severity by a stated condition rather than by judgement in the moment: a High finding requires a named condition expected within roughly two quarters, not a feeling that something looks worrying. Neither control is perfect, and we are not claiming otherwise. Both are checkable, which is why they are written down here rather than asserted on a sales call.
The disagreement is recorded, not resolved quietly. Class disputes escalate to a third engineer outside the engagement, whose decision is final for the report. Substantive disagreements about what the evidence means, where escalation produces no consensus, are published in the report as a recorded dissent with both positions stated. Clients occasionally find this uncomfortable, and it is still the right behaviour. A report presenting manufactured unanimity is less useful than one showing where two experienced engineers read the same system differently.
Yes, without modification. Nothing in the six dimensions depends on how the code was produced. The framework was written for AI-built software because that is where the gap between a working product and an operable one is widest, and because generated code fails in characteristic ways: controls applied inconsistently across sessions, three solutions to one problem, a data model built for the demo. Those patterns are what the sub-criteria are tuned to catch. It runs on hand-written legacy systems too.
Assessed against a published standard
To have this method applied to your product, start with the AI Code Audit, or the AI Application Security Audit if a customer security review is the reason you are here.
Related resources
Go deeper with our thinking
From Spreadsheets to a Scalable Platform – building Canada’s Most Tenant- First Coliving System
Read case study
How to Build a SaaS Product
Read guide
The MVP Development Guide
Read guide
Building a Digital Engineering Roadmap That Aligns With Business Goals
Read article
Engineering IoT Products: Key Challenges and Best Practices
Read articleNot sure where you stand?
Take a free two-minute readiness scorecard built for your industry.