AI & applications
Audit AI-built code before production
Updated
A security audit for AI-built code examines the resulting attack surface rather than how the code was produced. Authentication, authorisation, tenant isolation, secret handling, input validation, dependencies, logging and secure deployment matter. Prompting replaces neither threat modelling nor reproducible tests.
For prototypes and rapidly developed internal or customer-facing applications before production rollout.
When is this approach the right fit?
For prototypes and rapidly developed internal or customer-facing applications before production rollout.
The first step is therefore not tool selection but a decision map. It separates essential processes from habits, names dependencies and shows which parts already sit inside Shopify's standard capabilities. Only the remaining gaps justify apps, middleware or custom development.
NICCOS considers a topic ready for delivery only when the objective, non-goals, owners and acceptance are documented. This prevents a concise page title from turning into an open-ended transformation programme whose effort nobody can explain reliably.
What architecture does it require?
The audit starts at trust boundaries: browser, API, database, third parties, model and admin surface. Data flows, permissions and secrets are then traced to concrete code paths.
The architecture is shaped around change frequency, outage impact and team ownership. A process that runs every minute needs different guarantees from a nightly catalogue export. Editorial content requires different approvals from a price or an order.
We always plan an observable path: stable IDs, logged state transitions, repeatable processing and a dashboard for exceptions. Without that operating layer, a technically working connection is only a demo rather than a dependable commerce solution.
Which data and process decisions come first?
Production data is not copied into unsafe tests. Test data represents sensitive fields and tenant boundaries realistically without multiplying real customer data.
For every relevant object we document source, destination, key, update frequency, conflict rule and error path. It sounds formal, but it removes the late loops caused when two systems hold the same field with different meanings.
Data is not merely migrated or synchronised; it is reconciled against business meaning. Samples must cover variants, taxes, markets, discounts, returns and historical exceptions. A successful import without business reconciliation proves only that files were read.
What does delivery look like from discovery to operations?
The delivery path is deliberately split into verifiable outcomes. Every phase ends with an artefact, a decision or test evidence. The team can change scope without losing the overall plan, and risks become visible before they block the critical path.
The order follows risk: data and processes first, then architecture and prototype, followed by implementation, migration, acceptance and staged rollout. Interfaces are not approved against sample data, and integrations are complete only after failure and recovery paths have been tested.
- Threat model and data-flow diagram
- Code and configuration review against OWASP categories
- Tests for auth, tenant isolation and input boundaries
- Prioritised findings with concrete remediation
- Launch gate and documented residual risk
Which risks require active control?
These risks need explicit controls in discovery, testing and monitoring. Before implementation, each one receives an owner, evidence requirement and fallback path.
Authorisation is checked only in the interface and can be bypassed through direct API calls.
Secrets or personal data end up in client bundles and logs.
Prompt injection can reach privileged tools or internal data.
What does NICCOS add beyond a standard implementation?
AI-built code receives the same security requirements as handwritten code, plus specific checks for prompt injection, tool misuse and uncertain dependency provenance.
We connect commerce decisions with SEO, data quality, analytics and operations. A solution is not complete when the happy path works. It must be discoverable, measurable, accessible, translatable and understandable to the team after the project.
We also document when the standard is the better decision. Not every requirement deserves custom software, not every data flow needs real-time processing, and not every historical exception should be carried into the target architecture.
How is quality measured before launch?
Acceptance measures are set before implementation and tested with real data. Functional tests alone are insufficient: completeness, speed, fault tolerance and the team's ability to recognise and classify exceptions are what matter.
- Gate 1
- No open critical or high-severity findings
- Gate 2
- Authorisation is tested server-side and per tenant
- Gate 3
- Secrets, logs and dependencies pass automated checks
| Check | Expected evidence |
|---|---|
| Gate 1 | No open critical or high-severity findings |
| Gate 2 | Authorisation is tested server-side and per tenant |
| Gate 3 | Secrets, logs and dependencies pass automated checks |
Keep exploring
Related playbooks
FAQ
Frequently asked questions
When is this approach useful?
For prototypes and rapidly developed internal or customer-facing applications before production rollout. The business value, data ownership and operating model must be explicit before implementation begins. A technology decision without those three points merely pushes unresolved questions into delivery.
How should the project start?
With a short discovery sprint covering current processes, interfaces, volumes, exceptions and acceptance criteria. Threat model and data-flow diagram The scope can then be split into testable delivery packages instead of being estimated from a feature list.
Which data must never be maintained twice?
Production data is not copied into unsafe tests. Test data represents sensitive fields and tenant boundaries realistically without multiplying real customer data. Every object needs one system of record, a defined direction and an owner for corrections. Double maintenance is not an integration pattern; it is a reconciliation problem waiting to happen.
What belongs in acceptance testing?
Acceptance covers visible behaviour as well as failure modes, permissions, retries, monitoring and realistic data. No open critical or high-severity findings The solution is production-ready only after load and partial outages have been addressed.
What is the NICCOS point of view?
AI-built code receives the same security requirements as handwritten code, plus specific checks for prompt injection, tool misuse and uncertain dependency provenance. We prefer understandable standards, a small number of justified exceptions and measurable release gates. That lowers project cost and leaves the internal team with a system it can operate.
Primary sources
Official documentation used for capabilities, constraints and implementation guidance.
Next step
Settle the decision before the build
We assess a security audit for AI-built applications against real processes, data and operating requirements, then turn it into a deliverable scope with clear release gates.