Extra fields are rejected
The typed contract prevents a model from quietly expanding the API or returning prose where downstream code expects a value.
Synthetic briefs · AWS deployment-ready · Human review preserved
BriefSpec
An auditable Generative AI reference implementation that identifies candidate risks in marketing briefs, grounds every finding in trusted rule and evidence catalogs, returns strict JSON, and preserves accountable human review.
✓ Clean-room implementation
✓ Offline evaluation path
Bedrock adapter included
Deployment-ready ≠ deployed
WorkflowDetect · validate · ground · review
Core stackBedrock · Lambda · Pydantic · SAM
ControlsStrict schema · allowlists · private ingress
SourceView repository ↗
01 / Problem
A confident finding is useful only if policy and evidence remain trusted.
Marketing briefs can contain absolute outcomes, health claims, hidden endorsements, sensitive-audience targeting, unsupported superlatives, or broad environmental language. A fluent model can identify plausible concerns while still inventing a policy, severity, recommendation, or citation.
BriefSpec treats every provider as untrusted. A provider may propose rule IDs, claim text, rationales, allowed evidence IDs, confidence, and a review flag. The service rejects unknown values and enriches accepted findings with severity, actions, categories, and source metadata from versioned trusted catalogs.
Responsible-use boundary
The system produces grounded risk-triage signals. It does not approve a campaign, provide legal advice, or replace accountable human review.
02 / System
Untrusted provider output inside deterministic trust boundaries.
IngressIAM invocationprivate · request ID
BoundaryPydantic requeststrict fields · 12k limit · hash
AnalyzeLocal rules / Bedrockstructured provider output
GroundTrusted catalogsrule + evidence allowlists
ReturnReview recordversions · safe CloudWatch logs
The optional Bedrock provider uses the Converse API and a strict structured-output schema. AWS imports are lazy, so the package and tests work without cloud credentials. A deterministic local rule provider makes CI fast and prevents pull requests from spending tokens or depending on a remote model.
The AWS SAM template creates no API Gateway or Function URL. The Lambda is private by default, has bounded concurrency and log retention, and may invoke only the exact model resource supplied at deployment.
03 / Contract
The provider proposes; the service controls policy.
Only catalog IDs survive
Unknown or rule-incompatible evidence IDs fail closed. Source organization, URL, retrieval date, and summary come from trusted files.
Severity is not model-generated
Category, severity, recommendation, and acceptable evidence are loaded from the versioned policy catalog—not accepted from a provider.
Prompt, rules, and evidence are first-class
Every response records all three versions, making regressions traceable instead of anecdotal.
Findings remain suggestions
Risk signals and rationales support triage; they never constitute approval, legal advice, or an autonomous publishing decision.
Content stays out of metrics
Operational logs record IDs, timings, versions, status, and counts—not the submitted document body.
04 / Evaluation
Regression testing for the workflow, not vibes about the model.
A 12-example labeled synthetic fixture set covers six rule categories, safe claims, multiple simultaneous findings, disclosed endorsements, and an instruction-injection attempt. The default evaluation measures the deterministic provider against examples designed from the same rule catalog; it is a regression sanity check, not evidence of real-world model quality.
PrecisionPipelinesynthetic local sanity set
RecallPipelinesynthetic local sanity set
Finding F1Pipelinesynthetic local sanity set
Exact-match ratePipeline12 deterministic fixtures
What is deliberately not reported
- No production accuracy claim: the examples are synthetic, intentionally small, and derived from the same deterministic rules.
- No Bedrock latency or cost claim until a timestamped live evaluation artifact exists.
- No legal or policy compliance score; these require domain owners and an appropriate gold set.
05 / Threats
The document is data, never an instruction.
The threat model treats prompt injection, oversized input, sensitive data, unsupported formats, malformed model output, evidence forgery, replay, cost abuse, and overly broad IAM as concrete engineering risks.
Delimit and minimize authority
Brief text is clearly separated from system instructions; document content cannot request tools or alter the schema.
Fail closed on malformed output
Schema or evidence failures return a controlled review state rather than partially trusted data.
Least privilege and bounded cost
The private function has reserved concurrency and may invoke only the configured Bedrock resource; no public endpoint is created.
Rollback by version
Prompt, schema, model, and code versions are recorded so a degraded release can be identified and reverted.
06 / Reproduce
Cloud-shaped code with a free, deterministic local path.
python -m venv .venv
.venv/Scripts/activate
python -m pip install -e "projects/briefspec[dev]"
briefspec analyze --text "Promise a guaranteed result for every customer."
briefspec evaluate --dataset projects/briefspec/data/synthetic_eval.jsonl
pytest projects/briefspec/tests
# Optional, requires AWS credentials and an allowed model
briefspec analyze --provider bedrock --text "Marketing brief text"
Deployment evidence required before saying “live”
A successful SAM build and deploy, a CloudFormation stack ID, sanitized CloudWatch invocation evidence, a cost snapshot, and a teardown command. Until those exist, this portfolio accurately labels the system deployment-ready.
Amazon Bedrock documents structured JSON output through the Converse API. This adapter uses that contract and validates the response again locally. A real deployment should add and evaluate use-case-specific Bedrock Guardrails separately; this repository does not claim that configuration.
More personal projects