You paste a prompt into a model, read the product description it writes, and sign it off. Then you hand that same prompt to someone with the rest of the catalogue to fill. A week later the fabric is wrong on one listing, a care line is missing on another, and nobody can say which version of the prompt wrote which.
The model did not get worse. Nothing about the job was ever pinned down: not the inputs, not the shape of the output, not the standard an output has to clear before a customer sees it.
Prompt engineering for LLM performance is not better wording. It is making one job repeatable: pin the inputs and the settings, state the output shape, score a sample of outputs against a written standard, and log every run. Wording is the last thing you change, and only when the score tells you to.
This guide is for the operator who owns the output after it ships
This is for the person putting a model into a job the business already does. Product copy at catalogue scale, first-pass support triage, a daily summary of order exceptions. You are accountable for what ships, so a model that impressed you once is worth nothing.
It is not for anyone benchmarking models, and not for the person who needs one clever answer once. If a job runs once, wording is the whole game. If it runs a thousand times, wording is a rounding error.
Autonomous Technologies builds and runs the systems behind Shopify stores: the integrations, the data and the jobs that keep running after launch. Putting models inside those jobs is how we learned the rules below.
Prompt engineering, plainly: writing the instruction, the input and the output contract around a model so the same job comes out the same way twice. Most of that work sits outside the sentence you type. It is the engineering around the model, and it is the part that survives a vendor update.
Pin everything except the wording before you judge a prompt
Reproducible does not mean identical. It means an output whose shape you can predict, whose source you can trace, and whose quality you can argue about with a number rather than a feeling.
So start with the things that move under you. Temperature is the setting that controls how much randomness a model adds when picking the next word. Zero means always take the most likely one. Most operators assume zero buys determinism. It does not.
Thinking Machines Lab ran the same prompt 1,000 times at temperature 0 and got 80 different completions, published 10 September 2025. The cause was server load changing the batch size, not the prompt or the settings.
| What was measured | Figure | Source |
|---|---|---|
| Completions requested at temperature 0 | 1,000 | Thinking Machines Lab, 10 September 2025 |
| Different completions returned | 80 | same run |
| Tokens identical across all 1,000 | the first 102 | same run |
| The split at token 103 | 992 one way, 8 the other | same run |
| Different completions after their fix | 1 | same run, batch-invariant kernels on |
So pin what you can, then judge a sample rather than one output.
| What to pin | What it costs you unpinned | How you hold it |
|---|---|---|
| Model name and version | The vendor moves the default and your copy changes with no edit in your files | Record the exact model string on every run, never "the current one" |
| Temperature and sampling settings | Same prompt, different length and tone across one afternoon | Set them explicitly in the call, never inherit the default |
| The input fields | A listing with no size chart reads as an invitation to invent one | Reject the input before it reaches the model, not the output after |
| The output shape | Every step downstream has to guess where the answer starts | Ask for named fields, then fail the run when one is missing |
| Who may edit the prompt | Two people fix opposite things in the same week | One owner, one file, one change log |
A prompt is finished when a sample of its outputs passes a written standard
The technique list is not the scarce part. Wei and colleagues published chain-of-thought prompting on 28 January 2022. Eight worked examples took a 540-billion-parameter model to state of the art on a grade-school maths benchmark. Chain prompting and defensive instructions are taught material, covered in Packt’s Prompt Engineering: For Optimal LLM Performance from December 2023.
What is scarce is the standard. An evaluation set is a fixed group of real inputs plus the written standard their outputs must clear. Without one, “it got better” means “I liked the last three”.
Write the standard before you touch the prompt. Pull real inputs from the job, weight them towards the ugly cases, and say what a pass looks like field by field. Then split the judging. A model can decide whether a sentence names a source. It cannot count the words in that sentence, so counting belongs in code.
This page went through exactly that gate. Every page on this site is scored by a model against a fixed rubric of 36 questions across six lenses: search fit, readability, structure, visuals, conversion and brand rules. The model answers only the judgement questions. Word counts, caption counts, sentence lengths and banned characters are counted by a script, because a model asked to count returns a number that sounds right.
| Field | The rule | Why the number exists |
|---|---|---|
| Subject line | 2 to 5 words | Long enough to name the subject, short enough to read at a glance |
| First paragraph | 35 to 45 words | The opener carries one cited fact and nothing else |
| Second paragraph | 30 to 40 words | One bridge sentence, one reason this reader |
| The ask | 8 to 18 words | One reply request, never two |
| Whole body | 110 words maximum | The examples we approved run 94 to 103 words |
| Any cited market fact | 18 words maximum, dated inside 90 days | Older than a quarter and the reader who lives in that market notices first |
Every factual sentence in that copy carries a claim id pointing at a stored, verified claim. The gate re-checks three things: that the id exists, that the claim was verified, and that the source text actually supports the sentence. The same three checks run in three places, print one error per line, and exit non-zero. Nothing reaches a person until all three pass.
The rule worth stealing: a model may judge, but it may not count, and it may never be the only thing that checks its own work. Counting, formats and eligibility belong in code that fails the run.
Log the run or you cannot tell drift from a bad day
Drift is the same prompt producing worse output this month than last, with nobody having edited the prompt. You only see drift if yesterday is still on disk.
One trap before you trust your automation: a headless model run that prints FATAL can still exit with a success code. Our research pipeline judges a stage on the files it actually wrote, not on what the process returned. Exit codes are not evidence.
| What you store | Why you need it later |
|---|---|
| The exact model string and settings | Tells you whether the vendor moved, or you did |
| The input as it was sent, after validation | Reproduces the run, and proves the input was not the problem |
| The full output, including the rejected ones | Rejected outputs are the honest test cases |
| The gate result, field by field | Turns "worse" into which field, on which inputs |
| Who or what triggered the run | Names the owner when the same failure repeats |
Eleven deterministic scripts produce the figures our store audits quote. The model never invents one. It reads what the scripts measured and writes the explanation: measurement in code, language from the model.
The guardrail is the structure, not the polite instruction
Prompt injection is when text the model reads carries instructions it follows as if you had written them. That text arrives in a customer message, a product feed or a web page.
OWASP ranks it first in its 2025 list for LLM applications, with seven mitigations. Four are architectural: constrain the model’s role, define the output format, enforce least privilege, segregate untrusted content. Three are operational: filter inputs and outputs, require human approval for high-risk actions, test adversarially. The same entry says it is unclear whether fool-proof prevention exists, because the model is stochastic. The March 2025 NIST taxonomy, AI 100-2 E2025, catalogues the same attacks as open problems rather than solved ones.
Read that as an instruction about architecture. A line in your prompt asking the model to ignore malicious instructions is a request, not a control. The control is what the model can reach, and what happens to its output before anything acts on it.
On the phone agent we built for our own product, the model talks and decides nothing that carries liability. Whether a number may be dialled is settled in code: the suppression list first, the calling window second. That is 18 state rules plus the federal default, held by 27 tests, live since April 2026.
The rule that build runs on: fail open where a missing part only costs quality, fail closed where a missing part creates liability. A missing adjective ships. A missing consent record stops the campaign.
What breaks first, and who owns it
The objection here is fair: this reads like a lot of scaffolding for something you can type in ten seconds. The standard for one job is usually a single page, and you write it once per job, not once per prompt. The scaffolding is what lets you change the prompt on a Tuesday without reading 400 outputs on Wednesday.
| What breaks | What you see | Who owns it |
|---|---|---|
| The model changes underneath you | Output shifts with no change in your files | Whoever holds the vendor account, with the version pinned and the sample re-run monthly |
| The input degrades | Blank fields, and confident invented answers filling them | Whoever owns the catalogue or the ticket form, validating before the call |
| The prompt gets edited in a hurry | Two fixes in opposite directions in one week | One named owner and a change log |
| Nobody reads the failures | A customer finds the same error for a third time | The operator, reading the failure log weekly |
| The job outgrows its sample | A new category appears that the evaluation set never contained | The operator, adding it to the sample before scaling |
Questions operators ask before putting a model into a real job
What is prompt engineering for LLM performance, in one sentence?
It is the work of making one model job repeatable. You pin the model, the settings and the inputs, define the output shape, score a sample of outputs against a written standard, and log each run so you can tell a bad day from real drift.
How many test cases do I need before I trust a prompt?
Enough to cover the shapes the job will really see, weighted towards the ugly ones. Start with the inputs that already caused a complaint. The number matters less than whether every category in the job is represented and whether the standard is written down.
Do I need a prompt management tool to do this?
No. A file with the pinned settings, a folder of test inputs, a written standard and a log of runs will get you most of the way. Buy a tool when the log outgrows what you can read, not before.
The model is right in the chat window and wrong in the job. Why?
In the chat window you supply the missing context without noticing, and you judge each answer as it arrives. In the job, nothing supplies context and nobody reads every output. Write down what you were doing by hand, then make it part of the input.
How often should I re-run the evaluation set?
On every prompt change, and on a fixed schedule too, since the model can move without you. Monthly suits most jobs. Re-run the day your vendor announces a model update.
What this gets back, in hours
Count the handoffs in your store where a person reads machine output and decides whether it can ship. Product copy read before publishing, support replies read before sending, exception summaries read before anyone acts. Each one is an hour a week that nobody budgeted for, and it does not go away because the model got better. It goes away when the job has a standard and a log.
