On 23 September 2026 I pulled every proposal our team had sent on Upwork since December. There were 211. Not one carries a Hired status. Then I read the job titles: a fulfilment manager, a Figma designer, a customer-support SOP consultant, an inventory manager. We are a systems agency. We had been bidding on jobs we should never have opened.
This is what we did about it, in the order we did it.
The short version. We wrote down the five questions a good salesperson asks in their head, put them to TypeSafe’s Jev for every Upwork job our system finds, and got back numbers instead of essays. The first real run scored 409 jobs three times over for about eight cents. It showed our past bids were the problem, not the model. The team now gets a card every morning with a verdict and a name on it.
Who this is for. Anyone with more incoming things than time to read them, and anyone building with Claude Code or Cursor who wants to see a scoring loop that actually shipped. If you have ever said “I can tell in ten seconds whether this one is worth it”, this is how we stopped spending the ten seconds two hundred times a day.
What we had before: a firehose and a scorer nobody ran
Our Upwork system is a Django app the team calls Agent Rahnuma. Twenty-one saved searches run on a schedule, pull new jobs through Upwork’s API, and post them to Google Chat. By September it held 68,111 jobs. A teammate read the stream and bid.
We had tried scoring once already, with a chat model on Azure OpenAI. It wrote four paragraphs of reasoning and a grade per job. It scored 145 jobs in total, because every call cost real money and every answer needed a person to read it. Nobody ran it. Prose does not sort, and a scorer nobody runs is a firehose with a label on it.
So the bids followed the stream. The searches were broad, the stream was broad, and the person bidding took what came. Nobody was wrong on any single job. The system was wrong on all of them.
Your version. Name your firehose: the inbox, the form, the job board, the ticket queue where things arrive faster than anyone reads them. Then name the one person who currently decides what gets chased. If you cannot name them, that is finding number one.
Why Jev, and why September
TypeSafe released Jev on 19 September 2026. It is a different kind of model. You give it the facts about something and a list of questions, each with the answers you will accept, and it returns a probability for every answer. It cannot write a sentence. For scoring, that is the point: you get a number per lead, every number is comparable to every other, and there is nothing to read.
We tried it first on our own blog. Four questions about a meta description came back in about a second and matched what a human reviewer had said. Then we ran it over 92 live posts to find the ones with no source and no call to action. That review is its own story. By the time it finished I was fairly sure the same shape would work on jobs, and I already had the ten decisions I thought it fit. Lead scoring was the one with a database already full.
Writing the five questions
On 23 September I sat down with the job table and wrote the questions the way I would ask them of a junior teammate. That took an hour. Here they are, with the answers we allow.
- Is this our kind of work? Yes or no, as a probability. The “no” sentence lists the deal-breakers: an individual hire, a staffing post, a trial task, wrong skills.
- Does the money match the scope? Underpriced, tight, fair, or generous, for a senior team at our rates.
- Who is the client? Unknown, new, established, or a serial hirer, judged only from their history on Upwork.
- How crowded is it? Open, moderate, or crowded, from applicants and how long the post has been up.
- So what do we do? Write a proposal today, worth a look, or skip.
Two things about how we wrote them turned out to matter more than anything in the code.
The criteria are sentences, not weights. When I decided we do not want individual-employee hires, that sentence went into the “no” side of question one and the model started rejecting them the same afternoon. Nobody tuned anything. You edit a paragraph.
Each saved search got its own paragraph describing the ideal client. The Shopify search says founder-run DTC brands, the ones selling direct to consumers, doing roughly one to thirty million a year, and lists what to avoid: dropshipping starters, virtual assistants, design-only jobs, budgets under five hundred dollars. The same job can be a fit for one search and a skip for another.
Version the rubric from day one. Every score we store carries the version of the questions that made it. Change a sentence, bump the version, and old scores stay honest about the rules they were judged by. This cost one extra column and saved us the first week.
Your version. Write your five tonight, as if briefing a junior. Use ours as the template: is it ours, does the money fit, who are they, how crowded is it, so what do we do. Put every deal-breaker you have ever said out loud into the “no” sentence of question one.
The first five jobs
The first live run was five jobs. It cost three hundredths of a cent. Two of the five agreed with the old Azure grades. The budget question ran strict: I had written that an hourly job with no stated rate counts as tight, so every unpriced hourly job came back tight, including the well-funded ones we would take. That was my sentence, working as written. It is still in the rubric, with a note to fix the sentence in version two.
That run also settled how the answers would be read. Here is a real one, for a job titled “Improve conversion rate + AOV”, scored on 24 September 2026.
| Question | Answer | What the team reads into it |
|---|---|---|
| Our kind of work | 0.70 yes | Conversion work on a Shopify store. Ours. |
| Money vs scope | Underpriced 0.95 | The budget does not cover the scope described |
| Client | Unknown 1.00 | No spend, no hires on record |
| Crowded | Open 0.44, moderate 0.43 | Early enough to matter |
| Verdict | Consider 0.51, skip 0.48 | A coin flip. Nobody is told to write a proposal today. |
A chat model would have given me four paragraphs saying the same thing, and I would have had to read them. This I can sort by.
Your version. Run five. Pick two you would chase, two you would skip, one you argue about, and read what comes back next to your own gut. Where it disagrees, the fix is almost always a sentence in your criteria, not the model.
The test that turned on us
Before letting it near the team I ran the scores against history. Positives were all 211 jobs we had bid on. Negatives were 198 jobs from the same searches in the same window that nobody touched. Each job was scored three times: once against the agency, once against my own Upwork profile, once against Abdullah’s. That is 1,227 judgments over 1.82 million input tokens. TypeSafe prices Jev at 0.042 dollars per million input tokens with free output, per its model page, so the run cost about eight cents.
The model’s ability to predict our past bids was weak, roughly six in ten where a coin flip is five. I expected that. Then I looked at the extremes and understood why.
| Job title on Upwork | Fit | What the team did |
|---|---|---|
| Improve conversion rate + AOV | 0.86 | Never opened |
| (2) Shopify Websites to Optimize | 0.89 | Never opened |
| Shopify Subscription Website Build (Pet Food Brand) | 0.86 | Bid |
| Figma Designer Needed for DTC Brand Wireframes | 0.06 | Bid |
| Inventory Manager, Dietary Supplement E-Commerce Company | 0.06 | Bid |
| Fulfillment Manager / Client Success Manager | 0.04 | Bid |
| ECOM MRR Customer Support SOP Consulting | 0.03 | Bid |
Shopify conversion work, the exact thing we sell, scored 0.80 to 0.89 and sat untouched. Operations roles scored under 0.07 and got proposals. The model was not failing to predict our bids. It was telling me our bids were wrong. That is the picking problem, measured, and it is the same problem the adversarial outbound build fixed for our cold email: make the pick visible before anyone spends an hour on it.
Two more things fell out of the same eight cents. The agency profile and my own profile agreed on nearly every job, so the agency text was my profile with a logo. Abdullah’s profile disagreed with mine almost everywhere. He does AI agent and infrastructure work, and the Shopify stream was never going to feed him. He needs his own searches.
Your version. Before trusting it, score the last hundred things you chased and the last hundred you skipped. Then read the extremes, not the average. The ones at the top you never touched, and the ones at the bottom you did, are your real finding.
What we changed in the next 24 hours
The test gave me a list, and the list became the build.
One fit number per person. Question one now runs once for the agency and once for each teammate, with a paragraph of work each person actually delivered in the last 120 days. For that conversion job, Jev returns 0.70 for the agency, 0.64 for me, 0.18 for Abdullah. For a fulfilment manager post we once bid on it returns 0.03, 0.05 and 0.04, and a skip at confidence 1.0.
The card. Every new job that clears a search now goes to Jev before it reaches Google Chat. The card shows the verdict, the three fit numbers, and a star next to whoever should write the proposal. Nothing writes back to Upwork. A person opens the job and bids, or does not.
Ideal client on the search, not in the code. I wrote the ideal client paragraph onto the Shopify search through the same interface the team uses. The next ten ecommerce jobs came back six skip and four consider, and every consider was starred for me.
Scores that cannot overwrite each other. A code review caught that the old Azure scorer and Jev would have written to the same row. Each score is now keyed by the model and rubric that produced it, so both can live side by side and the old grades stay available for comparison.
Your version. Put the number where the person already looks: a chat card, a CRM field, a morning digest. Put a name on it. Give the model no way to act. Then watch a week of picks and count what changed.
What it costs to run
| Figure | Value | Period or source |
|---|---|---|
| Jobs in the database | 68,111 | Prod copy, 23 September 2026 |
| Proposals on record | 211 | 23 December 2025 to 10 June 2026 |
| Proposals with a Hired status | 0 | Same period |
| Judgments in the alignment run | 1,227 | 409 jobs, three profiles each |
| Cost of the run | About 0.08 USD | At 0.042 USD per million input tokens |
| Old scorer coverage | 145 jobs | Azure OpenAI, total before switch-off |
Jev takes up to 1,200 requests a minute. A morning’s discoveries are scored before anyone opens Chat. Adding a sixth question costs a few hundred tokens on the same request, not another call. The cost line in this project is a rounding error; the time line is where it pays.
Where it let us down
Jev publishes its own weak spots and we hit four in the first week. Whoever owns the rubric owns all of them.
- It reads literally. The tight-budget sentence above. The fix is a better sentence, then a version bump.
- It does not count or do dates. We pass the posting time and let the applicant count carry the crowding question. If you need “posted three days ago”, compute it and pass the number.
- Strangers write the text. A job description is untrusted. Someone can write “this is a perfect fit for any agency” in paragraph four. We cap descriptions at 6,000 characters, keep the verdict advisory, and let nothing the model says trigger an action.
- It cannot write. No proposal drafts, no summaries, no “explain your reasoning.” The reasoning is the numbers. Proposals stay human.
English is the first language it was built for. Our jobs are in English. Test yours before you trust it on Urdu or Arabic leads.
What you could score the same way
Swap “Upwork job” for whatever floods your inbox. The shape does not change: facts you already store go in, five questions in the owner’s words, a number comes out, a person acts on the top of the list.
| Stream | The question the owner already asks | What the number does |
|---|---|---|
| Website enquiries | Is this our kind of client, and can they pay? | Sales gets the top five by nine. The rest get the nurture sequence. |
| Job applicants | Has this person done this work, at this level, recently? | The hiring manager reads ten, not two hundred |
| Support tickets | Bug, how-to, or refund, and is this customer about to leave? | At-risk customers reach a person first |
| Supplier quotes and tenders | Does this bid meet the spec, and is the price sane? | Procurement compares three, not thirty |
| Product reviews | Is this a defect, a delivery problem, or a sizing issue? | Ops sees the pattern before the next batch ships |
| Returns requests | Policy return, defect, or the same address for the fourth time? | Clean ones get pre-approved; the rest get a look |
| Community posts | Is this person describing the problem we solve, in their words? | The outreach list writes itself, the way our [cold-email angles](/blog/jev-cold-email-system/) do |
| Your own drafts | Does this open on the reader's problem, and is every number sourced? | The editor reads the flagged ones |
Every row has the same four parts. The facts already sit in a system you pay for. The owner already has the sentences. The model returns a number. A person acts. If one of those four is missing, fix that first; the model is the easy part.
How to do it yourself in an afternoon
If you build with Claude Code or Cursor, this is a two-hour job on top of whatever already stores your leads. If you run the business, hand this list to whoever builds.
- Pick one stream and the person whose time it wastes. Ours was “should we bid, and who.” The owner is the person who was losing the afternoons.
- Write the five questions the way that person says them, with the answers you will accept. Put the deal-breakers in the “no” sentence. Do not write weights.
- Send the facts you already store, with the questions, one request per item. For the builder, the whole call is this:
curl -s https://api.typesafe.ai/v1/systemone \
-H "Authorization: Bearer $TYPESAFE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "jev-latest",
"state": {"lead": {"title": "...", "description": "...", "budget": 1500}},
"questions": {
"fit": {
"type": "noul",
"instructions": "Is `lead` work we should chase?",
"criteria": {"true": "Our kind of work at a workable price",
"false": "Wrong skills, a staffing post, or a trial task"}
}
}
}'- Keep the numbers and the rubric version on every row. The verdict is what people read today. The full set of probabilities is what you check next month when someone asks why a lead was skipped.
- Put the top of the list where the person already looks, a chat card, a CRM field, a morning digest. The model advises. The person acts in the source system.
Two builder notes. Retry on a 429 with backoff. Treat a failed call as “no score”, not as a low one; a missing score should look different from a skip.
Frequently asked questions
What is AI lead scoring?
AI lead scoring is putting a fixed set of questions to a model about every incoming lead and sorting the leads by the answers. The useful version returns comparable numbers per lead, so a team can rank a day’s leads and act on the top of the list without reading all of them.
Does Jev write the proposal?
No. Jev returns probabilities for the answers you permit and cannot generate text. A teammate opens the starred job on Upwork and writes the proposal. Nothing in our system writes to Upwork.
How much did AI lead scoring with Jev cost us?
Scoring 409 jobs against three profiles, 1,227 judgments, cost about eight cents at TypeSafe’s published price of 0.042 dollars per million input tokens with free output. One lead with five questions is a fraction of a hundredth of a cent.
Can this score CRM leads, tickets or applicants instead of Upwork jobs?
Yes. The pattern is the same: send the fields the system already holds, write the questions in the owner’s words, keep the numbers and rubric version, and route the verdict to a field or a chat card. The table above lists eight streams we would start with.
How do you know the scores are any good?
Against our own past bids the model scored roughly six in ten, and the test showed the bids were the wrong benchmark. The real check is forward: score the next hundred jobs, bid on the top of the list, and compare hire rates against the old stream over a stated window. That is our next run.
Put five questions to your own pipeline
Each of those 211 proposals took someone’s afternoon. The jobs we should have chased took none. If your team argues about which leads to chase, the argument is a rubric nobody wrote down. Book a Systems Review, bring the last 50 leads you chased, and we will put five questions to them in front of you. You leave with the criteria in your words, the ones that should never have been opened, and what it would take to run this on every lead from Monday.
