An AI system on a zero budget: six decisions, three mistakes

On the side I run Freelancerpilot, a tool for freelancers: it searches several project marketplaces, scores every project it finds with a language model and sends matches by email. It is free, with no account requirement, no subscription and no profit motive.

That is where the most interesting constraint of the last few years comes from: the system must cost close to nothing to run. No budget you can raise when things get tight. No invoice you pass on to a client. Whatever the free tiers of the providers allow is the budget.

The running costs are accordingly near zero: the server exists anyway, the domain costs a few euros a year, and the model calls stay inside the free tiers. What the project really costs is working time. That distinction matters, otherwise the rest of this article reads like „AI is free", and that would be the wrong message.

The rule forced me into decisions that in paid projects I often make only after the first surprising invoice. And in hindsight they are almost all decisions I would recommend to clients anyway. Hence this article: not as a product pitch, but as a collection of six decisions and three mistakes I made along the way.

The task: classify, do not compose

Before architecture, look at the task. The model receives project listings and returns three things for each:

• Is this a freelance project at all? A yes or no that has to be right
• A summary in two or three sentences
• Keywords with weighting: which technologies appear and how central they are

This is text classification with structured output. It is not a task that needs a frontier model, and it is not a task where creativity helps. That assessment is the most important decision of all, because it determines every other one.

Skip it, reach reflexively for the strongest available model, and you pay ten to fifty times the price for a task a small model solves just as well. In a side project you notice immediately, because the quota is exhausted after an hour. In a company project you notice after the first monthly statement.

Decision 1: the smallest model that does the job

In the code the model choice is a preference list, not a single name:

// Preferred Gemini models, in case the configured name disappears.
// The lite variants first: they have the most generous free quotas,
// and the task does not need a frontier model.
var geminiModelPreferences = []string{
	"gemini-2.5-flash-lite",
	"gemini-2.5-flash",
	"gemini-2.0-flash",
	"gemini-flash-latest",
}

The order is the actual statement: smallest first, then upwards. Not the other way round.

That an older model sits in the list is not an oversight. A name there is a preference, not a promise: once it disappears from the lineup, the resolution from decision 4 skips it and takes the next one that actually exists. Such a list is allowed to age without anyone maintaining it. That is the difference from a single hardcoded name, and this system has already come to a halt over exactly that.

The common approach is the opposite: start with the biggest model because it will „definitely be enough", and optimize later. Later rarely comes. The reverse path costs an afternoon: take the smallest model, check the output against fifty real examples, and only step up where you can point at actual failures.

The side effect is speed. Small models answer faster, and for a job that works through thousands of listings per run that is not a comfort question. It decides whether the run finishes in minutes or hours.

Decision 2: batch instead of asking one by one

The second lever is even more mundane and even more effective: do not ask about each project separately, send several in one call.

func (s *AIService) ProcessProjects() {
	batchSize := 19
	projects, err := s.repo.GetUnprocessedProjects(batchSize)
	...
}

One request instead of nineteen. That saves not only nineteen connection setups but, more importantly, nineteen copies of the system part of the prompt. And with a classification task that has clear rules, that part is the longer one: the instruction about what makes a freelance project runs to several hundred words, the individual listing often to a few dozen.

Ask one by one and you pay for that instruction with every single project. With providers that bill per token, batching is therefore the cheapest optimization available, and with free tiers limited per request and per minute it decides whether the processing keeps up at all.

Why nineteen exactly? The trigger was a per minute request limit, and the number has stayed that way since. That is more honest than any retrospective justification, and it describes the normal case: such constants come from a concrete limit and outlive its cause.

The upper bounds are nameable regardless. The context window has to hold all projects plus the answer. And the larger the batch, the more expensive a failure, because then the whole group is affected instead of a single project. Anyone raising the size should measure both: whether answers are still complete, and what a failed attempt costs.

Decision 3: three providers instead of one

The system does not talk to one provider but to a chain: Groq, then Mistral, then Gemini. Each with its own free quota, each a full substitute for the others.

The first reason is banal: three free quotas are more than one. The more important reason is availability. A model provider is a third party system whose reachability you do not control, and the free tiers are predictably the first to be throttled. With only one, you stand still the moment that one reports a limit.

The precondition is that the application is not written against one provider. In the code each provider hides behind the same narrow interface: in goes a prompt, out comes a JSON answer. Whatever peculiarities a provider brings stay in its own file. The rest of the system does not know who it is talking to.

That is the same thought I preach to clients about payment providers, and it applies here just as much: the abstraction is not an end in itself, it is the precondition for being able to switch provider at all, planned or in the middle of the night.

Mistake 1: a fallback that only catches one type of error

The chain existed from the start. The processing still came to a halt one day.

Groq had removed the model llama-3.3-70b-versatile from its lineup. The request came back with 404 model_not_found. And the chain, as built, only moved to the next provider on rate limits. A 404 is not a rate limit. So it aborted. Mistral and Gemini were available and were never asked.

Today the lesson sits in the code, right above the chain:

// The chain moves on with EVERY error, not only with rate limits. [...]
// A fallback that only catches one type of error fails to protect at
// exactly the moment a provider actually goes down. Which error it is
// does not matter for the decision: if a provider does not deliver,
// the next one is up.

That is the general point behind the specific bug: resilience that only knows the expected failure is not resilience. You build it against the scenario you imagined, a rate limit, and the real thing arrives in a different shape.

Except: the answer in the code is the coarse one. It was right for that incident and it is too simple as a general rule, because not every error is a provider problem. If your own request is broken, invalid JSON, a text that is too large, an expired key, then it fails at provider B just the same. The chain then hides your own bug behind three attempts, and what ends up in the log is the last provider’s message rather than the cause.

The rule I would give a client therefore distinguishes by error class, not by individual case:

The third row is the one you forget while building, and the only one where a fallback does harm: it turns an immediately visible programming error into a vague „something with the AI is not working".

Read closely and you will notice that decision 4 already draws exactly this line: there, only „model not found" triggers a re-resolution, explicitly not a network error and not a limit. So the nuance was already present, just in one place. The chain itself still works coarsely, and in this system that is tolerable, because all three providers get the same request and a request error comes through in the end anyway. Tolerable is not exemplary.

The case shows something else in passing. Every takeover is now logged.

log.Printf("✅ %s took over after previous providers failed", p.Name())

Without that line a permanently broken primary provider goes unnoticed, because everything keeps running. Silent failover is a slow outage.

Mistake 2: „generate" contains „rate"

The second mistake is smaller, more embarrassing and more instructive.

Detecting a rate limit was originally a substring search for "rate" in the error message. That works until a message reads failed to generate content. It contains „generate". Every generation error was therefore treated as throttling and triggered the logic meant for it.

Bugs like this do not come from ignorance but from convenience at the wrong moment: a substring search takes thirty seconds to write, a proper error classification twenty minutes. The trade works until it does not.

The general rule I take from it: errors from third party systems belong classified, not searched. And where you do depend on text patterns, because the provider gives you no clean codes, then on complete markers rather than fragments. Today the code carries an explicit list, and the comment next to it says why it is narrow:

// Deliberately narrow: a network error or a limit must not trigger a
// re-resolution, otherwise the system swaps its model on every hiccup.

Decision 4: model names are consumables

The Groq incident led to a second change, and I consider it the most interesting architectural decision in the project.

Hardcoded model names are a time bomb, especially with free providers: models get introduced, renamed and retired quickly there. The obvious answer would be to move the name into an environment variable. That only moves the problem, someone still has to notice the outage and change the value.

Instead the system heals itself: on exactly the error „model does not exist", the provider queries its own model list and picks the first one suitable for the task.

// The API returns names as "models/gemini-2.5-flash".
ids = append(ids, strings.TrimPrefix(m.Name, "models/"))
...
picked := pickModel(ids, geminiModelPreferences)

Two details decide whether this is a good idea or a dangerous one.

First, the narrow trigger. Only „model not found" causes a re-resolution. A network error or a limit must not, otherwise the system swaps its model on every hiccup, and afterwards nobody knows which output was produced with what.

Second, the filter. The model list also contains models that cannot answer chat requests, embedding models for instance. Only what supports generateContent is considered. A self healing mechanism that picks the wrong model is worse than none: it replaces a loud failure with a quiet one.

And the configured name remains the preferred choice. Resolution only kicks in when it no longer exists. Self healing does not replace configuration, it catches its decay.

Decision 5: the prompt is a filter with a default

A language model makes a decision here that users feel directly: is this a freelance project or a permanent position? Both error directions are possible, but they are not equally bad.

Nobody notices a missed project. Everybody notices a permanent job in a freelance alert, and after the third one nobody unsubscribes. They simply stop reading the emails.

The prompt is therefore deliberately asymmetric. It lists not only what argues for a freelance project but just as explicitly what argues against it, and it sets a default:

If NO clear freelance signals are present → FALSE
If the text is ambiguous → FALSE (safety first)
"Remote" or "home office" alone is NOT a freelance signal → FALSE
When in doubt → FALSE

At its core this is what an allowlist is in classic software: reject when in doubt. Only here the rule is written in prose rather than code, which makes it no less program logic. It decides the behaviour of the system, it changes often, and it has to be revertible. That is why a prompt like this belongs in version control and not in a text field inside some tool.

A second, less visible point of the same kind: the HTML is stripped from the listings before they reach the model. Markup costs tokens, adds nothing to meaning and distracts the model if anything. Across thousands of listings per run that is not a detail.

Decision 6: make silent failures loud

The last decision has the least to do with AI and is worth the most in operation.

When the AI processing fails, nothing dramatic happens. The website runs, login works, the scraper keeps collecting projects. Only: unprocessed projects trigger no notifications. To a user that does not look like an outage. It looks like a quiet week on the project market.

That is the most dangerous kind of fault, the one nobody reports. So the system counts consecutive failures and raises an alarm when a threshold is reached, with exactly the details you need at three in the morning:

The AI processing has failed for %d consecutive runs.
Provider chain: %s
Unprocessed projects: %s
Last error: %v

While processing is down, no notifications go out for new projects.

Three things about it are deliberate:

The threshold. A single failure is normal operation, particularly with free quotas. Alert on every error and you train yourself to ignore alerts.

The cooldown. After a warning it stays quiet for a while, even if failures continue. Twenty identical emails help nobody.

The all clear. When it works again, a second message follows. Without it you never know whether a problem was fixed or the warnings just stopped.

And the last line of the warning is the most important one: it says not what is broken but what that means for the user. That is the measure of how urgently you get out of bed.

Mistake 3: no fixed evaluation set for far too long

The third mistake stayed with me longest because it never hurt. There was no fixed set of examples with known correct answers. I judged prompt changes on individual listings: look at one, try the new wording, „looks better", move on.

That works as long as one model does the job. Here three do. A prompt that classifies cleanly on Groq can come out more cautious or more generous on Gemini, because the models read the same instruction with different strictness. Check against five examples and you may well be checking the wrong provider, because which one answers is decided by the fallback chain.

On top of that comes the self healing from decision 4. It keeps the system alive when a model name disappears, and in doing so it swaps the model without anyone intervening. That is exactly when output quality may shift too. Without an evaluation set you notice none of it: the system runs, the metrics are green, and the classification drifts quietly. Self healing without measurement trades a visible outage for an invisible degradation.

The effort would have been laughable: fifty listings, hand labelled as freelance or permanent, an afternoon of work. After that every prompt change and every model swap is a number instead of an opinion, and you see immediately whether an improvement holds for all providers or only for one. That is what LLM evaluation means, and it is far less demanding than the term suggests.

It is the work you postpone most happily in a side project, and in client projects it separates „we think it got better" from „it got better".

How you know it works

Everything described so far is worthless if nobody can see whether the system does a good job. With AI that is harder than with classic software: a service that answers counts as technically healthy. Whether it answers sensibly is another matter.

The technical side is the easy part. The service exposes Prometheus metrics, a Grafana dashboard shows throughput and error rate. That answers: is processing running, how fast, how often does it fail.

It does not answer the business side. That needs other figures, and the most interesting one is free: the backlog. How many projects are waiting to be processed? That single number catches almost every fault without modelling each one: a provider down, an exhausted quota, a window that got too small, an inflow growing faster than processing. Which is why it is in the warning email too: not just „something is stuck" but „something is stuck, and this much has piled up".

The two views answer different questions and do not replace each other. The metrics say whether the system is working. The evaluation set from the previous section says whether it is working correctly. With only the first you see a healthy service reliably producing nonsense.

Where the approach hits its limits

So this does not read like a recommendation for every case: three things this design does not carry.

It is not for hard response times. A chain of three providers can make three failed attempts in the worst case before an answer arrives. For background processing that is irrelevant. Behind a user interface that should answer in under a second it would be unacceptable. There you need tight timeouts per attempt and a circuit breaker that skips a permanently broken provider entirely for a while.

It is not for data that must not leave the building. Free tiers usually come with different data processing terms than paid ones. For publicly advertised project descriptions that is uncritical. For customer data, contracts or personnel records the first question is not which model is good, but where it runs and what happens to the input.

And it replaces no commitment. A system running on free quotas has no availability guarantee, with none of the three providers. For a free tool that is honest and appropriate. Anyone promising availability to their customers has to pay for it, with at least one provider in the chain.

What of this applies to paid systems

The constraint was artificial, a side project that must not cost anything. The conclusions are not:

If I had to single out one, it would be the simplest: an AI system has a price per transaction, and you should know it before the volume grows. With free quotas you learn it immediately, because the quota is empty after an hour. In a company project you learn it after the first monthly statement, and by then the architecture is already built.

That is the real reason the side project paid off: it punished every decision immediately that in a paid project would only have hurt months later.