You interviewed four people for the same role. Three of them were sharp in the room. You picked the best one, they started six weeks ago, and right now you are quietly redoing half their work at 9pm while wondering how you read it so badly.
What’s in This Article
You did not read it badly. You used a bad instrument. Most Aussie Shopify founders hire off a CV, two conversations and a gut feel, then act surprised when the result is random. It is random. The largest re-analysis of selection research we have, Sackett and colleagues in 2022, put the predictive validity of an unstructured interview at .19. That is barely better than picking the person with the firmest handshake.
The cost of getting it wrong is not abstract. Australian recruitment data puts a bad hire at 30% to 150% of that person’s annual salary once you count lost output, your time, rehiring, and the damage done while nobody was watching. On a 75k ops coordinator that is somewhere between 22,500 and 112,500 dollars. You would never spend that on a Meta test without a hypothesis. A gut-feel hire is exactly that.
There is a better instrument, and it is not a better interview. It is a short, paid, real piece of work. Automattic, Canva and most serious remote-first companies already run one. Almost no Aussie DTC brand under 10 million does. Here is how to build a trial that actually predicts performance, and how to run it without tripping over Fair Work.
Why the Interview Is the Weakest Tool in Your Hiring Kit
An interview measures one thing reliably: how good someone is at interviews. That is a real skill. It is just not the skill you are paying for, unless you are hiring a presenter.
The numbers are worth sitting with. In that 2022 re-analysis, structured interviews landed at .42, cognitive ability tests at .31, work sample tests at .33, and unstructured interviews at .19. Two things jump out. Structure matters more than format. And anything that touches real work beats the chat you are currently having.
Bias tells the same story. Research on interview structure shows the bias effect shrinking from d=.59 in unstructured formats to d=.23 in structured ones. When you have no written criteria, you are not evaluating anyone. You are pattern matching against people you already like.
The companies that hire well figured this out years ago. Automattic runs a paid trial for every single role, no exceptions, at a flat 25 USD per hour, typically 15 to 35 hours spread across a few weeks. Candidates keep their current job while they do it. The trial is not a formality bolted onto the end of the process. It is the decision.
Closer to home, Canva runs a craft challenge as a core stage out of Sydney, with an average process length around 29 days. You do not need 29 days and you do not need Canva’s budget. You need one well-designed task and a scorecard you wrote before you saw the work.
The Fair Work Line Most Founders Get Wrong
This is where founders either freeze up or blunder in, so let us be precise about it.
The Fair Work Ombudsman accepts unpaid trials only in very narrow circumstances. The trial must exist for the sole purpose of demonstrating the key skills for the role. It must be actively supervised the entire time. And it can last only as long as is genuinely needed to see those skills, which in practice means minutes, and rarely more than an hour or two. Anything beyond that must be paid at the appropriate minimum rate.
Here is the simple test I give members. If the output is useful to your business, you pay for it. A candidate spending four hours building a real Klaviyo flow you intend to ship is not demonstrating a skill. They are working. Pay them.
Paying also kills the objection that costs you the best candidates. Strong people already have jobs. They will not donate a weekend to your recruitment process. They will happily take a paid, scoped, interesting project, because it lets them audition you as much as you audition them.
- Under an hour, supervised, no usable output. That can sit inside the unpaid trial exemption. Think a live screen share where they walk you through setting up a Shopify collection with metafield conditions.
- Anything longer, or anything you keep. Pay for it. Engage it as a one-off contractor project or a short casual shift, depending on how you normally engage people.
- Make the paperwork boring. A one-page trial agreement covering scope, hours, rate, IP, and a line stating the trial does not guarantee ongoing work. Have your accountant or an employment adviser sight your standard version once, then reuse it forever.
Budget-wise this is close to nothing. Six hours at 45 dollars an hour is 270 dollars. Run three finalists and you have spent 810 dollars to de-risk a decision that could cost you 40,000. It is the cheapest insurance you will buy this year.
The Five Rules of a Trial Task That Actually Predicts
Most trial tasks fail because they are either a puzzle nobody does in real life, or a vague brief that tells you nothing when it comes back. A trial only predicts if it is built to these five rules.
- 1. It has to be real work. Pull it from your own backlog. Not a hypothetical. If you would not have paid someone to do this task anyway, it is theatre.
- 2. It has to fit in four to eight hours. Long enough to see judgement, short enough that a good candidate with a full-time job can say yes. Automattic runs longer because they can afford a two-month runway. You cannot.
- 3. It needs a written brief and a deadline. Verbal briefs test memory, not capability. Written briefs also expose the candidate who never asks a clarifying question.
- 4. It must be gradeable against criteria you wrote first. Write the scorecard before you send the brief. If you write it after, you will write it to match whoever you already liked.
- 5. It must not need access that can hurt you. No live theme edits, no customer data exports, no payment settings. More on scoping access in a minute.
Here is what that looks like for the four roles Aussie DTC brands hire most often.
- Ecommerce or ops coordinator. Give them a messy supplier spreadsheet of 60 SKUs and your product taxonomy. Ask for a clean Shopify-ready CSV plus a written list of every assumption they made and every question they would ask you.
- Email and lifecycle. Hand over your last 90 days of Klaviyo flow performance and ask for a one-page teardown: the three biggest gaps, the sequence they would build first, and the subject lines for it.
- Media buyer. Share read-only Meta Ads Manager access and one live campaign. Ask what they would kill, what they would scale, and the exact reasoning with the numbers that support it.
- Customer support. Send eight real tickets with the customer details stripped out, including two angry ones and one where the customer is wrong. Ask for their replies plus a note on which one they would escalate to you and why.
Notice the pattern. Every one of these asks for the work and the thinking behind it. The work tells you if they can do the job. The reasoning tells you whether you will need to check everything they produce for the next two years.
The Trial Brief Template You Can Copy Today
Send this as a single document or email. Do not spread it across a call and three Slack messages. The brief is part of the test.
- The role. One line on what this person owns once hired.
- The task. Two or three sentences. Concrete deliverable, no interpretation required about what you want handed back.
- The context. Links, logins, files, and the background a current team member would already have. Deliberately leave one gap. You want to see whether they ask.
- The time budget. “Please cap this at six hours. If it is taking longer, stop and tell me where you got stuck.” How they handle that instruction is data.
- The deadline. A specific day and time, at least five days out so they can fit it around their current job.
- The pay. Rate, total, when it lands, and that it is paid whether or not they get the role.
- What good looks like. Three bullets on the outcome you are after. Not the method. If you prescribe the method, you learn nothing about their judgement.
- The questions channel. “Email me anything at any point.” Then track what they ask and when.
One more line worth adding at the bottom: “I will send one round of feedback and ask for a revision.” That single sentence turns a one-shot test into the most useful part of the whole process, and we will come back to why.
Give Them Access Without Giving Them the Keys
The most common reason founders skip trials is fear of handing store access to a stranger. Fair enough. Shopify’s permission model solves this in about four minutes, and almost nobody uses it properly.
First, know your limits. Staff account allowances run from a small handful on the standard Shopify plan through 15 on Advanced and unlimited on Plus. Collaborator accounts do not count toward that staff limit, which matters if you are near your cap.
Setting up trial access, step by step:
- Go to Settings, then Users and permissions in your Shopify admin.
- Click Add user and enter the candidate’s email. Use their personal email. Do not create a company address yet.
- Untick everything, then tick only what the specific task needs. For a product data task that is Products view and edit, nothing else. For a support task it is Orders view only.
- Leave Settings, Finances, Apps, Themes and Customers export switched off. There is no trial task that legitimately needs any of them.
- Confirm two-step authentication is enforced on your store so the invite cannot be accepted from a compromised inbox.
- Put a calendar reminder for the day after the deadline to remove the user. Do it even if you are about to hire them. You reissue proper access on day one with a proper email address.
For anything that touches the theme or the front end, do not give live store access at all. Duplicate your theme, or spin up a development store, and let them work there. Same for anything involving customer records: export a sample, strip names and emails, and hand over a spreadsheet instead of a login.
This scoping habit is worth building regardless of hiring. If you have never audited who currently holds access to your store, that is a separate job worth doing this week. Most stores we look at have at least one ex-freelancer still sitting in the users list with full permissions.
The Scorecard: Grade the Work Before You Feel Anything
A trial without a scorecard is just a longer gut feel. The whole point of the structure research is that the scoring, not the format, does the heavy lifting.
Use five criteria, each scored 1 to 5, written down before the briefs go out. These five work for almost every role in a DTC business.
- Output quality. Is the deliverable something you could actually use? A 5 means you would ship it as-is. A 1 means you would redo it.
- Brief adherence. Did they do what was asked, in the format asked, by the time asked? Talented people who ignore briefs are expensive.
- Communication. Did they ask about the gap you deliberately left? Did they flag a problem early, or go quiet and hand in something broken?
- Estimate accuracy. They were given six hours. Did they use roughly six? Someone who burns 14 hours on a six-hour task will burn your payroll the same way.
- Response to feedback. You sent one round of notes. Did the revision get better, get defensive, or get literal?
Set your thresholds before you score. My default for members: 20 or above out of 25 is a hire, 17 to 19 is a maybe that needs a second trial or a reference check, and anything under 17 is a no regardless of how much you like them. Write those numbers down first, because the moment you see a name attached to a score, the numbers start moving.
If you have a second person in the business, get them to score independently before you compare. Two independent scores that agree is a strong signal. Two that disagree wildly usually means the brief was ambiguous, which is your fault, not the candidate’s.
What the Trial Tells You That No Interview Ever Will
The deliverable is the least interesting output of a trial. Everything around it is where the signal lives.
When they ask questions. Great operators ask two or three sharp questions in the first day, then go quiet and deliver. Weak ones ask nothing, then explain at the deadline why the missing information made the task impossible. Same information gap, completely different behaviour, and you have just watched a preview of every project you will ever run with them.
How they handle the time cap. The instruction was six hours, stop and flag it if you are over. The candidate who emails at hour five to say “I am at five hours, here is what is done and what is not, want me to keep going” has just shown you they will never surprise you with a blown budget.
The second draft. This is the single most predictive moment in the whole process, and it is why the feedback round is non-negotiable. Send genuine, specific notes. Three responses are possible. They improve the work and go past what you asked, which is your hire. They change exactly what you flagged and nothing else, which is a solid executor who will need direction forever. Or they explain why your feedback misses the point, which tells you what the next two years of management look like.
How they document. Did anything come back with a note on what they did and why? People who document unprompted are the people who will build your standard operating procedures instead of hoarding the knowledge in their head. That habit is worth more than two years of experience.
Whether they want it. A trial is a two-way test. Candidates who go through six paid hours inside your business and still want the job have made a genuinely informed decision. That is a real predictor of whether they stay past year one, and Australian data shows why it matters: employees with two years or less of tenure are 38% more likely to be looking for their next job.
The Maths That Makes This Obvious
Founders resist trials because they feel slow and awkward. Put actual numbers against them and the resistance disappears.
Australian benchmark data from ELMO puts the average cost to fill a vacant position at around 10,500 dollars when you combine job board spend with internal time. That is the cost of filling the role once, correctly. Fill it twice because the first one did not work, and you have paid it twice, plus the 30% to 150% of salary the failed hire burned on the way through.
Now price the trial. Three finalists, six hours each, 45 dollars an hour. That is 810 dollars and about three hours of your own time writing the brief and scoring the work. Against a downside of 22,500 dollars on the low end for a 75k role, you are spending roughly 3.6% of the risk to substantially reduce it.
There is a second-order benefit that is harder to see and worth more. Small Australian organisations of 2 to 19 people run about 11% annual turnover, against 15% for medium businesses and 21% for large ones. Small teams have a natural retention advantage, and every mis-hire throws it away. In a five person business, one wrong seat is 20% of your capacity pointed in the wrong direction.
Wiring It Into a System You Run Every Time
Individually, each piece here is a small improvement. Run together on every role, they compound into something that changes the shape of your business.
The written brief forces you to define the role properly, which is the same clarity you need for the job ad and the first 90 days. The scorecard becomes the performance review template, because you are now measuring the same five things at the six-week mark that you measured before they started. The trial task itself becomes a reusable asset. Build four of them, one per function, and hiring stops being a project you dread and becomes a process you run.
It also changes what you decide to hire for at all. When you have to write a six-hour task that represents the role, roles that are actually a pile of unrelated admin become obvious. That is usually the moment a founder realises the job should be automated or bundled instead of filled, which is the same call we walk through in the AI or hire test. And if you have not built the role sequencing yet, the first hire playbook covers which seat to fill before this one.
Your Paid Trial Checklist
Print this, or drop it into your hiring doc. Every box gets ticked before an offer goes out.
- Before the brief. Task pulled from your real backlog. Scoped to four to eight hours. Scorecard with five criteria and thresholds written and saved.
- The brief itself. One document. Role, task, context, time budget, deadline, pay, what good looks like, questions channel. One information gap left deliberately. Feedback round flagged upfront.
- Access. Minimum permissions only. No settings, finances, apps, themes or customer exports. Theme work on a duplicate or dev store. Removal reminder in the calendar.
- Compliance. Paid at a fair rate. One-page trial agreement covering scope, hours, rate, IP, and no guarantee of ongoing work. Anything unpaid kept under an hour and supervised.
- During. Log when they asked questions and what they asked. Note whether they flagged the time cap.
- After. One round of specific feedback, then a revision. Score independently before comparing. Hire at 20 plus, second look at 17 to 19, no below 17.
- Always. Pay every candidate, on time, whether or not they get the role. Give every one of them the feedback you wrote. Your reputation as an employer in a market this small is a real asset.
The founders who build teams that hold together are not better judges of character than you. They just stopped relying on judgement of character. They watched people work, wrote down what they saw, and made the call on evidence.
Six hours and a few hundred dollars. That is what it costs to stop guessing.
Where to Start This Week
Do not wait for your next vacancy. Open your backlog, find the job that has been sitting there for three weeks because nobody owns it, and write the six-hour brief for it now. You will have a reusable trial task and a much clearer view of what that role actually is.
Inside eCommerce Circle, hiring is one of the core pillars we work on with every member, and the trial brief is usually the first thing we build together. If you want a second opinion on yours, let’s talk.
