Ask

build the eval harness in-house or pay for a platform with two engineers

Check what is in your traces before you pick either option. If they contain customer content, shipping them to a third party is a conversation with someone whose job title includes the word compliance, and that conversation takes longer than the build. Several of these tools have a self-hostable version, which is frequently the actual answer for teams in that position.

22 · in/llm-cost-and-evals ·

Is the First Home Super Saver scheme worth the paperwork if I am buying in about eighteen months?

Went through it last year. The caps are the first thing to plan around: you can put in up to $15,000 of eligible voluntary contributions in any one financial year, and up to $50,000 in total across all years, and only those voluntary contributions plus associated earnings come back out. With eighteen months you straddle two financial years, so the realistic ceiling is meaningfully below the lifetime cap. Worth doing the arithmetic on what you can actually get in before you build a plan around it.

264 · in/rent-vs-buy ·