is it normal that 30 hand written eval questions can't tell me if a change helped eval
I wrote 30 questions with expected answers, I score them myself, and every change I make lands somewhere between 20 and 24 correct. Chunk size, reranker on or off, a different prompt: all within that range. Either…