Agents are hard to judge from a single answer. They take many steps, use tools and change things along the way, so a good test has to look at what they actually got done.
Start small. Twenty to fifty tasks drawn from real failures will tell you more than a thousand made-up ones. Write down what success looks like for each, so two people would grade it the same way.
Then grade every run. Code can check the outcomes it can verify, a model can judge tone and reasoning, and people should spot-check both.
Finally, rerun the whole suite after every change. A regression then shows up the day it lands, not weeks later in a customer’s inbox.