AI is starting to reshape how impact teams collect data, generate insights, and communicate results. But most leaders are still figuring out where inside of these goals AI adds value versus where it introduces unnecessary risks or uncertainties.

During a September 29 virtual Leader Roundtable, practitioners from across the NationSwell community came together to explore how AI can strengthen measurement, reporting, and storytelling. Some of the takeaways from the discussion appear below:

Takeaways:

Keep humans at the wheel throughout the workflow. The most reliable AI-enabled impact work treats a person as the driver through every stage: shaping strategy, defining analysis, and signing off on each deliverable. Framing this as being human “through the loop” guards against the perfunctory review that erodes quality and preserves meaningful human influence and decision-making.

Train evaluation and scoring tools on your own criteria and institutional knowledge. Large language models (LLMs) carry the biases of their creators, so the safeguard is to ground them in your own corpus: past applications, previous year selections, and the rationale behind them. Narrowing to a small set of clear decision criteria sharpens the tool and gives reviewers a fast, consistent checklist to weigh AI recommendations against their own judgment.

Backtest new tools against past decisions and audit what they screen out. Before trusting an AI scoring tool, run prior cycles through it and compare its top picks against the choices you made. The hit rate builds confidence or reveals the gaps where the tool falls short. Reviewing what a tool rejects matters as much as reviewing what it surfaces, since early rounds are where hidden bias and erroneous exclusions tend to appear.

Break complex analysis into smaller pieces you can validate separately. Scoping work into narrow, well-defined components gives far more control over quality than asking a tool to deliver a finished product in a single pass. This stacking approach isolates bias or error at each step and is especially valuable for quantitative work, where one incorrect figure can distort the whole.

Mine longitudinal data for insights that were previously out of reach. Organizations sit on years of grantee reports and data that were once too resource-intensive to analyze at scale. This is exactly the material AI tools can now parse for cross-cutting patterns and multi-year trends. 

Consider lightweight automation versus AI for repetitive tasks. For deterministic, tedious work like moving data between spreadsheets, generated code and scripts can deliver full accuracy with no hallucination, often at lower cost and energy use than using an AI model.

Capitalize on efficiencies while building the capabilities that were never possible before. It is easy to anchor on AI as a way to do existing work faster, and time savings on repeatable tasks are worth tracking. But there is also opportunity in net-new capacity that were previously out of reach because of bandwidth or access.