The EcoRatings ESG Copilot

At EcoRatings the job was to turn a company's sustainability disclosure into a defensible ESG score. "Defensible" is the operative word: a regulator or a client can challenge any rating, so every number has to trace back to evidence. That constraint — auditability over raw accuracy — shaped the entire copilot.

Problem

The manual process took a senior analyst most of a day per company: read a 100-page report, extract claims, match them to rubric criteria, score each, and write a rationale. It did not scale, and consistency drifted between analysts.

An early single-prompt automation was worse than useless. It produced a score with a paragraph of reasoning, but:

• The reasoning and the score frequently disagreed with each other.
• There was no way to see which criterion a bad score came from.
• A reviewer could only accept or reject the whole thing — no place to intervene.

The requirement was a system where every score is attributable to specific evidence and specific rubric logic, and where humans own the borderline calls.

Approach

I decomposed the workflow into specialized agents, each with one job and a typed contract, orchestrated as a LangGraph state machine. Decomposition is what makes the system debuggable: when a score is wrong, you can point at the node that produced it.

The pipeline: classify the disclosure type, retrieve evidence per rubric criterion, assess each criterion with a tool-using agent, validate the structured output against a schema, and route uncertain results to a human review checkpoint.

Architecture

from langgraph.graph import StateGraph, END
 
def build_copilot() -> StateGraph:
    g = StateGraph(ESGState)
    g.add_node("classify", classify_disclosure)
    g.add_node("retrieve", retrieve_evidence)      # hybrid RAG per criterion
    g.add_node("assess", score_criteria)           # tool-using agent
    g.add_node("validate", validate_and_repair)    # schema guardrail
    g.add_node("review", human_checkpoint)         # human-in-the-loop
    g.set_entry_point("classify")
    g.add_conditional_edges("validate", needs_review, {
        "review": "review",
        "done": END,
    })
    return g

Three ideas did the heavy lifting.

Tools over freeform reasoning. The assessment agent does not "think about" numbers. It calls lookup_rubric_criterion, fetch_prior_year_metric, and compute_delta. Turning reasoning into checkable tool calls is what makes an answer auditable.

A validation loop with a hard budget. The validate node runs the output through a Pydantic schema and, on failure, feeds the errors back for exactly one repair attempt before escalating. An agent that retries forever is an outage with a progress bar.

Confidence-gated human review. The copilot aggregates per-criterion scores into an overall rating and an uncertainty estimate. I model the aggregate as a weighted mean of criterion scores $s_i$ with rubric weights $w_i$:

Anything with dispersion $\sigma(S)$ above threshold $\tau$ — the borderline and internally-inconsistent cases — routes to a human. High-agreement cases pass straight through. Human review is a designed edge, not a fallback.

The shared state object is the audit trail: for any rating you can replay exactly what evidence each agent saw, which tools it called, and where a human intervened.

Outcomes

After rollout across the ratings team, measured over a quarter of production runs:

The headline is the 62% reduction in analyst time, but the number that mattered internally was 98% schema-valid, fully-traceable outputs. When a client challenged a rating, we could open the trace and show the evidence, the rubric criterion, and the reviewer who signed off.

The lesson generalizes well beyond ESG: in high-stakes domains, accuracy is table stakes and operability is the product. A system you can inspect, correct, and explain will beat a marginally more accurate black box every time a human has to stake their name on the output.