AI-Generated Status Reports

Sending regular project status report to clients and stakeholders is a pain point for most project managers, and so became a natural opportunity space for the early AI rollout at KantataKantata is B2B SaaS company in the professional services automation space..

Read the complete story about AI at Kantata here:

Kantata AI Case Study

Project History

Like many companies in early 2024, Kantata has built an AI chatbot, intended to run across our entire platform. We had enormous amounts of project data for the model to work with, consequently outputs were very promising. That said, questions remained:

Initial Questions

• Was a chatbot the right interface for our large, complex application?
• Would people trust its outputs?
• The market was hungry for AI solutions, and our sales team was able to capitalize on this to get meetings with potential clients, but would it actually help close deals?
• Running our chatbot was costly. Would customers be willing to pay a premium for it?

To help answer these questions, I started designing a series of AI assistants that would work in different areas of the site. These needed to be small (so they could be built and released quickly) and specific (so outputs could be thoroughly tested).

Project Status Report AI Assistant

The first of these AI assistants was a project status report generator. The AI powering our chatbot had proven particularly adept at project status reports, and the user need was specific enough that an assistant seemed a better interface.

MVP

My MVP for the project status report generator lived up to its name: minimal enough to be built quickly, but viabile enough to generate valuable user data.

Our assumption early on was that few project managers across our diverse customer base would agree on what information to include in a status report, so while the MVP was in development I began prototyping ways we might allow customization.

Iteration

Several versions followed, each pursuing a different approach to customization and user control.

Final Version

This final version combined a few of the interface options I had discovered in my iterations, including a partially editable prompt and interactive elements to control what aspects of the project would be published in the report. This version also included a way we might reduce token costs.

It would cost Kantata $0.35 every time a customer clicked “generate”. By providing all of the configuration options as a first step, we hoped to reduce the number of generations customers would require.

An Agentic Approach

Moving from a simple prompt to an agentic system would be a significant step forward. This approach allowed for more dynamic and responsive status reports, where each agent could make decisions based on real-time project data.

Diagram of the Project Status Report AI Assistant as an agentic system:

The planning agent schedules each status report, based on input from reporting agents that monitor activity on the project internally (task tracker, activity feed) and externally (Slack, Jira, email, etc.); and might also act as a "critic" for the writer agent through a reflection framework.

A large language model powers the entire system, which also includes stateful memory and customizable guardrails. The memory is available to all of the agents at all times, while the guardrails prevent the writing agent from publishing content deemed inappropriate, such as specific team member names, team sentiment toward the client or project, etc.

Do we need an agentic system?

An important consideration before we got too far down the road is whether or not an agentic system is necessary at all. The first step in answering this is do outline the capabilities of an agentic system, then determine if these capabilities are a good fit for the user need.

I started with the scheduling function. The diagram to the right illustrates how the planning agent might make the decision to send a status report. Time elapsed since the previous report counts "down" (in the diagram), activity on the project counts "up". When the yellow bars "meet", a report is sent. The client benefits from more frequent reports when project activity is high, while not being bothered by regular recurring reports when there is little happening.

Interesting, but if research revealed that stakeholders preferred their status reports delivered on a consistent basis (e.g., every Friday) regardless of the activity on the project, an agentic solution may be overkill.

Would People Trust It?

Trust was already a concern for our AI assistant. In an agentic system that works autonomously, trust becomes even more critical. The system must be reliable, aligned with the user's expectations and values, and transparent enough to make these obvious.

This solution makes an editable version of the status report available to the project manager before publishing. Section-level confidence indicators provide insight into data sources, calling attention to hallucinations or other liberties the agent may be taking with the content. This balances automation with human oversight, ensuring that client communication remains aligned with the PM's voice and judgement.