How Fannie Mae Developed Its Own Way to Assess Copilot’s ValueHow Fannie Mae Developed Its Own Way to Assess Copilot’s Value
Determining the value and benefits from their Microsoft Copilot deployment was no box of chocolates.
Over the last year, Fannie Mae rolled out Microsoft Copilot to 500 of its knowledge workers. During their session at Enterprise Connect, the project’s architects provided a glimpse into the process, what they learned, and how they’re using that initial project phase to inform the ongoing deployment in the company.
Fannie Mae started with an early adopter group of 50, but today, with 500 licenses deployed and 500 remaining – and a subset of existing users who barely use the licenses they were granted – Fannie Mae’s at a crossroads: they need to define the value of Copilot to the users and figure out who gets it next. Microsoft Copilot comes with a cost – $30 per user per month – so determining its business value is critical. “We're not trying to solve just the Copilot problem at our company. We're trying to solve the AI problem at our company,” said James Redmore, Technology Director with Fannie Mae. “As we run into these challenges, we're developing new tools and new ways to handle that problem.”
Throughout its phased deployment of Copilot to increasingly larger cohorts of different types of users – which they called ‘personas’ – Fannie Mae found that those individuals benefited from some core functionality such as drafting email, rewriting sentences, and jumpstarting document creation from an idea typed into the Copilot prompt window. “Users told us that Copilot made it easier to get to the first draft, which they could then edit and complete the task more quickly. That was a big value add,” Redmore said. “The other huge value add was content summarization, especially for the executives, researchers, etc., who have to parse a lot of information daily.”
Redmore said that user training was critical to successful use of Copilot. Fannie Mae created multiple workshops in which they would train the people in how their role (persona) might best use, and benefit from, Copilot. They sent weekly ‘tech tips,’ established specific office hours for support, and a created ‘center of excellence’ community where people could ask questions and learn from each other. Another important step was creating a library of the ‘best-known prompts’ with proven outcomes for obtaining the best results from Copilot.
One group of early users in Fannie Mae’s rollout was the single-family customer engagement team. Redmore said this group really ‘leaned into’ learning how Copilot could help them in their work. For example, this team engages in monthly calls with their many customers. They took the most recent historical interactions with each of those customers, fed them into Copilot and, as a team, developed “large, 600-word prompts” to obtain a recommendation of what they should focus on for the next customer engagement.
“That’s not generating text; it's not summarization. It's assimilating a lot of information and then producing output. That was a very sophisticated use, and I was really surprised by the quality of the prompt,” Redmore said.
Despite these and other success stories, Redmore’s team found that not all personas who had licenses were using Copilot. And, usage within personas also varied – some used it a lot while others hardly touched it. So, Fannie Mae conducted another workshop where they asked the pilot participants to self-identify the tasks they used Copilot for, how many times a month they completed those tasks and how much time they were saving with Copilot.
“That way we could do some math and figure out how much value Copilot was providing,” Redmore said. “But because it was a relatively unstructured way to collect feedback from the users, we had a wide diversity of responses. Some people had very little time savings while others had extraordinary time savings. The lesson we learned is to have a more structured feedback mechanism.”
That mechanism didn’t come from any of the tools Microsoft provided. It arose from the single-family analytics team within Fannie Mae, a group of “data nerds” who worked with Redmore’s group to develop an analysis framework which will now be applied to almost all AIs that Fannie Mae considers deploying.
That framework consists of 10 questions related to Fannie Mae’s business delivered in a two-hour long workshop. Different personas take the test, and they are allowed to research the answers. However, only half of the questions can be answered with AI assistance while the other half must be answered without it. How long it takes to complete the questionnaire is recorded.
“We then used Claude Sonnet 3.5 to give us a 1 through 10 score on the accuracy, quality, and time it took to answer those questions with and without AI assistance,” Redmore said. “So now we have this framework where we can test how any AI product suits us and how it empowers our employees to perform better. It also gives us a baseline against which we can assess the product today, and over time, as the product matures – using the same framework.”
The next step is to implement a usage-based claw back mechanism so that Fannie Mae can reclaim un- or under-used Copilot licenses – because it isn’t free and they’d rather provide licenses to those who will benefit from them the most. Being able to assess how the AI product suits different user personae and to assign licenses based on proven user engagement are two administrative tasks Fannie Mae found important for managing AI – not just as a technological deployment but as an operational expense.
About the Author
You May Also Like
