Beyond the Survey
Using AI to evaluate and grow leaders' skills, boost team performance
By Abigail Covington
Photos by Kat Lawrence
Above: Research team members Scott Tonidandel, Wenwen Dou and George Banks evaluate how leaders' speech, particularly how something is said, can signal leadership behaviors.
George Banks, co-director of the Center for Leadership Science and chair of UNC Charlotte’s Department of Management in the Belk College of Business, didn’t start his career intending to reshape how we think about leadership development. Before entering academia, he worked as a consultant for private firms and federal agencies, where he helped develop robust leadership pipelines. Over time, he noticed that the tools used to evaluate leaders — surveys and follower feedback — often didn’t reveal what leaders were actually doing.
“Surveys don’t capture the behaviors that make someone effective; they capture perceptions,” Banks said. He saw firsthand how relying on subjective perceptions rather than objective data made it difficult to develop leaders.
Soon, Banks left the private sector to earn a Ph.D. in management before joining the faculty at UNC Charlotte, where he eventually met Wenwen Dou, an associate professor at the Charlotte Visualization Center in the College of Computing and Informatics. Dou shared Banks’ interest in artificial intelligence and is an expert in visual-text analytics as did long-time colleague Scott Tonidandel, management professor and director of the organizational science Ph.D. program. All are affiliated with the Center for Leadership Science.
Many leaders are thrust into leadership roles without any formal training."
Scott Tonidandel
“A survey may show that a follower perceives a leader as charismatic,” Tonidandel explained, “but it doesn’t say what specific actions make them appear that way.”
Banks, Dou and Tonidandel realized that if their research could identify the specific actions that characterize a leader as charismatic, ethical or transformational, they could potentially teach existing or emerging leaders to take identified actions and, hopefully, remove some of the guesswork and subjectivity from the leadership development process. Together, the colleagues set out to build a new methodology for evaluating leaders based on key behaviors and powered by artificial intelligence.
Building a behavior-based AI model
The first step in the team’s process was to develop or combine existing taxonomies of behaviors commonly associated with four leadership styles — ethical, transformational, charismatic and destructive.
“We began by asking what behaviors are associated with ethical leadership. What makes someone appear transformational or destructive?” Tonidandel said. Meanwhile, Dou got to work building the tool's backend in Python, a popular programming language used to develop machine learning algorithms.
The team came together to train the AI on massive amounts of textual data, including CEO letters to shareholders, speeches and meeting transcripts, to recognize certain behaviors like metaphors and rhetorical questions (charismatic leadership) as well as virtue signaling (ethical leadership) and statements that question critical assumptions (transformational leadership). To help the AI identify indicators of destructive leadership, Banks and Tonidandel along with their doctoral students, fed it email exchanges from the Enron scandal that are publicly available online.
“The nice thing about the Enron emails,” said Banks, “is that they are from the early 2000s and contain lots of inappropriate language that people now would not put in writing.”
While the AI can easily recognize some behaviors, others, such as cursing, have proven more difficult. The issue is context. Training the AI on enough data so that it understands how a curse word is being used and if it is destructive or not has been one of the most difficult challenges the team has encountered yet. But Dou, who is in charge of validating the AI’s outputs, says the system gets better, smarter and faster every day.
AI in Zoom meetings
Divya Doshi uses virtual reality goggles to observe and describe to fellow Ph.D. students characteristics of leadership seen in virtual interactions. Doshi and project lead George Banks also engage Business Honors Program undergraduates in similar studies using the goggles.
Along with co-PI Eric Heggestad, the researchers are testing their AI in virtual settings, funded by a grant from the U.S. Army Research Institute for Behavioral and Social Sciences. Divya Doshi, an organizational science Ph.D. student working specifically on the grant project, which will last three years, is excited to explore what leadership looks like in virtual settings. She is especially interested in seeing if people use certain behaviors on Zoom versus in face-to-face communication. She and fellow Ph.D. students, Milad Rogha and Matt Gerson, are creating a scoring framework that the researchers will use to assess the leadership skills on display in dozens of Zoom meetings they plan to analyze.
To gear up for this in-depth analysis, the team enhanced its AI tool so that it can collect and analyze data from multiple sources, including audio files and video recordings in addition to text. It can analyze not just what leaders say, but how they say it. In an era where virtual workspaces are becoming the norm, this capability is crucial.
Surveys don’t capture the behaviors that make someone effective; they capture perceptions."
George Banks
Perhaps the most innovative feature of the platform the team is developing is its ability to provide real-time feedback to leaders. During or after a meeting, the AI-powered platform will generate a detailed behavioral report, showing leaders how they perform in the four key areas during different times in the meeting. It could, for example, reveal which subjects a leader is better at communicating about charismatically and which topics make them defensive or destructive. The platform could one day even allow leaders to compare their behavior over time or against renowned public figures like Oprah Winfrey.
Democratizing and diversifying leadership training
When it comes to the impact of their research, its potential to democratize leadership training is at the top of everyone’s mind. As Banks noted, traditional leadership development programs, such as those offered to top executives at large corporations, remain out of reach for many leaders. Tonidandel noted that many leaders are thrust into leadership roles without any formal training.
The team’s hope is that the AI-powered tool will offer an affordable and scalable way for leaders everywhere to improve their skills. Dou thinks it could be particularly useful in nonprofit settings where an effective, charismatic, but ethical, leader can help an organization attract crucial funding.
Team lead George Banks, center, strategizes with Ph.D. students Matt Gerson, left, and Divya Doshi on behaviors they have pinpointed through analysis of text and other cues from virtual settings.
On a practical level, Banks hopes that the platform will solve the survey problem that in part inspired the team’s research, and he is optimistic that their tool will help reduce bias and subjectivity in leadership assessments, an important step in the larger effort to achieve equality in the workplace.
Abigail Covington is a freelance writer based in Brooklyn, New York.