Beyond Coding: Using AI to Enrich Qualitative Analysis of Faculty PD Needs

One of the persistent challenges in faculty development work is making sense of qualitative data from needs assessments. When I recently distributed a professional development survey to faculty across departments, I received nearly 200 responses with rich, nuanced open-ended comments. The quantity was overwhelming, but the potential insights were too valuable to reduce to mere word counts or simplistic themes.

This scenario presents a familiar tension in qualitative analysis: how to honor the complexity and nuance of faculty voices while still identifying actionable patterns? Traditional coding approaches, while methodologically sound, are extraordinarily time-intensive. Meanwhile, basic word-frequency analyses miss the contextual richness that makes qualitative data so valuable in the first place.

With these constraints in mind, I decided to experiment with using Claude, a large language model, as a collaborative analysis partner rather than simply a data processing tool. The results challenged my thinking about qualitative analysis in surprising and productive ways.

The Collaborative Analysis Process

Rather than approaching Claude as a replacement for human analysis, I used it as an initial sense-making tool and thought partner. The process unfolded through several iterative phases:

1. Data preparation: I anonymized all responses, removing identifiable information while preserving contextual elements like department type and faculty rank that might inform interpretation.

2. Initial pattern recognition: I asked Claude to identify recurring themes, tensions, and outlier perspectives across the responses without imposing predefined categories.

3. Dialogic refinement: Instead of simply accepting Claude's initial analysis, I engaged in a back-and-forth process, questioning categorizations, asking for evidence, and pushing for more nuanced interpretations.

4. Counter-narrative exploration: Critically, I asked Claude to identify perspectives that might be missing or underrepresented in the data, and to highlight tensions between dominant narratives.

5. Synthesis and verification: Finally, I returned to the raw data myself to verify key themes and ensure the analysis reflected the actual responses rather than algorithmic artifacts.

The goal throughout was not to automate analysis but to enrich it—to use the model's pattern recognition capabilities to identify connections I might otherwise miss, while maintaining my responsibility for interpretive decisions.

What Emerged: Beyond Binary Needs

What surprised me most was how this approach revealed complex tensions in faculty needs that binary survey questions would have obscured.

For instance, a traditional coding approach might have simply tallied mentions of "time constraints" as a barrier to implementing new teaching approaches. But the collaborative analysis revealed a more nuanced reality: many faculty were actually expressing something more complex than mere time scarcity.

They described a tension between their intrinsic desire to innovate pedagogically and institutional reward structures that didn't recognize that work. As one faculty member put it, "It's not really about not having hours in the day—it's about what those hours are valued for."

This distinction has significant implications for how we design professional development offerings. Rather than simply making workshops more "efficient" (the solution to a time scarcity problem), we need to address the deeper tension between faculty values and institutional incentives.

Other rich patterns emerged as well:

• Faculty across ranks expressed a desire for more engagement with learning science research, but wanted it translated into disciplinary contexts rather than presented generically
• Early-career faculty sought technical skill development, while mid-career faculty more often requested community and reflective practice opportunities
• Nearly a third of respondents described a tension between student expectations and evidence-based teaching approaches

Each of these insights suggests specific directions for our professional development programming that wouldn't have been as clear from a more traditional analysis.

Methodological Reflections: Trust and Verification

As with my classroom ungrading practices, I found that approaching this analysis through a lens of "trust, not compulsion" yielded the richest results. I trusted the AI to help identify patterns but didn't compel it to fit responses into predetermined categories. Simultaneously, I maintained a critical stance, consistently verifying suggested patterns against the raw data.

There were limitations, of course. The model occasionally overweighted particularly eloquent responses and sometimes missed subtle disciplinary differences in how faculty expressed similar needs. These limitations reinforced the importance of the human analyst in the process—the partnership worked precisely because it combined different analytical strengths.

Implications for Faculty Development Practice

This approach to needs assessment analysis has immediate implications for how we design professional development offerings:

1. Moving beyond binary needs: Instead of designing workshops based on surface-level needs, we can address the deeper tensions faculty experience between competing values and constraints.

2. Differentiating by career stage: The analysis revealed distinct developmental trajectories that suggest tailored programming for different faculty career stages.

3. Creating disciplinary translations: Faculty consistently expressed desire for learning science principles translated into their specific disciplinary contexts.

4. Addressing institutional tensions: Many faculty needs can only be addressed through institutional policy changes, not just better workshops.

Perhaps most importantly, this approach models a commitment to truly hearing faculty voices rather than simply tallying their "votes" for predefined options. It's an approach to professional development needs assessment that aligns with the same principles I advocate for student assessment—prioritizing rich understanding over reductive measurement.

Questions for Further Exploration

As I continue to refine this approach to qualitative analysis, several questions remain:

• How might we combine this approach with more traditional methods to maximize both depth and methodological rigor?
• What are the ethical considerations of using AI tools in analyzing faculty responses about their professional needs?
• How can we most effectively share these complex findings with institutional decision-makers accustomed to more simplified data presentations?
• What does this approach suggest about how we might analyze student feedback in our courses?

I'm particularly interested in how others are experimenting with similar approaches to qualitative analysis in educational contexts. Are you finding ways to leverage AI tools while maintaining intellectual integrity and methodological soundness? What tensions or opportunities have you encountered?

I'd love to hear your experiences as we collectively reimagine how we understand and respond to faculty professional development needs.

AI Use Acknowledgment: I acknowledge that I have used Generative AI tools in the preparation of this blog post, specifically Claude. The AI tool was utilized in analyzing qualitative response data from faculty surveys and helping to structure this narrative about the process. I have verified the accuracy of all representations of the analysis process and findings through comparison with the original survey data. This analysis represents my own critical thinking and evaluation, supported by AI-assisted data processing and narrative development.