Get our docs in a row, part 1: Internal needs assessment

One of the interesting things about being a knowledge base software company is that we advise people on knowledge management best practices all the time, but we aren't always that great at following our own advice. There's definitely a bit of the old "Do as I say, not as I do" when it comes to our own documentation.

Not in our beloved Support KB, thankfully, but our internal knowledge base has suffered some growing pains the last few years. It is a good source of knowledge, but it suffers from a lot of the weaknesses we talk to people about every day:

• Some sections are nicely built out; others are missing entirely.
• Some information is accurate and up-to-date; other articles or whole categories haven't been updated in a long time.
• Since we never had a well-defined information architecture, we often struggle to find things by browsing and rely a lot on search (which is great if you search the "right" words but not that great if you don't—a struggle for new hires!).

In short, we've known for a while that our internal knowledge base, called Silly Moose, was due for a massive overhaul. We've taken this on as a company goal (or "feather", as we call them here!). And since we're going through this pain ourselves, we figured we'd write some blog posts about it a) to help you know that you're not alone with your knowledge management struggles, and b) to provide examples of how you can move forward (or not, as the case may be).

Here, in our first blog post of the series, I share how we decided to kick off the project.

What problem are we trying to solve?

The topic has come up repeatedly in phrases like: "We need to do an audit," "I wasn't sure where to put this," or "Oh, I didn't know that lived in Silly Moose." So we did have some broad consensus that we needed to do this work and something needed to change.

But then the question was: how should we begin?

As our resident Documentation Goddess, I volunteered to help with this first piece. I have the most experience performing comprehensive audits and knowledge base restructuring. Not only do I advise customers on it all the time, but I also had to do this in our Support KB several years ago, and I remembered what did and didn't work well from that exercise.

But I realized that the Silly Moose audit was different. When I overhauled the Support KB, though I did seek some feedback from other owls, I was able to make decisions by myself because at that time I "owned" the Support KB.

Silly Moose belongs to owl of us, though. It's used by every owl here. Different teams store very different information in it in a variety of formats. If the overhaul was all up to me, I'd be making decisions about content that I'd never seen and had no ownership over. I couldn't just come in and move everyone's cheese in the ways that made sense to me; the changes needed to feel right for everyone.

At the same time, while we all agreed that something had to be done and we'd had several team conversations about it, I also knew that we didn't have total consensus about what "done" looked like. We wanted Silly Moose to "be better," to "be easier to use," and to be trustworthy as a single source of truth. But in conversation, some owls seemed interested in splitting Silly Moose out into multiple, discrete knowledge bases. Others didn't seem to want that. We all agreed that our information architecture didn't make sense, but no one had a feel for what they wanted instead.

In short: we didn't have a clear sense for a) what the problem we most wanted to solve was, and b) consequently, we didn't have a feel for what the solution should be.

I decided this meant that we needed to get a little "meta" in our approach, so instead of pitching a full plan and timeline out of the gate, I focused on defining those things. The tool I selected for this?

An internal needs assessment.

Why a survey?

I knew that I needed to get input from the whole team to see if I could find patterns in both what was bothering owls about our current knowledge base and what they felt a good solution might be.

I also knew that, even though we have a highly collaborative team, it was easy for the most confident owls to dominate conversations. I wanted to give the owls who needed more time to process how they think/feel about something a way to have their voices heard, too.

And, perhaps less obviously, I wanted to know what was working well already.

Too often our conversations had focused on frustrations owls had. Making decisions based on frustration often leads you to throw the baby out with the bath water (the owlet out with the bath...umm...the owlet out with the nest rubbish? Someone please figure out an owl idiom for this).

I was also craving both data and anonymity in submitting that data. Some of our newer owls had the freshest perspective but might not have felt comfortable offering pointed criticism if it had to come straight out of their mouths.

Basically: a survey felt like the right step.

Why a "needs assessment"?

Once I knew I wanted to do a survey, I felt drawn to calling it a needs assessment.

A needs assessment is, by its very definition, a tool to determine needs or "gaps" between current reality and desired outcomes.

It also helped to more clearly set expectations for the team. The survey was fairly detailed and took some time to fill out, and I wanted my team to understand that going in and to take it seriously.

The process

I drafted a set of initial questions and iterated on those.

We had two other champions for the goal identified, so I shared the initial draft of it with them to improve the questions.

Thanks to their feedback, I made a few further tweaks before we rolled it out to the team.

I also tucked additional images of Linus, our mascot owl, onto each page of the survey to add a little delight and encouragement, since the survey did require some time and thought.

The survey

I set up our needs assessment as a multi-page survey. Here's a breakdown of each major page of questions:

Current Silly Moose usage

These questions help us gauge how you're currently using the Silly Moose knowledge base.

1. How often do you access Silly Moose for information?
   • Multiple choice, options ranging from multiple times a day to "What's Silly Moose again?"
2. Which sections do you access most often?
   • Checkboxes with major categories
3. When you access Silly Moose, what kinds of information are you looking at?
   • Checkboxes offering different content "types" that I mostly made up after quickly looking through those categories.
   • Examples include things like:
     • How to do a task
     • How something at KO works
     • How to run a meeting/what to do in an incident/outage
     • Learning resources
     • KO company culture/handbook

Kate's note: We have a LOT of different kinds of content stored in this knowledge base, and I was trying to determine what the most common usage patterns were to figure out what people might be the most strongly invested in, which areas we might target first, and/or which areas might cause the most pushback if we changed them. It was also an interesting exercise trying to categorize into content types.

The Silly Moose experience

These questions are designed to help us understand the experience of using SM, and where we can improve it!

1. When you use Silly Moose, how do you find what you need?
   • Checkboxes for various access patterns
   • Examples include:
     • Search
     • Favorites
     • Recent articles
     • Table of contents
     • Direct link
2. How easy is it to find what you need in Silly Moose?
   • Slider with "Super hard" at the left, "It's okay" in the middle, and "Wicked easy" at the right
3. Is Silly Moose a single source of truth for internal knowledge?
   • Likert Scale ranging from "Definitely not", "Not really", "Maybe", "Somewhat", and "Yes, totally"
4. What are your feelings on Silly Moose moving forward?
   • Multiple choice designed to help us figure out if we kept a single knowledge base or split it out and assess how strongly people felt about that.
   • Examples include:
     • Keep everything in a single KB + very minimal reorg (please don't move my cheese!)
     • Keep everything in a single KB + reorganize what we have so it makes more sense (move my cheese, but keep it in the same room)
     • Split out some areas into their own separate KBs (move my cheese all over the place, maybe give me ice cream)
     • Do whatever, I really don't care what happens with Silly Moose, just tell me how to find what I need (I eat Kraft singles. Only.)
     • Do whatever, I don't use Silly Moose (I'm lactose-intolerant)
5. Why did you choose that answer?
   • Freeform text field where people could expand on that selection.
6. If we did split our internal knowledge into multiple knowledge bases, how might we do that?
   • Freeform text field

Kate's note: I thought of this section as the "how do you feel about Silly Moose" section. I was curious what people preferred to use as navigation patterns, since that could take some pressure off of us to get the information architecture "right," how interested they were in massive changes, and whether people were generally for or against splitting things into separate knowledge bases. Since I had no idea whether the majority would be for or against splitting into separate KBs, I gave a lot of freeform text options here so I could capture people's feelings in their own words. A lot of the call for this project was based on feelings about Silly Moose and I wanted to try to quantify those feelings a bit more.

Knowledge gaps

While we're going to do a comprehensive audit of what's in Silly Moose, we also want to know where we have gaps.

1. Do you feel like the info in Silly Moose is up-to-date and trustworthy?
   • Multiple choice
   • Options included:
     • Yes
     • No
     • Not sure—what I use seems to be good but I can't speak for other things
     • Other
2. Is there any info that you wish we documented internally that isn't currently in Silly Moose? (This can be in your ideal world, not necessarily something we'd do this quarter.)
   • Multiple choice options for Yes/No/Other, specifically so we could use conditional logic to display the next question
3. What information would you like to see in an internal KB that isn't currently in Silly Moose?
   • Freeform text, question only shown if they answered Yes or Other to the previous question.
4. Are there any other problems or questions that you wish Silly Moose could help address? (Not necessarily specific pieces of content, but maybe entire classes or types of knowledge we don't currently have?)
   • Multiple choice options for Yes/No/Other, specifically so we could use conditional logic to display the next question
5. What are the problems or questions you wish Silly Moose could help address?
   • Freeform text, question only shown if they answered Yes or Other to the previous question.

Kate's note: This was not strictly necessary for our auditing project, but it was in keeping with the "needs assessment" concept. Since this was the only time we've ever surveyed our team about internal knowledge management, I might have taken advantage of it just a bit. 😉

Authorship

These last few questions are about authoring or creating content, not reading or consuming it.

• How frequently do you make changes as an author in Silly Moose? This could be writing new content from scratch or updating existing content.
  • Multiple choice, options ranging from "Habitually (1+ times a week)" to "Rarely (once a quarter)" and "Never." Next question is only displayed if they answer anything other than "Never."
• So you have been an author on something in Silly Moose. Fantastic! When you've authored stuff, what has motivated you to do it?
  • Checkboxes, select all that apply
  • Options include:
    • Increase the bus factor: I want to stop getting asked questions about something, or I know I'm the only person who knows it.
    • Improve accuracy: I fix things when I notice they're outdated or inaccurate
    • Sense-making: I document stuff when I spent a lot of time figuring it out to save other owls or my future self from being confused
    • Strengthening our culture: I add policies, procedures, or cultural artifacts that help new and current owls connect more with who we are and what we do
    • Don't forget: I add stuff I'm worried I'll forget
    • Other
• Whether you've authored content or not, have you ever felt discouraged from creating or editing content for any of these reasons?
  • Checkboxes, select all that apply. Options include:
    • I don't feel I know enough/I'm not enough of an expert
    • I don't understand how Silly Moose is organized and didn't want to ask where something should go
    • I don't have enough time to write or update documentation
    • I don't see the point in adding or updating things in Silly Moose
    • I feel intimidated to create or update Silly Moose
    • I don't know if there are authoring or review processes
    • I'd be happy to help with docs but I don't know what I could help with
    • I'm not a good writer; no one wants docs from me
    • Other

Kate's note: With internal knowledge bases, one of the biggest hurdles to people using them is getting folks to create or update content. While I could pull statistics for this in terms of articles created, published, or modified, I was really curious to get at some of the underlying motivations and feelings about things. We had no formal program for onboarding owls into Silly Moose itself, no guidance on creating or updating content, no templates, and no formal authoring/review process. My hypothesis was that some owls didn't feel comfortable authoring or updating content because of this lack of structure; this section was mainly designed to test that hypothesis.

The rollout

Once we had the needs assessment survey finalized, we rolled it out to the team.

KnowledgeOwl is a small team, so the rollout for this was fairly straightforward. Due to timing, I ended up launching it on December 14th, which wasn't really ideal for holiday/vacation time.

Here's how we communicated it:

1. At our weekly team meeting during the announcements section, I announced what the needs assessment survey was, tied it to the company goal, and let everyone know they'd see a message from me about it.
2. That same day, I posted in Slack: "As mentioned in Braintrust, for our 'Get our docs in a row' quarterly goal, we're starting with a needs assessment. Please fill out this survey to help us get a feel for our internal knowledge management needs so we can craft feathers appropriately!"
3. Five days later, Marybeth, our CEO, sent a team reminder in Slack encouraging people to submit it.
4. I sent one final reminder on December 20th.
5. I planned to review the results in early January to craft our next steps.

Next steps

In my next post, I'll share the results of our needs assessment and how we chose to proceed. If you've done a similar type of survey or needs assessment for knowledge management at your organization, I'd love to hear from you!