Is Your Free ChatBot Giving You the Right Answers?

When “Turn It On” Is the AI Strategy

It’s understandable how organizations get here.

A free chatbot may already be available through software the organization uses. Another tool might be inexpensive and promise a relatively simple setup. Give it access to some content, configure a few options, add it to the website and suddenly your organization has a chatbot.

There’s nothing wrong with starting that way. A relatively simple tool might be all an organization needs. The problem comes when everyone moves on to the next project.

Consider an association preparing for its annual conference. During the months leading up to the event, a suggested chatbot question such as “How do I register for the annual conference?” makes perfect sense.

Then the conference happens.

Three months later, that same question is still sitting there waiting for visitors to click it. Maybe the answer still points toward registration. Maybe the bot finds an old conference page. Either way, the technology itself hasn’t failed. Nobody remembered that the chatbot needed attention after the conference was over.

That’s an easy mistake to make because most website problems announce themselves. A broken link goes nowhere. A malfunctioning form doesn’t submit. A missing image leaves an empty space.

A chatbot can be wrong quite gracefully. The result can be a polished, confident response based on information that was perfectly accurate . . . last year.

Your Old Content Has a New Audience

This also makes an old website problem more important.

Most established organizations have outdated content somewhere on their websites. There may be old event pages, PDFs from previous campaigns, former staff biographies, outdated program information or pages describing policies that have since changed. Sometimes those pages haven’t caused much trouble because they’re difficult for human visitors to find.

But a chatbot configured to retrieve information from those sources may find them.

That means content maintenance is no longer just about keeping the visible website tidy. If your chatbot relies on website content or another organizational knowledge source, maintaining that information becomes part of maintaining the chatbot.

More content isn’t automatically better, either.

Imagine an association with a website that has been accumulating content for 10 or 15 years. Giving a chatbot access to all of it might sound helpful. But somewhere in those thousands of pages could be an old dues structure, a discontinued member benefit, instructions for a program that no longer exists and information about dozens of events that ended years ago.

The question isn’t simply how much information the chatbot can access. It’s whether it has access to the right information.

That requires somebody to make decisions about content.

Try Asking Your Own Chatbot 10 Questions

There’s a very simple test for any organization that already has a chatbot. Go use it.

Don’t ask one question and stop when you get a reasonable answer. Ask 10.

Ask what membership costs. Ask when the next event is. Ask how to renew. Ask who runs the organization. Ask about a program that recently changed. Click the suggested questions. Follow the links in the answers.

Then ask the same question in an odd way. Give it an ambiguous question. Ask about something the chatbot shouldn’t know. See what it does when it doesn’t have a good answer.

You can learn quite a bit in 15 minutes. The important part is doing it again later.

Six months from now, the chatbot may be exactly the same, but your organization won’t be. Events will have come and gone. Staff members may have changed. New pages will have been published. Programs and policies may have been updated.

Testing a chatbot at launch tells you how it performed at launch. It doesn’t tell you how it’s performing today.

Who Owns It?

This may be the most useful question an organization can ask about its chatbot:

Whose job is it to make sure this thing is still right?

It doesn’t necessarily require a full-time employee or an elaborate AI governance committee. But somebody needs responsibility for it.

That person or team needs to know what information the chatbot uses, periodically test its answers, review the questions people are asking and make adjustments when something changes. That process can produce some unexpected benefits.

Suppose visitors continually ask the chatbot how to renew their memberships. The chatbot may be answering correctly, but the repeated question tells you something else: perhaps renewal information is too hard to find on the website.

Or people repeatedly ask about a service that gets only a paragraph on an interior page. Maybe that service deserves better content.

A chatbot that struggles with a common question can even expose weaknesses in the underlying website. If your own organization can’t point to a clear, current source for the answer, it’s difficult to expect the chatbot to do much better.

Suddenly, testing the chatbot isn’t just about the chatbot. It’s another way to see where the digital experience is helping visitors and where it’s getting in their way.

Installing AI Isn’t the Same as Implementing It

This is where the distinction matters. We’re seeing organizations understandably eager to experiment with AI. They should. Useful tools are becoming easier to access, and an organization doesn’t need a giant AI initiative before it starts learning what the technology can do.

But putting a chatbot on a website proves that you can put a chatbot on a website.

A successful AI implementation starts with what the organization and its users actually need. From there, the work may involve selecting and maintaining reliable content sources, connecting other systems, establishing rules about what information can be retrieved or what actions can be taken, testing responses and deciding who will monitor the system after launch.

A large language model may be part of that solution. Another AI technology may be more appropriate. In some cases, a chatbot may not be the best solution at all.

That’s why we’re increasingly interested in a different question from “How do we add AI to our website?”

We’d rather ask: “What are people trying to accomplish on your website, and could AI help them do it better?”

Once you know the answer to that, you can choose the technology.

Go Check Your Chatbot

If your organization already has a chatbot, we’re not suggesting that you turn it off or replace it simply because it was inexpensive or easy to install.

We’re suggesting that you go talk to it.

Ask about something that changed last month. Ask about your next event. Ask about an old event. Click the questions it suggests. See where its answers come from.

You may discover that it’s doing a terrific job.

Or you may discover an event from six months ago that your chatbot apparently still thinks is right around the corner.

Either result is useful, because now you know.