Case Study: AI Comment Moderator

How it all started

Beat Of Hawaii, founded in 2008 by Jeff and Rob, is a beloved source for Hawaii travel news, tips, culture, events, and deals, attracting 10 million yearly visitors. Our success with Beat Of Hawaii inspired us to create TechSpokes, where we specialize in innovative technology and superior marketing strategies for the vacation rental industry. To date, with 14 years in this business, our clients have generated over $127 million in direct online booking revenue using our solutions.

The Challenge: Overwhelming Comment Moderation

In the early days of Beat Of Hawaii, managing comments was simple and straightforward. Sixteen years ago, our community was smaller, and spam activity was minimal. We could easily keep up with the handful of comments we received daily, engaging with our readers and fostering a vibrant community.

However, as Beat Of Hawaii grew in popularity, so did the volume of comments and the influx of spam. Fast-forward to recent years, and we deal with around 100 legitimate comments daily, alongside another 100 spam comments. What was once a manageable task became a daily challenge that consumed significant resources.

Moderating this volume of comments became daunting and time-consuming, requiring about three hours of work each day. This workload was split between filtering, managing comments, and actively engaging with our readers.

Filtering Comments

Approximately an hour and a half was spent sorting through the comments, distinguishing between spam, trash, valid comments, and high-quality interactions. This task was tedious and required meticulous attention to ensure that no valuable comments were missed while keeping the spam at bay.

Editing and Responding

The remaining hour and a half was dedicated to editing some of the comments for clarity and appropriateness, as well as replying to them to maintain engagement with our community. This interaction was crucial for building and maintaining the trust and loyalty of our readers, but it also added to the already heavy workload.

While we tried multiple solutions available to WordPress to combat spam, spammers evolved as well, making these solutions less effective over time. As a result, moderating our website comments became increasingly tedious.

the Solution: AI Comment Moderator Plugin

When the AI revolution began in the early 2020s, we at TechSpokes realized we had a unique advantage. Our CTO, Serge, not only has extensive experience in software development but also a solid background in linguistics.

With the growing volume of comments and spam on Beat Of Hawaii, spending countless hours on moderation was no longer an option. We needed a more efficient solution to keep up with our expanding audience. Leveraging our in-house expertise, we created the AI Comment Moderator plugin.

The idea was simple yet transformative: develop an AI-powered tool to automate comment moderation, save us time, and ensure our community remained vibrant and engaged. Thus, the AI Comment Moderator plugin was born, tailored specifically to meet our needs at Beat Of Hawaii.

Development and Implementation

The development and implementation of the AI Comment Moderator plugin were straightforward, thanks to our extensive experience in developing WordPress plugins at TechSpokes. Our main focus was identifying the daily tasks that consumed most of our time and could be automated by AI.

We pinpointed the key tasks and designed a system to rate each comment based on its quality, using the following scoring system to determine the appropriate action:

• Score 0: Discard the comment and dump it immediately.
• Score 1: Place the comment in the trash for potential future review.
• Scores 2-3: Mark the comment as spam, keeping it separate from the trash.
• Scores 4-7: Move the comment to the moderation queue for human review.
• Score 8: If the author has previously approved comments, publish the comment. If not, put it back in the moderation queue.
• Score 9: Publish the comment immediately to keep the conversation lively.

To enhance our efficiency further, we improved the comment management interface in WordPress by introducing the sorting of comments in the admin tables by their score. This allowed us to sort comments in the moderation queue from high to low scores, enabling us to quickly approve and/or reply to high-quality comments.

By doing so, we kept the conversation dynamic on the front end, while still addressing the lower score comments and performing necessary edits. This method ensured that high-quality interactions were prioritized and published quickly, maintaining an engaging and responsive community.

Additionally, we added special settings to allow us to modify the AI comment moderator system prompts. This flexibility ensures that we can easily adjust our comment policy and fine-tune how the AI scores comment value, allowing us to stay adaptable and maintain our high standards for community interaction.

Managing Costs

While the costs associated with using the OpenAI API are relatively low, we wanted to ensure our solution was both cost-effective and robust against massive spam attacks. On average, we spend less than 20 cents daily, translating to approximately $1 per 1k comments scored by the AI Comment Moderator plugin.

To effectively address occasional spam attacks, we developed a special “fortress mode” for the plugin. This mode cuts the costs by analyzing the links in comments locally (before any API calls). The plugin compares them to links in spam comments, already scored by the AI engine. If a comment contains links to domains previously flagged as spam, the plugin handles it locally without calling the API.

This local processing is both fast and efficient, allowing us to maintain robust defenses against the most aggressive spam attacks while keeping costs down.

Results

After implementing the AI Comment Moderator plugin, we observed huge improvements in our comment moderation process. While we still spend about an hour and a half editing and replying to comments, the comment-filtering task itself has been reduced from an hour and a half to just 10-15 minutes.

Over the last nine months, this reduction has effectively saved us about an hour of daily moderation work. This not only frees up valuable time but also streamlines our moderation process, allowing comments to appear faster on the front end and keeping the conversation lively and engaging.

Given the popularity of Beat Of Hawaii, timely moderation is crucial. When we publish a blog post, Google begins indexing it and pushing it to Google News within minutes. The first comments, including spam, start rolling in within five to ten minutes. However, with the AI Comment Moderator plugin, we can focus on what truly matters: maintaining an engaged and vibrant community. The plugin handles the bulk of the comment filtering work, allowing us to ensure that our readers’ comments are visible quickly, enhancing the overall interaction on our site.

Conclusion

To wrap up, let’s look at some numbers. The AI Comment Moderator plugin license costs $15 a month if you pay monthly, or $10 a month if you pay yearly. The API costs are about 20 cents a day for handling hundreds of published comments and filtering another hundred spam comments. This setup effectively buys back one hour of our time each day, or roughly 30 hours a month, for a cost close to $20 a month.

At approximately $0.70 per hour saved, this plugin is a game changer for us. It not only saves time but also ensures our moderation process is efficient and our community engagement remains high. What about you? Would you like to buy back an hour of your time for just 70 cents?

Special Offer

To celebrate the launch of our store on May 15th, 2024, we’re offering an exclusive discount for our early subscribers. Sign up for our newsletter below, receive your promo code, and be among the first to experience the time-saving power of our AI Comment Moderator plugin. Don’t miss this chance to streamline your comment moderation and reclaim valuable hours every day.