Skip the Automated Readability Test – Do a Usability Test

from John Wooden, Director of Usability Services

on February 27, 2012

Back in January it was reported that President Obama’s State of the Union address was written at an 8th grade reading level. In fact, a Smart Politics study of the 70 orally delivered State of the Union Addresses since 1934 found “the text of Obama's 2012 speech to have tallied the third lowest score on the Flesch-Kincaid readability test, at an 8.4 grade level. Obama also delivered the second lowest scoring address in 2011 (at an 8.1 grade level), and the sixth lowest in 2010 (at an 8.8 grade level).”

It makes sense for President Obama to focus on readability. He signed the Plain Writing Act of 2010, which requires the federal government “to write new publications, forms, and publicly distributed documents in a clear, concise and well organized manner that follows the best practices of plain writing.” I’m all for plain language and I applaud organizations that embrace it. (Kudos to Hennepin County in Minnesota for following the Plain Language law even though they don’t have to.) But I question the value of using automated tests to assess readability and I doubt the meaningfulness of the reading grade levels these tools spit out.

What automated tests don’t tell you

The Flesch-Kincaid test is a tool that many of us have used to calculate the reading grade level of text we’ve written or edited. But what do automated readability tests really tell us? What’s it mean when they say some piece of text is written at an 8th grade level? In truth, not much.

Automated readability tests are based on formulas and the formulas are based on elements that can be counted. The Gunning-Fog index, the SMOG (Simple Measure of Gobbledygook) test, and the Flesch reading ease scale are all based on counting words per sentence and syllables per word. In addition, as Janice (Ginny) Redish has pointed out*, the assumption underlying readability formulas – “that any text for any reader for any purpose can be measured with the same formula” – is simply invalid. Redish notes that automated readability tests leave a lot of questions unanswered:

• Is the content a good match for the audience?
• Is the audience familiar with the words used?
• Is the document organized so that users can find what they are looking for?
• Does the document include headings? Are the headings useful and meaningful to the audience?
• Does the page layout help users find what they need?
• Are the sentences grammatical?
• Are the style and tone appropriate?

Needless to say, these questions are all very pertinent in assessing readability, but they don’t lend themselves to simple counts. And there are other substantial weaknesses, too:

• Automated readability tests don’t work well on web pages that are written to be scanned. These tests count sentence length from period to period. So if you have text on a web page that uses lots of bullet points that do not all end with periods (which is just fine in my book), this will be a problem for an automated test. For the same reason, text presented in a table causes a problem, too.
• Readability tests don’t tell you how to make your text more readable. And in the act of trying to get a better readability score or lower reading grade level, according to the formula, you could end up creating new problems that render your text less readable. (I could eliminate the five-syllable word “readability” from this text and get a lower reading grade, but that wouldn’t be helpful.)

Why usability testing is better

The simple answer is that it provides a much more comprehensive and accurate gauge of how your users/readers perceive your text. I’ll give you an example.

A while back, I subjected some text that is sent to applicants for Minnesota unemployment benefits to the Flesch-Kincaid test to find out the reading grade level. It was high, too high. One of the reasons for this was the common use of such four-syllable words as “Minnesota” and “unemployment,” and the three-syllable word “applicant.” But would typical applicants struggle with these words? No. How do we know this? Because we did usability testing with actual applicants. A two-syllable word, “appeal,” caused more difficulty than the longer words that would be flagged in an automated test. But what testers struggled with more than any particular word was the order of ideas. The correspondence followed a general-to-specific structure, beginning with background and reference to a Minnesota statute, before going on to explain the specific determination of the applicant’s eligibility for unemployment benefits. What our testers told us was that they wanted to see this order reversed: start with what’s specific to the recipient – their eligibility – and then go on to the general and background information. We suspected this would be the case going into the tests, but it was good to have it confirmed by actual applicants. In any case, no automated readability test could have helped either identify or solve this problem.

Similarly, the biggest issue with another piece of correspondence was tone. Automated readability tests will not tell you anything about that either.

A usability test with a special focus on readability will tell you much, much more about how actual readers perceive your text than the narrow focus on syllables and word count in an automated test. They are quick, easy, and free, and they have the allure of quantitative data. But you get what you pay for.

What about that 8th grade reading level?

It’s interesting that President Kennedy had the highest average reading grade level in his State of the Union addresses: 13.2. But JFK understood the power of rhetoric pretty well, and I’ll bet that despite the high reading grade level, some of his words stick with you.

*"Readability Formulas Have Even More Limitations Than Klare Discusses," ACM Journal of Computer Documentation, August 2000/ Vol. 24, No.3.

4 comments

Articulate vs. Captivate Part 4:  And the winner is. . .

from Tony Tao, Instructional Designer and eLearning Developer

on February 03, 2012

Author’s Note: This blog entry is part of a series I started to explore two of today’s most popular eLearning rapid development tools: Articulate Studio and Adobe Captivate. Here is a link to an article that contains the whole Articulate vs. Captivate series.

In the previous blog entries, we have explored the major features of Articulate and Captivate, and discussed the strengths and limitations of each tool. Of course, there really isn't a winner. As I wrote at the beginning of this series, the only answer to the question “Which is better?” is “It depends.” The tools have different strengths and the best fit depends on your needs.

And for larger organizations or those with more complex or varied learning needs, the answer to the question “Which should I buy?” is often “Both.”

I've created a summary chart that I think clearly highlights the strengths of the two tools. Of course, some of these items can’t be reduced to a simple yes-or-no answer, so in some cases this chart simply reflects my opinion.

In 2012, we will see new players joining the rapid eLearning tool game. For example, Articulate Storyline and ZebraZapps are already attracting a lot of attention. There is also the possibility of new releases of Articulate Studio, Adobe Captivate, and SmartBuilder.

One of the interesting trends that we have noticed is the rise of mobile learning, and how the rapid eLearning tools are quickly incorporating functionality that gives them the potential to create mLearning content. For example, most of the new tools can publish your project as HTML 5 or in the mp4 video format. This gives eLearning developers an easier path to get a course running on Apple mobile devices such as the iPad.

I expect to see more projects developed with these new tools in 2012 and I will be using them myself for Fredrickson's Learning business. As always, I'm glad to share my thoughts and findings with you and I appreciate your comments on these blog entries.