The quality of your automatic content analysis depends on the quality of your search strings, which in turn depends on the reliability of your search strings. The reliability of a study refers to the question of whether this study is replicable or not. When replicable, other researchers find the same results using your research method. Besides reliable, your research method should also be valid, meaning that you actually measure what you intend to measure. Inreliable search terms per definition lead to invalid results. In automatic content analysis reliability is high, as different computers with identical instructions will generate the exact same results. However, validity is low, since a computer recognizes words but not concepts. In contrast, human coders with the same instructions will not always get the same results due to personal interpretations and cultural backgrounds, so the reliability of manual content analysis is often lower than the reliability of automatic content analysis. However, human coders are capable of recogizing concepts, which improves the validity of the results.

An important distinction with regard to reliability in content analysis is the distinction between precision and recall. Precision refers to the question of whether the results found by AmCAT have justly been identified as positive results (i.e. texts that include the concept of your interest). Type 1 errors, search results that were wrongly identified as positive, decrease the precision of your search term. Recall refers to the question of whether all the results that include your concept of interest have indeed been found by AmCAT. Type 2 errors, search results that were wrongly identifies as negative, decrease the recall of your search terms. When the precision of your search terms is high, the recall is generally lower and vice versa. As a researcher, you thus need to find a balance between concessions with regard to precision and concessions with regard to recall. However, before you calculate the precision and recall of your search terms, it is important that you check the face validity of the search terms and the results.

Face validity

You can check the face validity of your search terms by taking a look at the AmCAT search results. You can do so by reading the documents that are identified as including your search terms and estimating whether they include the concept you intend to measure. AmCAT provides you with various opportunities to get access to these documents:

Precision

Recall

AmCAT Version
This page describes a feature in AmCAT
View other version: 3.3 - 3.4 - 3.5