3.3:Reliability of search strings Revision as of 13:02, 13 April 2015 by Britt (Talk | contribs) (diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

The quality of your automatic content analysis depends on the quality of your search strings, which in turn depends on the reliability of your search strings. The reliability of a study refers to the question of whether this study is replicable or not. When replicable, other researchers find the same results using your research method. Besides reliable, your research method should also be valid, meaning that you actually measure what you intend to measure. Inreliable search terms per definition lead to invalid results. In automatic content analysis reliability is high, as different computers with identical instructions will generate the exact same results. However, validity is low, since a computer recognizes words but not concepts. In contrast, human coders with the same instructions will not always get the same results due to personal interpretations and cultural backgrounds, so the reliability of manual content analysis is often lower than the reliability of automatic content analysis. However, human coders are capable of recogizing concepts, which improves the validity of the results.

An important distinction with regard to reliability in content analysis is the distinction between precision and recall. Precision refers to the question of whether the results found by AmCAT have justly been identified as positive results (i.e. texts that include the concept of your interest). Type 1 errors, search results that were wrongly identified as positive, decrease the precision of your search term. Recall refers to the question of whether all the results that include your concept of interest have indeed been found by AmCAT. Type 2 errors, search results that were wrongly identifies as negative, decrease the recall of your search terms. When the precision of your search terms is high, the recall is generally lower and vice versa. As a researcher, you thus need to find a balance between concessions with regard to precision and concessions with regard to recall. However, before you calculate the precision and recall of your search terms, it is important that you check the face validity of the search terms and the results.

Face validity

You can check the face validity of your search terms by taking a look at the AmCAT search results. You can do so by reading the documents that are identified as including your search terms and estimating whether they include the concept you intend to measure. AmCAT provides you with various opportunities to get access to these documents:

• Using the Summary function, you can list all the articles including your search terms. You can access each of these documents by clicking on the titles in the list. You search terms are highlighted in red.
• Using the 'Graph' option of the Graph/Table function, you can click on every dot in the line and you will get a list of relevant articles. By clicking on the titles in the list you can access each document.
• Using the 'ClusterMap' option of the Summary function, you can make a Venn diagram. By clicking on a dot in the Venn diagram, you get access to this particular document. If you have a relativey large number of documents, the venn digram displays a single large dot. By specifying your search instructions by selecting a certain period or medium, you can narrow the number of documents down and dots will appear.