Skip to main content

Table 4 Inter-annotator agreement for the labelling of HF2200 posts by three annotators

From: Automatically identifying the function and intent of posts in underground forums

Bulletin board Task Annotator 1&2 Annotator 1&3 Annotator 2&3 Fleiss’s \(\kappa \) Agreement
Beginners hacking Post type .917 .908 .928 .736 Substantial
Author intent .760 .535 .537 .239 Fair
Addressee .893 .871 .903 .879 Almost perfect
Premium sellers Post type .854 .935 .844 .583 Moderate
Author intent .849 .834 .799 .405 Moderate
Addressee .879 .804 .910 .840 Almost perfect
Mixed forums Post type .889 .910 .912 .719 Substantial
Author intent .770 .786 .731 .280 Fair
Addressee .868 .721 .765 .760 Substantial