From: Automatically identifying the function and intent of posts in underground forums
Bulletin board | Task | Annotator 1&2 | Annotator 1&3 | Annotator 2&3 | Fleiss’s \(\kappa \) | Agreement |
---|---|---|---|---|---|---|
Beginners hacking | Post type | .917 | .908 | .928 | .736 | Substantial |
Author intent | .760 | .535 | .537 | .239 | Fair | |
Addressee | .893 | .871 | .903 | .879 | Almost perfect | |
Premium sellers | Post type | .854 | .935 | .844 | .583 | Moderate |
Author intent | .849 | .834 | .799 | .405 | Moderate | |
Addressee | .879 | .804 | .910 | .840 | Almost perfect | |
Mixed forums | Post type | .889 | .910 | .912 | .719 | Substantial |
Author intent | .770 | .786 | .731 | .280 | Fair | |
Addressee | .868 | .721 | .765 | .760 | Substantial |