Skip to main content

Table 4 Inter-annotator agreement for the labelling of HF2200 posts by three annotators

From: Automatically identifying the function and intent of posts in underground forums

Bulletin board

Task

Annotator 1&2

Annotator 1&3

Annotator 2&3

Fleiss’s \(\kappa \)

Agreement

Beginners hacking

Post type

.917

.908

.928

.736

Substantial

Author intent

.760

.535

.537

.239

Fair

Addressee

.893

.871

.903

.879

Almost perfect

Premium sellers

Post type

.854

.935

.844

.583

Moderate

Author intent

.849

.834

.799

.405

Moderate

Addressee

.879

.804

.910

.840

Almost perfect

Mixed forums

Post type

.889

.910

.912

.719

Substantial

Author intent

.770

.786

.731

.280

Fair

Addressee

.868

.721

.765

.760

Substantial