Labelbox

Configure Auto Consensus

Configuring Auto Consensus

There are two main settings for Auto Consensus:

  1. The percentage of assets to be labeled more than once (referred to as the Consensus Subset)
  2. The labels per asset the Consensus Subset will be labeled by unique labelers
Auto Consensus configuration dashboard. Accessible from the project overview by Settings -> Quality.

Auto Consensus configuration dashboard. Accessible from the project overview by Settings -> Quality.

A Simple Example

Gal is setting up a project in Labelbox and attaches 16,000 images. She is working with a team of 5 labelers. To ensure labeling quality she wants 10% of the data rows to be labeled by 3 out of the 5 labelers. Auto Consensus is configured to represent this by setting the Percentage of assets to 10% and Labels per asset to 3. With this configuration set, 1,600 of the images (10% of the 16,000 images) will be labeled 3 times.

To complete the initial labeling for this project, 17,600 images must be labeled in total. After one day, 3,000 images were labeled. Therefore, in order to finish the initial labeling task, 14,600 images still need to be labeled. The Labelbox project overview stats will say 3,000 Submitted, 14,600 Remaining, 0 Skipped, and 17% Complete.