Leaderboard rankings questions

Hello!

  • How are the overall leaderboard rankings done across tasks?
  • How will the results for the secret tasks be reported, if at all?
  • Will there be any live feedback regarding submissions, for example, will we be notified if there are any errors in our submission or when the submission results are available?

Thanks!

  • Currently, we plan to do no overall leaderboard ranking. There will be one ranking per task.
  • Each secret task has measures associated with it, that will be published. Also, some high level metadata about the task (e.g. the domain, the size, etc.) will be shared.
  • If we have any problems running the submission, we will reach out the submitters directly over email.

Thank you!

1 Like

@auroracramer

The first leaderboard will be up in two weeks. Please see recent updates about rolling leaderboard submissions