Evaluation Queues

Evaluation Queue helper functions

class challengeutils.evaluation_queue.SubmissionQuota(firstRoundStart: str = None, roundDurationMillis: int = None, numberOfRounds: int = None, submissionLimit: int = None)

Bases: object

A SubmissionQuota object https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/SubmissionQuota.html

challengeutils.evaluation_queue.set_evaluation_quota(syn: synapseclient.client.Synapse, evalid: int, **kwargs)

Sets evaluation submission limit quota. This WILL erase any old quota you had previously set. Note - round_start must be specified with either round_end or round_duration and number_of_rounds must be defined for the time limits to work. submission_limit will work without number_of_rounds.

Parameters

• syn – Synapse object

• evalid – Evaluation id

• **kwargs –

  round_start: Start of round (local time) in YEAR-MM-DDTHH:MM:SS

  format (ie. 2020-02-21T17:00:00)

  round_end: End of round (local time) in YEAR-MM-DDTHH:MM:SS format

  (ie. 2020-02-21T19:00:00)

  number_of_rounds: Number of rounds round_duration: Round duration in milliseconds submission_limit: Number of submissions allowed per team

Returns

A synapseclient.Evaluation

Examples

>>> set_evaluation_quota(syn, 12345,
                         round_start="2020-02-21T17:00:00",
                         round_end="2020-02-23T17:00:00",
                         number_of_rounds=1,
                         submission_limit=3)