Admin Built-in Commands

The following commands must be ran with challengeutils, e.g.

$ challengeutils <built-in command> <options>

Note that the majority of the commands listed here will require certain permissions on the relevant Sypanse entities, e.g. admin permissions or score permissions.

For more information on Synapse permissions, see here.

Create a challenge

Synopsis

create-challenge: “CHALLENGE NAME HERE” [–livesiteid SYNID]

Description

Create a challenge space in Synapse, including two Projects and four Teams that are needed for a challenge. For more information on various Challenge components, see challenge administration.

Positional

"CHALLENGE NAME HERE": Name of the challenge

Optional

--livesiteid SYNID: Overwrite an existing live site

Mirror wiki changes

Synopsis

mirror-wiki: source_id dest_id [–force]

Description

Sync changes made from a source wiki to the destination wiki, e.g. from staging to live. Updates will only occur in wiki pages that have matching titles in the source wiki.

Important

This command is different from copying wikis. To copy, see synapseutils.

For best practice, make all edits to the staging site of a challenge only, then use this command to sync over the changes to the live site.

Positional

source_id: Synpase ID of the source wiki to be synced

dest_id: Synapse ID of the destination wiki to be synced to

Optional

-f, --force: Force-update the wiki pages, even if there are no changes

List evaluations

Synopsis

list-evaluations: project_id

Description

List all evaluation queues of a Synapse project.

Positional

project_id: Project ID on Synapse, e.g. syn12345678

Set an evaluation quota

Synopsis

set-evaluation-quota: eval_id [–round_start yyyy-MM-ddTHH:mm:ss [–round_end yyyy-MM-ddTHH:mm:ss] [–round_duration n] [–num_rounds n] [–sub_limit n]

Description

Set the quota of an evaluation queue. Quota options include the round starting date, round ending date, round duration, number of rounds, and submission limit.

Warning

When this command is used, all settings previously set for the queue will be erased. For any settings you do not want to update or remove, pass the original values into the optional parameters defined below.

Positional

eval_id: Evaluation ID on Synapse, e.g. 9876543

Optional

--round_start: Start of round in yyyy-MM-ddTHH:mm:ss (local military time) format

--round_end: End of round in yyyy-MM-ddTHH:mm:ss (local military time) format; do not use with round_duration

--round_duration: Round duration in milliseconds; do not use with round_end

--num_rounds: Number of rounds (must set for time-related quotas to work)

--sub_limit: Number of submissions allowed per round

Update an evaluation ACL

Synopsis

set-evaluation-acl: evalid user_or_team permission_level

Description

Set the evaluation permissions for user_or_team with permission_level access.

Positional

eval_id: Evaluation ID on Synapse, e.g. 9876543

user_or_team: Synapse user or team ID, e.g. 1234567

permission_level: One of: view, submit, score, admin, remove

Query an evaluation

Synopsis

query: “QUERY” [–outputfile file] [–render] [–limit n] [–offset n]

Description

Query an evaluation queue.

Positional

"QUERY": SQL-like query in URI format

Optional

--outputfile file: Print query results to this file (default: prints to stdout)

--render: Render submitterId and createdOn values in leaderboard

--limit 20: Only return this number of results (default: 20)

--offset 0: Return results starting at this offset (default: 0)

Download a submission

Synopsis

download-submission: sub_id [–download_location path] [–output file]

Description

Download a Submission object.

Positional

sub_id: Submission ID on Synapse, e.g. 9876543

Optional

--download_location path: Specify download location (default: current working directory)

--output file: Print JSON results to this file (default: prints to stdout)

Annotate a submission

Synopsis

annotate-submission: sub_id json_file [-p ] [-f]

Description

Annotate a Submission object with a JSON file. The file should comprise of a list of key-value pairs, where the key is the annotation and the value is the annotation value, e.g.

{
    "round": 1,
    "score": 100
}

Positional

sub_id: Submission ID on Synapse, e.g. 9876543

json_file: Filepath to the JSON file containing the annotations

Optional

-p, --to_public: Allow the annotation to be viewable by the public (default: annotation is viewable by the queue administrator(s) only)

-f, --force: Force the update, even if the key has a different ACL

Update a submission status

Synopsis

change-status: sub_id status

Description

Update the status annotation of a Submission Status object.

Positional

sub_id: Submission ID on Synapse, e.g. 9876543

status: One of: RECEIVED, ACCEPTED, INVALID, VALIDATED, SCORED, OPEN, CLOSED, EVALUATION_IN_PROGRESS, REJECTED

Stop a Docker submission

Synopsis

stop-submission-over-quota: eval_id time_quota

Description

Terminate a Docker submission (usually applies to submissions that have a runtime longer than the alloted time).

Positional

eval_id: Evaluation ID on Synapse, e.g. 9876543

time_quota: Time quota in milliseconds allowed for a submission

Update an entity ACL

Synopsis

set-entity-acl: ent_id user_or_team permission_level

Description

Set the entity permissions for user_or_team with permission_level access.

Positional

ent_id: Entity ID on Synapse, e.g. syn12345678

user_or_team: Synapse user or team ID, e.g. 1234567

permission_level: One of: view, submit, score, admin, remove

Validate a Project Submission

Synopsis

validate-project: submissionid challengewiki permission_level [-p] [-a] [-o]

Description

Validates a Synapse Project submission by making sure it is shared with the correct users.

Positional

submissionid: Submission ID on Synapse, e.g. 9876543

challengewiki: Synapse ID of challenge project

Optional

-p, --public: Check that the Project is shared with the public

-a, --admin: Check that the Project is shared with this admin username/team

-o, --output: Output the validation results into a json file

Archive a Project Submission

Synopsis

archive-project: submissionid admin [-o]

Description

Archives a Project submission by copying it

Positional

submissionid: Submission ID on Synapse, e.g. 9876543

admin: Synapse ID of user or team

Optional

-o, --output: Output the results into a json file

Delete a submission

Synopsis

delete-submission: sub_id

Description

Delete a submission

Positional

submissionid: Submission ID on Synapse, e.g. 9876543

Validate a Docker Submission

Synopsis

validate-docker: sub_id [-o]

Description

Validates a Docker Submission to make sure the image and sha-digest exists.

Positional

submissionid: Submission ID on Synapse, e.g. 9876543

Optional

-o, --output: Output the results into a json file

Pull a Synapse Project’s Wiki

Synopsis

pull-wiki: projectid [–workdir]

Description

Download each wiki page of a synapse project as individual .md files and saves a json configuration (wiki_config.json).

Positional

projectid: Synapse project ID e.g. syn123455

Optional

--workdir: Path to download markdown files and wiki_config.json Defaults to location of where code is being executed

Push a Synapse Project’s Wiki

Synopsis

push-wiki: projectid [–workdir]

Description

Updates Synapse wiki by uploading markdown files linked together by a wiki_config.json. The markdown and wiki_config.json can be obtained by the pull-wiki command. The wiki_config.json is validated prior to any updates, if there are any valiation errors, your wiki pages will not be updated.

Positional

projectid: Synapse project ID e.g. syn123455

Optional

--workdir: Path to markdown files and wiki_config.json Defaults to location of where code is being executed

Scan an evaluation queue for cheating

Synopsis

cheat-detection: evaluationid [-s]

Description

Scans an evaluation queue for possible pairs of users who are trying to skirt the daily submission limit by creating multiple users to submit to a challenge. A report is generated with possible linked users and the strength of evidence that any two users are linked.

Positional

projectid: Synapse evaluation ID e.g. 9615080

Optional

-s, --submission_status: List of submission statuses to use when scanning for valid submissions. Accepts multiple inputs. Defaults to ACCEPTED