Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Propose queue changeTaskRunPriority #190

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,3 +68,4 @@ See [mechanics](mechanics.md) for more detail.
| RFC#180 | [Github cancel previous tasks](rfcs/0180-Github-cancel-previous-tasks.md) |
| RFC#182 | [Allow remote references to .taskcluster.yml files processed by Taskcluster-GitHub](rfcs/0182-taskcluster-yml-remote-references.md) |
| RFC#189 | [Batch APIs for task definition, status and index path](rfcs/0189-batch-task-apis.md) |
| RFC#190 | [Queue change task run priority](rfcs/0190-queue-change-task-run-priority.md) |
55 changes: 55 additions & 0 deletions rfcs/0190-queue-change-task-run-priority.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# RFC 0190 - Queue change task run priority
* Comments: [#0190](https://github.com/taskcluster/taskcluster-rfcs/pull/190)
* Proposed by: @lotas

## Summary

Since we refactored queue internals it became possible to see what tasks are pending for a particular
worker pool / task queue.

This RFC proposes new API method that would allow to change the priority of the existing task run.

## Motivation
lotas marked this conversation as resolved.
Show resolved Hide resolved

There might be several use-cases when some worker pool has a lot of pending tasks and there is a need
to move some more important task to the top of the waiting list, or also move down less important ones.
lotas marked this conversation as resolved.
Show resolved Hide resolved

Instead of cancelling existing tasks we can instead choose to change task run priorities.

## Details

I propose we implement this on a task run level and not the task itself:

* tasks are supposed to be immutable
* changing the priority makes more sense in the context of a specific run, not the whole task
lotas marked this conversation as resolved.
Show resolved Hide resolved

Queue service will expose new method `queue.changeTaskRunPriority(taskId, runId, newPriority)`.
lotas marked this conversation as resolved.
Show resolved Hide resolved

This will try to do the following:

* update `task.runs[runId]` in the database to reflect priority change (possibly extra field to keep the original)
* update `priority` field in the `queue_pending_tasks` table

If any of this fails (for example task is being changed by some other process), this method will also fail.

This will also require introduction of the new scopes (anyOf):

* `queue:change-task-run-priority-in-queue:<taskQueueId>`
* `queue:change-task-run-priority:<taskId>`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering if there's a case to be made for the source and target priorities being included in the scopes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point!
source <> target would be 8 to 8 permutations, maybe too much?

We can, however, add more generic ones like lowering or raising the priority, so two separate scopes?
queue:lower-task-run-priority:<taskId>
queue:raise-task-run-priority:<taskId>



Current order for picking up tasks is based on the task priority and insertion time (FIFO).
This RFC proposes to change the priority only, and leave the insertion time as is.


### Other considerations

Chain of Trust validation might be broken if we change too much of the task definition or run.
lotas marked this conversation as resolved.
Show resolved Hide resolved
We need to make sure we don't break it with this change.




## Implementation

_pending_
1 change: 1 addition & 0 deletions rfcs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,4 @@
| RFC#180 | [Github cancel previous tasks](0180-Github-cancel-previous-tasks.md) |
| RFC#182 | [Allow remote references to .taskcluster.yml files processed by Taskcluster-GitHub](0182-taskcluster-yml-remote-references.md) |
| RFC#189 | [Batch APIs for task definition, status and index path](0189-batch-task-apis.md) |
| RFC#190 | [Queue change task run priority](0190-queue-change-task-run-priority.md) |