Run Tasks from the catalog without downloading and applying them

[ghost]

Hello!

I'm starting to explore ways for folks to use tasks without kubectl applying them first and I wanted to open a discussion about it.

To get straight to the fun part, I'm imagining something like the following syntax to run the golang-test task from the catalog (without applying it to your cluster!):

kind: Pipeline
tasks:
- name: test
  taskRef:
    apiVersion: gitref.tekton.dev/v1alpha1
    kind: Task
    bundle: https://github.com/tektoncd/catalog.git@main
    name: task/golang-test/0.1/golang-test.yaml

At the moment this ^ does not validate in a PipelineTask but does validate in a Run, which makes it interesting to try locally but not practically useful.

I've created a proof-of-concept Custom Task that fetches tasks from the catalog using a slightly off-brand syntax that does pass Pipelines' validation:

  tasks:
    - name: unit-test
      taskRef:
        apiVersion: catalogtask.tekton.dev/v1alpha1
        kind: Task
        name: golang-test--0.1
      params:
        - name: package
          value: github.com/tektoncd/pipeline
        - name: context
          value: ""
      workspaces:
        - name: source
          workspace: shared

This will fetch the golang-test task (v0.1) from the catalog and run it. The task from the catalog is injected as the taskSpec field of a TaskRun.

This is o.k. - unsurprisingly much quicker to get up and running because no need to download individual tasks - but what I don't particularly like about it right now is that the Custom Task is responsible for submitting a TaskRun. It might make more sense to instead have it return the Task (or any other runtime.Object) to the Pipelines controller for it to process. The Pipelines Controller might decide, for example, to send it to Results to be recorded as a historical artifact.

So I think we should consider defining an interface that any Custom Task can meet in order to be treated as a "TaskResolver" by Pipelines. This could involve opening up the bundle and name fields to support this as I suggested at the top or could use a completely different syntax because... why not.

I imagine a TaskResolver interface for custom tasks that is similar to our existing remote resolver interface.

I see a lot of benefits to this approach but the idea's still new in my mind so I'm probably wearing rose-colored glasses. Here are the immediate pros I can think of:

• Remote task resolution no longer a responsibility of the pipelines controller
• Operators control which remote task resolvers are supported in their clusters
• Adds remote resolution to Pipelines core without having to make hard decisions up-front about required formats, supported registries, best practices, caching, configuration:
  • Because it's pluggable: support whichever catalog / VCS / cloud storage / hard-drive-under-a-desk you store Tasks in
  • Because it supports per-remote configuration separate from the pipelines controller
    • In my GitTaskRef deployment I might want to configure which repos are allowed.
    • In my GCSTaskRef deployment I might want to configure it with a specific bucket name so that all refs of the form "task-name" are resolved to that single bucket.
  • Because caching requirements can differ per-provider. You might want to cache git repos with tasks in but have no problem re-fetching tasks from GCS every time a new Run is created.
  • Because credentials for resolving tasks can be managed alongside that resolver's deployment
• A pipelines release could come with an OCIImageRef and CatalogTaskRef implementation built-in so catalog tasks in those two formats are immediately fetchable when the controller starts up

So what are folks' initial reactions here? For additional context, we've already got several different approaches to remote task resolution in play:

• First, obviously, users can fetch a remote task themselves and apply it

• The built-in remote OCI resolver in pipelines

  The problem I see with this right now is that the resolvers are baked into Pipelines' core. This prevents operators from using any storage they want for their org's Tasks. We can add configuration and more resolver types to the project over time but I am questioning how far that maintenance will need to stretch.

• The uses syntax being proposed as part of the Step Reuse TEP

  The issues that I see here are, again, the resolver implementations are baked into Pipelines' core. And in this TEP's case it's tackling both remote resource resolution as well as add brand new concepts / features (Step Reuse). I wonder if we might be able to pull the remote resolution part out and make it more generally useful while we dig into the composition questions in other threads?

Further Extensions

• Taking this even further - what if we codified some TaskResolver interfaces as part of our conformance? Thinking extremely common ones like gitref.tekton.dev/v1alpha1.Task, catalogref.tekton.dev/v1alpha1.Task, ociref.tekton.dev/v1alpha1.Task - so any conforming platform can provide an implementation of these.
• What about a localref.tekton.dev/v1alpha1.Task that reads from your local disk during development on a new pipeline / task?

[afrittoli]

Thank for starting this, I like the idea, definitely worth exploring further!

Remote task resolution no longer a responsibility of the pipelines controller

This might help with the current issue with remote resolvers (OCI) that is impacts the duration of the reconcile cycle.
When do you envision resolution would happen?

If you have a large pipeline doing remove resolution upfront may add a significant amount of time before a pipeline is started. On the flip side, doing lazy resolution of tasks implies delayed validation - we could be running through a pipeline that will never succeed because the interface of the last task does not match the pipeline task.

Having the remote task resolver as a pluggable, separate process might help on this too. I can imagine a couple of optional implementations where the controller asks for a batch resolution of all tasks but holds the reconcile for a short time for an answer (enough to get the first task).

1. The task resolver supports caching

• if all tasks are returned immediately (perhaps they were cached by the task resolver), validation can happen immediately
• as soon as one task is returned execution can start
• a parallel process in the controller continues to receive the task definitions
• once they are all in, if validation fails, the parallel process marks the pipeline as failed (or cancels it)
• next reconcile cycle the controller asks again for batch resolution of all the tasks, more (or all) will be cached now

2. The controller writes resolved pipeline tasks into the pipeline status, and runs them from there

• if all tasks are returned immediately (perhaps they were cached by the task resolver), validation can happen immediately
• as soon as one task is returned execution can start
• a parallel process in the controller continues to receive the task definitions and adds them to the pipelinerun status, which triggers reconcile and thus validation. If validation fails the pipeline is marked as failed (or cancelled)
• next reconcile cycle the controller asks for batch resolution of all the missing tasks (could be none)

Another question I have is about pipeline reusability, and at which level we would allow the resolver to be specified:

• on each pipeline task
• at pipeline level
• at pipeline run level
• at pipeline run level, for specific pipeline tasks

Limiting the configuration at authoring time, would make the behaviour of a pipeline much more predictable and easier to verify / approve. Allowing runtime configuration would allow for more reusability.
Allowing runtime to override author time might be a good trade off too.

If security is a concern, one could ship the pipeline with a sha calculated on all fetched tasks.
At runtime the task resolver could be configured to a local cache or provider, and the pipeline would be executed only if the sha on the task from the runtime provider matches.

[ghost]

This might help with the current issue with remote resolvers (OCI) that is impacts the duration of the reconcile cycle.
When do you envision resolution would happen?

If you have a large pipeline doing remove resolution upfront may add a significant amount of time before a pipeline is started. On the flip side, doing lazy resolution of tasks implies delayed validation - we could be running through a pipeline that will never succeed because the interface of the last task does not match the pipeline task.

Yeah, good points! In the proof of concepts I've been working on the fetch + validation happens after the Run is created, so it's lazy. But after being fetched once the tasks are cached on an in-memory emptyDir volume in the resolver so that resolution is very quick.

In the case of a "catalog resolver" the entire catalog can be cloned to an in-memory volume when the resolver first starts up, so every resolution is quick. For a resolver with broader scope - like a general "git resolver" that can fetch a task from any repo - the caching would need to be smarter. That resolver might employ an LRU cache for example instead to avoid exhausting the available space.

Having the remote task resolver as a pluggable, separate process might help on this too. I can imagine a couple of optional implementations where the controller asks for a batch resolution of all tasks but holds the reconcile for a short time for an answer (enough to get the first task).

That's a good idea, I will give this a try! Having resolution in a separate deployment generally motivates the protocol design to be asynchronous and so dissuades blocking individual reconciles / threads in the pipelines controller. Even if a single Pipeline's reconcile is waiting for remote task resolution, other pipelines are able to continue to reconcile just fine.

I think there's some configuration that pipelines controller will need too - maybe a separate, much smaller timeout for remote task resolution.

Another question I have is about pipeline reusability, and at which level we would allow the resolver to be specified:

• on each pipeline task
• at pipeline level
• at pipeline run level
• at pipeline run level, for specific pipeline tasks

Limiting the configuration at authoring time, would make the behaviour of a pipeline much more predictable and easier to verify / approve. Allowing runtime configuration would allow for more reusability.
Allowing runtime to override author time might be a good trade off too.

I'm generally of the perspective that resolvers will definitely be configurable at the org-level: whoever's managing the cluster chooses which resolvers are installed with their pipelines installation. Then resolvers themselves are each individually configurable - catalog resolver configured to point only at the open source catalog, git resolver configured to only allow fetches from specific repos or hosting platforms, oci resolver configured with a single internal company registry, etc. So each resolver maybe has allow lists and deny lists to configure the domains / repos that are allowed by teams using that cluster.

When it comes to how resolution can be used in the composition of pipelines, I think this should be lead from the composition side. Whatever composition patterns we decide to add - Pipelines in Pipelines, Pipelines in a Pod, Tasks in a Task - should be well supported by the task resolution system we build.

[afrittoli]

Slightly related TEP: tektoncd/community#352

[ghost]

In commit https://github.com/sbwsg/pipeline/commit/a571869336ba09732f830743fbee938f4d40ac4d I've modified the PipelineRun reconciler to observe a protocol similar to the one I described at the top of this discussion. It uses Custom Tasks to perform the Task YAML resolution and then the PipelineRun reconciler is responsible for submitting the TaskRun to the cluster.

Here's a snippet of the example pipeline's YAML which uses remote task resolution:

  tasks:

  - name: welcome
    taskRef:
      apiVersion: ociref.tekton.dev/v1alpha1
      kind: Task
      bundle: kind-registry:5000/tasks/print-welcome:0.1
      name: print-welcome

  - name: clone
    runAfter: [welcome]
    taskRef:
      apiVersion: catalogref.tekton.dev/v1alpha1
      kind: Task
      name: git-clone
    workspaces:
    - name: output
      workspace: shared
    params:
    - name: url
      value: https://github.com/tektoncd/results.git

Here ^ we can see a task being fetched from a Tekton Bundle (uses tkn bundle pull under the covers) and another task being fetched from the open source tekton catalog.

Each remote resolver (oci, git, gcs, catalog) is its own Custom Task controller. I've put them all together in a single commit here: https://github.com/sbwsg/experimental/commit/1140f236ec0ffaf529835e913139eafa847608a8

Testing this is a bit annoying - you need to install both the patched pipelines controller and the custom task controllers (+ some secrets and a local oci registry where applicable). Since it's annoying to get running right now I'll update this comment with a link to the demo I did in the API Working Group this week when that video is available.

Edit to add: here's the link to the video from the API WG - demo starts around 11:09. In order to view you'll need to be a member of the tektondev google group (more info here).

There are a considerable number of both pros and cons to this approach (and I definitely haven't considered everything) so my next step here is to propose a problem-statement TEP that discusses this and some other possible implementations.