RFC: Simple pipeline merging and templating

[pms1969]

No description provided.

[cppforlife]

@pms1969 just to throw in an alternative solution... @cdutra and i recently were discussing how one might use ytt to help manage complexity of concourse pipelines. given that one can define functions in ytt it becomes pretty easy to decompose and reuse jobs, tasks, etc. this i think nicely keeps concourse out of business of templating and allows other tools to do what they do best. (ytt template -R -f . | fly -t prod set-pipeline --pipeline example --config -)

[pms1969]

@cppforlife ytt looks interesting. I'll have a deeper look some time soon.

[samrees]

I would not be a big fan of concourse bringing in the templating hell of helm-esque systems.
Maybe useful to mention this for anyone else seeking composability at a pipeline level: https://cuelang.org/docs/about/

[ringods]

Why not drop any type of declarative language and adopt the solution Pulumi brings to Infrastructure as Code? You could even adopt Pulumi by implementing a resource provider against the currently unofficial REST api of Concourse.

[vito]

Hi @pms1969 - sorry it took so long to get to this. We recently adopted a more fluid and transparent process for RFC review, so we're catching up on the backlog of RFCs. Thanks for putting this together!

My gut reaction is to stay out of the business of templating, since a) no one can agree on what templating system to use because everyone uses something different, and b) the need for templating is to me an indicator that there may be something missing in Concourse's design. IMO templating is a last resort; introducing it as a first-class feature before exhausting the possibilities for core improvements may introduce a local maxima that increases the cognitive overhead for users while reducing critical feedback loops that allow us to improve Concourse further.

In addition, I think a few of the items on the v10 roadmap address at least some of the underlying needs:

• RFC: set_pipeline step #31 allows you to codify your templating + configuration flow; a preceding step can render the pipeline YAML before passing it to set_pipeline step
  • this won't help with fly, but the overall goal is to be using fly less and have your Concourse self-configure from a bootstrap pipeline (or project with RFC: Projects #32)
• RFC: Projects #32 allows you to split pipeline definitions up and share things like resource definitions between pipelines in the project's file tree
• RFC: across step #29 + RFC: set_pipeline step #31 allows you to dynamically configure pipelines across either dynamic or static var matrixes which may be an alternative to templating multiple job definitions
  • RFC: Projects #32 also aims to allow passed constraints to span pipelines, though it doesn't cover that in great detail at the moment (TODO added)
• RFC: Prototypes #37 addresses #7 a bit more directly; instead of reusable tasks you'd just write prototypes, as you would with resource types

Given all that, I'm going to propose that we postpone this until more of the v10 roadmap is delivered. Per the RFC process this doesn't mean it's final say - it'll bump this RFC to the top of the table on concourse-ci.org to get more eyes on it, so it's a chance to get more feedback and for anyone to challenge the resolution.