Using terragrunt scaffold with GitLab URLs

[gmaghera]

Describe the bug
Scaffold does not seem to work with GitLab URLs

To Reproduce

remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/rivian/dc.git/' not found
WARN[0000] Failed to find last release tag for git::https://gitlab.com/rivian/dc.git
INFO[0000] Scaffolding a new Terragrunt module git::https://gitlab.com/rivian/dc.git//platform/terraform-modules/s3.git to /Users/gmaghera/Workspaces/dc-terraform/tools/us-east-2/scaffold-sandbox
ERRO[0002] error downloading 'https://gitlab.com/rivian/dc.git': /usr/local/bin/git exited with 128: Cloning into '/var/folders/1q/b7d7p2g91fddxq6jwt237cdm6xcxmy/T/getter2517239202/temp'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/rivian/dc.git/' not found
ERRO[0002] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

or

❯ terragrunt scaffold https://gitlab.com/rivian/dc/platform//terraform-modules//s3
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/rivian/dc.git/' not found
WARN[0000] Failed to find last release tag for git::https://gitlab.com/rivian/dc.git
INFO[0000] Scaffolding a new Terragrunt module git::https://gitlab.com/rivian/dc.git//platform/terraform-modules/s3 to /Users/gmaghera/Workspaces/dc-terraform/tools/us-east-2/scaffold-sandbox
ERRO[0002] error downloading 'https://gitlab.com/rivian/dc.git': /usr/local/bin/git exited with 128: Cloning into '/var/folders/1q/b7d7p2g91fddxq6jwt237cdm6xcxmy/T/getter1330873649/temp'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/rivian/dc.git/' not found
ERRO[0002] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

not applicable

Expected behavior
Please provide an example for how to do this against a GitLab repository.

Nice to have

• Terminal output
• Screenshots

Versions

• Terragrunt version: 0.55.20
• Terraform version: 1.5.5
• Environment details (Ubuntu 20.04, Windows 10, etc.): Darwin 23.3.0, macOS 14.3.1
  Additional context
  Add any other context about the problem here.

[denis256]

Hi,
git ssh URLs are also not handled correctly?

[denis256]

I suspect wasn't used .git URL to clone repo
In my tests for https://gitlab.com/denis256/terragrunt-tests I used:

$ terragrunt scaffold https://gitlab.com/denis256/terragrunt-tests.git//scaffold/default-template
INFO[0000] Scaffolding a new Terragrunt module git::https://gitlab.com/denis256/terragrunt-tests.git//scaffold/default-template?ref=v10.0.0 to /tmp/test1 
INFO[0004] Running boilerplate generation to /tmp/test1 
[boilerplate] 2024/03/28 20:43:07 Loading boilerplate config from /tmp/boilerplate967725454/boilerplate.yml
[boilerplate] 2024/03/28 20:43:07 Loading boilerplate config from /tmp/boilerplate967725454/boilerplate.yml
INFO[0004] Scaffolding completed

$ terragrunt scaffold gitlab.com/denis256/terragrunt-tests.git//scaffold/default-template
INFO[0000] Scaffolding a new Terragrunt module git::https://gitlab.com/denis256/terragrunt-tests.git//scaffold/default-template?ref=v10.0.0 to /tmp/test1 
INFO[0004] Running boilerplate generation to /tmp/test1 
INFO[0004] /tmp/test1/terragrunt.hcl was updated        
INFO[0004] Scaffolding completed    

[lorengordon]

Using .git on the end of the clone url is something of a github thing. It's not strictly necessary, and for example, codecommit doesn't use/support it at all.

[gmaghera]

Hi @denis256, thanks for looking into this. If you look at my original quote, I did try without the .git piece terragrunt scaffold https://gitlab.com/rivian/dc/platform//terraform-modules//s3 failed.

The error suggests to me that there is an assumption about directory level. Your example apparently uses a personal GitLab project which is always two levels deep: gitlab.com/<username>/<projectname> -- and that works. Compare that to the error I am getting:

❯ terragrunt scaffold https://gitlab.com/rivian/dc/platform//terraform-modules//s3
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/rivian/dc.git/' not found
WARN[0000] Failed to find last release tag for git::https://gitlab.com/rivian/dc.git
INFO[0000] Scaffolding a new Terragrunt module git::https://gitlab.com/rivian/dc.git//platform/terraform-modules/s3 to /Users/gmaghera/Workspaces/dc-terraform/tools/us-east-2/scaffold-sandbox
ERRO[0002] error downloading 'https://gitlab.com/rivian/dc.git': /usr/local/bin/git exited with 128: Cloning into '/var/folders/1q/b7d7p2g91fddxq6jwt237cdm6xcxmy/T/getter1330873649/temp'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/rivian/dc.git/' not found
ERRO[0002] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

The project path is https://gitlab.com/rivian/dc/platform//terraform-modules/s3,
but the error say this fatal: repository 'https://gitlab.com/rivian/dc.git/' not found.
"dc" is a group not a git project, and you gotta go two levels deeper before the s3 project is found.

(and I see that I used the // notation twice, by accident, but I get the same exact error with just one or none, too.)

[gmaghera]

@lorengordon I am not sure about GitHub, but with GitLab if you go into the web UI both SSH and HTTPS URLs end in .git. But they do work without it too.

[gmaghera]

Hi, git ssh URLs are also not handled correctly?

Apparently not. Or I'm not using the call signature correctly...

~/Workspaces/dc-terraform/tools/us-east-2/scaffold-sandbox experiment-with-scaffold* 1h 36s
(⎈|dev-us-east-1:kube-system)❯ terragrunt scaffold [email protected]:rivian/dc/platform/terraform-modules/s3.git
INFO[0002] Scaffolding a new Terragrunt module git::ssh://[email protected]/rivian/dc/platform/terraform-modules/s3.git?ref=4.2.0 to /Users/gmaghera/Workspaces/dc-terraform/tools/us-east-2/scaffold-sandbox
ERRO[0002] error downloading 'ssh://[email protected]/rivian/dc/platform/terraform-modules/s3.git?ref=4.2.0': /usr/local/bin/git exited with 128: fatal: not a git repository (or any of the parent directories): .git
ERRO[0002] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

~/Workspaces/dc-terraform/tools/us-east-2/scaffold-sandbox experiment-with-scaffold*
(⎈|dev-us-east-1:kube-system)❯ terragrunt scaffold [email protected]:rivian/dc/platform/terraform-modules/s3
INFO[0001] Scaffolding a new Terragrunt module git::ssh://[email protected]/rivian/dc/platform/terraform-modules/s3?ref=4.2.0 to /Users/gmaghera/Workspaces/dc-terraform/tools/us-east-2/scaffold-sandbox
ERRO[0001] error downloading 'ssh://[email protected]/rivian/dc/platform/terraform-modules/s3?ref=4.2.0': /usr/local/bin/git exited with 128: fatal: not a git repository (or any of the parent directories): .git
ERRO[0001] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

[denis256]

Hi,
yes, there is an issue... happens in my tests too, after some debugging looks like it is an issue in go-getter library which is used to pull content, something similar happens in go-getter CLI too

$ go-getter "github.com/denis256/s3.git//module2" /tmp/qwe
2024/04/01 18:01:56 success!

$ go-getter "gitlab.com/denis25-tests/dc/platform/terraform-modules/s3.git//module2" /tmp/qwe
2024/04/01 18:02:19 Error downloading: error downloading 'https://gitlab.com/denis25-tests/dc.git': /usr/bin/git exited with 128: Cloning into '/tmp/getter72444115/temp'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/denis25-tests/dc.git/' not found

Will raise an issue in the library GH project

[denis256]

hashicorp/go-getter#479

[gmaghera]

Thanks for the update @denis256.

[gmaghera]

@denis256 could you remove the "awaiting response" label? Is there something else I can provide?

[gmaghera]

@denis256 my colleague @tvizor is looking into addressing the issue with go-getter, but I think there is some ambiguity to this.

Considering this example from https://blog.gruntwork.io/introducing-terragrunt-catalog-and-scaffold-aead126dcf10:

terragrunt scaffold \
  github.com/gruntwork-io/terragrunt-infrastructure-modules-example//modules/s3-bucket

The repo URL in this case is github.com/gruntwork-io/terragrunt-infrastructure-modules-example and the module is under the subpath modules/s3-bucket.

That may be clear in the context of Terragrunt, where // means download everything under that directory, but it may not mean anything to Hashicorp's go-getter. I assume that the above example would also work without the double slash.

For a GitLab URL which can nest groups up to 20 levels deep, this presents a problem. Does Terragrunt equate //with the end of the (GitLab) repo URL and tries to use go-getter to retrieve only the part up to it? For handling both repo-only URLs and repo+subpath URLs, I think the usage of // would become mandatory.

[denis256]

Hi,
I think we need to use the approach when a double slash is the separator of remaining path, similar to how it is for Terraform modules

https://developer.hashicorp.com/terraform/language/modules/sources#modules-in-package-sub-directories

FYI: @brikis98

[josh-padnick]

@yhakbar @denis256 Any chance we can make a final call on this so we can keep things moving?

[gmaghera]

Hi @josh-padnick.

My colleague, @tvizor, has submitted a PR for addressing hashicorp/go-getter#479, opened by @denis256.

Could we work together on resolving this? Scaffold is a game-changer, in my view. I'd love to see it work (more robustly) with GitLab.

[yhakbar]

Hello! Sorry this hasn't been moving along at the rate you would prefer, @gmaghera .

Is my understanding right that the only issue here is with the underlying go-getter implementation for GitLab repositories nested under a group? If so, how can we help to resolve this?

I believe we would need your pull request to be merged by the maintainers of go-getter to have your repositories supported, right?

[gmaghera]

Hi @yhakbar -- yes, if that PR gets merged, and we get a new version of Terragrunt which includes that code, that should solve the issue.

It's not just our repositories, in its current shape go-getter and scaffold only support a fraction of GitLab. There may be a gap with testing non-free GitLab repos which can contain nested groups in their URL. And there is a request to add the same feature to GitHub... See https://github.com/orgs/community/discussions/4837

[gmaghera]

Go-getter seems abandoned, perhaps there is some lull due to the IBM/Hashicorp thing...

@yhakbar could we explore some alternate solutions? We could spin a release from the fork, or perhaps some other way to get this working. Anything we can do to get scaffold functional with GitLab Enterprise...

[lorengordon]

Does it work in terraform natively? If not, and you can easily reproduce the issue, might be worth opening an issue there. At this point, the go-getter project is primary only updated to support use cases from terraform...

[gmaghera]

Does it work in terraform natively? If not, and you can easily reproduce the issue, might be worth opening an issue there. At this point, the go-getter project is primary only updated to support use cases from terraform...

Yes, the issue is not specific to Terragrunt at all. @denis256 has already opened an issue for it, which is linked to the PR which would fix it. See #3031 (comment)

[lorengordon]

Does it work in terraform natively? If not, and you can easily reproduce the issue, might be worth opening an issue there. At this point, the go-getter project is primary only updated to support use cases from terraform...

Yes, the issue is not specific to Terragrunt at all. @denis256 has already opened an issue for it, which is linked to the PR which would fix it. See #3031 (comment)

That issue is on go-getter, not on terraform. Like I said, if the issue is not opened against terraform and confirmed there to impact terraform, no maintainer on go-getter is going to address it.

[gmaghera]

True but by that token, maybe Terragrunt should be looking for a replacement.

[gmaghera]

I've tried to find an example in Terraform core where the go-getter issue could be reproduced, per @lorengordon's suggestion.

Here's the rough example, a main.tf file with this content:

module "gitlab_nested_group_example" {
  source = "git::https://gitlab.com/foo/bar/fred/terraform-modules/null.git//?ref=0.1.5"
  desired_count = 1
}

That works just fine, even though we know go-getter cannot handle such paths. The Terraform code base must be using something else to pull down modules hosted on gitlab.com. denis256 would it be possible to use the same path with Terragrunt?

Is there perhaps a better example I can try to reproduce? I cannot think off the bet of something analogous to terragrunt scaffold in signature, where a module URL is passed as a command argument.

[gmaghera]

@lorengordon apparently, someone did find a manifestation of the issue in Terraform! I'm hoping this results in traction.

See hashicorp/terraform#34771 (comment)

[gmaghera]

Hello again.

Gruntwork, I would like to kindly ask that you consider pursuing a direct solution to this. Relying on getting a fix via Hashicorp's go-getter does not seem like a viable option. We have created an issue there, detailed the line of code where the issue stems from, placed a PR with the fix, yet there is no traction after months of waiting.

Scaffold is such a powerful feature! It bridges a gap in making Terragrunt accessible to engineers who may not have much experience with it, and it is also great for encouraging best practices. In its current form, however, the feature is broken for all GitLab enterprise customers.

CC: @josh-padnick @denis256 @yhakbar

[josh-padnick]

In its current form, however, the feature is broken for all GitLab enterprise customers.

@yhakbar @denis256 @levkohimins This alone makes it worth considering a hacky workaround until hashicorp/go-getter#479 is merged (which could well be never).

But in addition, @gmaghera has really gone above and beyond to track down (and even attempt to resolve) this problem here. Can we address his underlying issue so he can begin using terragrunt scaffold with GitLab Enterprise?

[denis256]

I can take a look to add handling of Gitlab cases

[yhakbar]

Before you dig into this, @denis256 , could I get confirmation on a couple things to verify that we do, indeed need to workaround go-getter, @gmaghera ?

I know we already asked about this, and you already provided a response, so I'm sorry that I didn't ask clarifying questions about this early on. I didn't have a GitLab account, so I didn't check, but I just made one to confirm that SSH URLs don't work for nested paths:

$ go-getter '[email protected]:gitlab-org/ai-powered/eli5.git//eli5' eli5
2024/09/03 13:33:21 success!

It seems based on that example that go-getter v2 does in fact work, and we might just need to update the version used in the catalog command to a version that supports your use-case.

Do you mind running the following:

go install github.com/hashicorp/go-getter/cmd/go-getter@latest

Then trying out the standalone go-getter binary to see if there's still a need to workaround the usage of go-getter?

If all we need to do is upgrade the version of go-getter we're using, that will be a better solution than introducing a workaround for GitLab nested projects.

[gmaghera]

@yhakbar I will check if it works today.

But, looking at your test, that does not appear to be a nested repo, or not nested enough. You need at least three "layers" before the //.

For example gitlab.com/group1/group2/group3/repo.git//foobar. Go-getter wrongly assesses that group2 is the repository, or at least it used to.

I'll reply with my test results next.

[gmaghera]

@yhakbar can you please share how you've installed v2 of go-getter? Running go install github.com/hashicorp/go-getter/cmd/go-getter@latest installs v1.7.6. I've played with various @references to get it to install v2, but every attempt so far had failed.

[yhakbar]

Sorry, I may have been mistaken that I was using v2.

To be extra super confident that we're trying out v2 appropriately, please do the following:

$ git clone [email protected]:hashicorp/go-getter.git
$ cd go-getter
$ git checkout v2
$ cd cmd/go-getter
$ go install .

In my testing on an open source repo I was able to find that was more deeply nested, it continued to work:

$ go-getter [email protected]:gitlab-org/ai-powered/custom-models/pocs/gitlab-docs-indexer-poc.git poc
2024/09/04 14:02:06 -> poc

It's been a while since I used GitLab, so please let me know if I'm still missing a nuance here on the level of nesting for repos.

[gmaghera]

Using go-getter v2 does work via the SSH protocol (but not via HTTPS).

Thank you for the suggestion, @yhakbar.

As long as it works with one protocol, that would be good enough for my needs.

[gmaghera]

Hello @josh-padnick @denis256 @yhakbar.

Just wanted to check in, and make sure you're not waiting on me. Do we have a path forward on this? I've run the tests recommended by Yousif, go-getter v2 does fix the issue (for SSH protocol only, but that's sufficient for me). Do you mean to address this with #3381?

Thank you!

[yhakbar]

Hey @gmaghera ,

I think it might be incidentally addressed via #3381, but I don't know if it will.

To be transparent, the team doesn't have a ton of bandwidth right now, and maximizing GitLab compatibility for scaffold isn't a priority right now.

The fact that there's some viable approach for GitLab developers to use Scaffold makes me feel like this can be addressed by a member of the community if they would like the improved ergonomics of not specifying the schema for the git URL in GitLab.

I'd like to leave this issue open, though, to make it clear that this is something that can be addressed.

[gmaghera]

I am a bit confused. Scaffold is broken for all GitLab Enterprise users. I appreciate the bandwidth restrictions, we all have to deal with it.

If there is a viable approach, I'd be more than happy to use it. However asking that we restructure where we keep Terraform modules in GitLab is not viable. It is also a typical setup within GitLab Enterprise.

There has been so much back and forth about this. @josh-padnick asked @denis256 to look into a fix via #3031 (comment), but then @yhakbar you asked me to look into whether that could be addressed via v2 of go-getter. I did all that, and it looked promising, and now I am being told that may not be a good path forward.

I realize what I am asking for, and this being an OSS project, and you having priorities (we used to be a paying customer of Gruntwork), but the communication around this has been contradictory.

[yhakbar]

Ah, sorry @gmaghera , my mistake.

My impression was that you tried using scaffold using a GitLab SSH URL and found that it worked. I didn't realize that you were saying that it was specifically using the go-getter binary standalone with an SSH URL that worked.

Your comment regarding #3031 fixing things makes more sense to me now, as I thought you were asking if that would fix all GitLab URLs. I agree that it doesn't look like there's a path forward for you to use scaffold without us upgrading go-getter, to v2.

I'll sync up with the team and see if we can prioritize a fix for this next week.

[yhakbar]

Hey @gmaghera ,

Would you mind testing out v0.68.0-beta2024100701 to see if upgrading go-getter to v2 results in GitLab support?

I don't have a convenient way to test this on my own, as I don't control a valid publicly available TF module in GitLab to scaffold.

[gmaghera]

Oh, I really, really appreciate this! I will test and get back to you.

[gmaghera]

@yhakbar it does work, but with a slight qualifier. I was about to report it not working after a number of tries, but then I stumbled upon a module URL usage that is a bit peculiar. This behavior is consistent between GitLab and GitHub. There seems to be an issue with scaffolding Terraform module repositories which host the HCL files at the root (the style the TF registry uses).

What works with GitLab:
../terragrunt_beta scaffold [email protected]:rivian/dc/platform/terraform-modules/s3//.

ℹ️ Note the extra '.' after the double slash.

Without that piece, I get an error. But that is not specific to GitLab. It can be reproduced with GitHub:

❯ ../terragrunt_beta scaffold [email protected]:terraform-aws-modules/terraform-aws-s3-bucket.git//
12:25:04.827 WARN   The working directory . is not empty.
12:25:06.474 INFO   Scaffolding a new Terragrunt module git::ssh://[email protected]/terraform-aws-modules/terraform-aws-s3-bucket.git?ref=v4.2.0 to .
12:25:06.619 ERROR  /opt/homebrew/bin/git exited with 128: fatal: not a git repository (or any of the parent directories): .git

12:25:06.619 ERROR  Unable to determine underlying exit code, so Terragrunt will exit with error code 1

When I look at the docs, the examples show modules that are subdirectories of Git repos, such as terragrunt scaffold github.com/gruntwork-io/terragrunt-infrastructure-modules-example//modules/mysql.

Is there a recommended way to address repositories that don't use subdirectories for module code?

[gmaghera]

After doing that I had a sinking feeling that I had to verify... and there was a surprise find! 🤦

As long as I add that extra period after the double slash, scaffold works with v0.67.10 of terragrunt too, with GitLab URLs containing subgroups!!!

[yhakbar]

Thanks for testing this out, @gmaghera !

It looks like it's related to this change, as we noticed some abnormalities with go-getter usage at that point.

Will file a bug report for us to address this in the future. For now, it seems like you're unblocked on using scaffold in GitLab, and we can comfortably de-prioritize this issue. Please feel free to point out if I'm making an invalid assumption here.

I'll mark the Scaffold v2 upgrade as ready for review, as it seems to work just the same as it does in the latest release for GitLab.

I'll close this issue out, as GitLab users aren't any more blocked on scaffold usage. It's a go-getter double-slash issue that's causing problems.

[gmaghera]

I'm unblocked indeed, will keep an eye out for future developments.

Thank you @yhakbar, @josh-padnick, @denis256 for all the collaboration on this!