Removals aren't actioned until environment is redeploy

[bassonj]

When there is a file removal and g10k deploys the environment
The removed file is still present until the environment is purged and re-deployed.

[xorpaul]

Can you give me an example so that I can reproduce it?

Do you meant a file is removed from the control repository by hand?

[bassonj]

We make use of a mix of forge modules and locally hosted git modules.

How to reproduce:
Create a new git feature branch in one of the locally hosted git repos.
Do a git rm <filename>
Commit and push that change.
Merge the feature branch into a (pre-production) branch in your workflow (testing/staging/canary)

Have g10k deploy the change to the environment via the command
/usr/local/bin/g10k -verbose -maxworker 50 -maxextractworker 20 -config /etc/g10k.yaml -branch <environment name>

After the command has completed the removed file is still on disk on the puppet server for that environment in the basedir

If I remove the environment and redeploy it, (either by deleting the environment directory or passing the -force flag) the expected behaviour happens, meaning the removed file is not re-deployed

Below is the contents of our /etc/g10k.yaml

---
# Puppet Controlled #
# DO NOT MODIFY     #
#
:cachedir: '/etc/puppetlabs/puppet/environments/g10k_cache'
use_cache_fallback : false

# A list of git repositories to create
sources:
    local:
        remote  : [email protected]:puppet/control
        basedir : /etc/puppetlabs/puppet/environments

[xorpaul]

I can't reproduce your bug.

I've created a control repository branch called single_remove and added an empty foobar file into it: https://github.com/xorpaul/g10k-environment/tree/189b8f9e6f44658f2b8d3939a60467a3f08b72a1

Then I forked a branch of this called single_remove_branch and did a git rm foobar and pushed this branch. (xorpaul/g10k-environment@8720074)

After that I merged single_remove_branch into single_remove (also xorpaul/g10k-environment@8720074)

Then I synced the control repository branch single_remove again:

$ g10k -config test.yaml -environment example_single_remove
Need to sync /tmp/example/example_single_remove
Removing unmanaged path /tmp/example/example_single_remove/foobar

As you can see the merged removal of the file foobar was removed.

Are you using the most recent version of g10k? https://github.com/xorpaul/g10k/releases

[xorpaul]

Ah, I've found it.

This is not a bug, if you want g10k to purge your control repository content from unmanaged files you need to set the purge_level to at least contain 'environment'.

See r10k documentation, which I'm trying to emulate: https://github.com/puppetlabs/r10k/blob/master/doc/dynamic-environments/configuration.mkd#purge_levels

The default value is ['deployment', 'puppetfile']

I'm setting these purge_level for my test g10k config file, that's why I couldn't reproduce your bug previously: https://github.com/xorpaul/g10k/blob/master/test.yaml#L5-L6

[xorpaul]

I've verified that r10k does exactly the same.

Without the 'environment' in the purge_level it leaves unmanaged files in the control repository untouched.

[bassonj]

Thank you very much, I'll adjust our configs appropriately

[bassonj]

Hi, in our setup we run g10k with the following flags
-quiet -maxworker 50 -maxextractworker 20 -config /etc/g10k.yaml -branch staging
which then checks out the specified branch which correlates to a Puppet environment into /etc/puppetlabs/puppet/environments

ie. /etc/puppetlabs/puppet/environments/staging

Based on the code I see here it looks like the cleanup only works when making use of the -environment flag?

Would it be possible to adjust the code to also purge unmanaged content when making use of the -branch flag?

[xorpaul]

Nope, it doesn't matter if you use the -environment or the -branch parameter for the unmanaged content to be purged:

$ touch /tmp/example/example_single_remove/foobar
$ g10k -config test.yaml -branch single_remove -info
Removing unmanaged path /tmp/example/example_remove
Synced test.yaml with branch remove with 0 git repositories and 0 Forge modules in 0.5s with git (0.0s sync, I/O 0.0s) and Forge (0.0s query+download, I/O 0.0s) using 50 resolve and 20 extract workers

Maybe you think that if you're missing the Removing unmanaged path line in your output the file also doesn't get removed, but this output only happens when you're using -info.

The file also does get deleted without -info or with -quiet:

$ touch /tmp/example/example_single_remove/foobar
$ g10k -config test.yaml -branch single_remove -quiet
$ ls -l /tmp/example/example_single_remove/foobar
ls: cannot access '/tmp/example/example_single_remove/foobar': No such file or directory

[walkleyn]

@xorpaul thanks for all the help thus far. We're still struggling to get the desired outcome.

The prescribed behaviour works for us on modules that don't exist in the puppetfile, but not on files inside modules/repos that are cloned when updating ie.
Bad module/file added to module path:

$ touch /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/badmodule
$ /usr/local/bin/g10k -info -config /etc/g10k.yaml  -branch WK_g10_cleanup
Removing unmanaged path /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/badmodule
Synced /etc/g10k.yaml with branch WK_g10_cleanup with 45 git repositories and 84 Forge modules in 5.2s with git (1.2s sync, I/O 0.0s) and Forge (0.1s query+download, I/O 0.0s) using 50 resolve and 20 extract workers

File removed/untracked from module repository - simulated:

$ touch /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/ssh/badmodulefile
$ /usr/local/bin/g10k -info -config /etc/g10k.yaml  -branch WK_g10_cleanup
Synced /etc/g10k.yaml with branch WK_g10_cleanup with 45 git repositories and 84 Forge modules in 5.6s with git (1.2s sync, I/O 0.0s) and Forge (0.1s query+download, I/O 0.0s) using 50 resolve and 20 extract workers

The latter example is more of a concern for us since we want the contents of the cloned repos to not contain untracked files.

[xorpaul]

How did a file inside a Puppetfile managed module get modified/created outside of r10k/g10k?

Currently g10k and r10k do not check the content of a module that was not updated by r10k/g10k in the current run:

$ touch /tmp/example/example_single_remove/external_modules/inifile/foobar
$ r10k deploy environment -c ./test.yaml example_single_remove -p
$ ls -l /tmp/example/example_single_remove/external_modules/inifile/foobar
-rw-rw-r-- 1 andpaul andpaul 0 Dez  4 16:34 /tmp/example/example_single_remove/external_modules/inifile/foobar

[walkleyn]

I did that just to illustrate the point, let me illustrate with the actual use case:
The workflow we are trying to get to is that files that are removed from modules that get deployed by g10k get removed from the code base.

I just ran the following test to illustrate again:
In my own git based puppet module repository (called hieradata as an example) and created on the puppetserver by g10k under /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/hieradata
I perform the following steps in my local checked out repo:

$ touch fqdn/test_removal.yaml (and commit)
$ git push 

On the puppetserver:
run g10k and check that the file exists:

$ ll /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/hieradata/fqdn/test_removal.yaml 
-rw-rw-r-- 1 root root 0 Dec  5 10:41 /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/hieradata/fqdn/test_removal.yaml

Great it's there, g10k made sure that the new file is present in the codebase

Now the removal of the file in the module repo
In my local checked out repo:

$ git rm  fqdn/test_removal.yaml (and commit)
$ git push

On the puppetserver:
Run g10k again, but the file is not removed

$ ll /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/hieradata/fqdn/test_removal.yaml 
-rw-rw-r-- 1 root root 0 Dec  5 10:41 /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/hieradata/fqdn/test_removal.yaml

The only way to ensure that it is removed, is by either removing it behind g10k's back or using force to remove all environments and recreate them. Neither of these seem like feasible options.

[xorpaul]

I can't reproduce your bug.

When I'm adding such a file in my test module here: https://github.com/xorpaul/g10k_testmodule/tree/update_master
and also added a fqdn/test_removal.yaml xorpaul/g10k_testmodule@51aa7f6

Then deployed the environment via g10k:

$ ./g10k -config test.yaml -environment example_single_own_modules -info
Need to sync /tmp/example/example_single_own_modules/modules/test
Synced test.yaml with 2 git repositories and 0 Forge modules in 1.2s with git (0.8s sync, I/O 0.0s) and Forge (0.0s query+download, I/O 0.0s) using 50 resolve and 20 extract workers
$ ll /tmp/example/example_single_own_modules/modules/test/hieradata/
total 4,0K
drwxr-xr-x 2 andpaul andpaul 4,0K Dez  5 13:53 fqdn/
$ ll /tmp/example/example_single_own_modules/modules/test/hieradata/fqdn/
total 0
-rw-rw-r-- 1 andpaul andpaul 0 Dez  5 13:53 test_removal.yaml

Then I removed the test file again:

$ git rm fqdn/test_removal.yaml ; git ci -m "remove fake hieradata" ; git push
rm 'hieradata/fqdn/test_removal.yaml'

and triggered the g10k run:

$ ./g10k -config test.yaml -environment example_single_own_modules -info
Need to sync /tmp/example/example_single_own_modules/modules/test
Removing unmanaged path /tmp/example/example_single_own_modules/modules/test/hieradata/fqdn
Removing unmanaged path /tmp/example/example_single_own_modules/modules/test/hieradata/fqdn/test_removal.yaml
Synced test.yaml with 2 git repositories and 0 Forge modules in 1.3s with git (0.7s sync, I/O 0.0s) and Forge (0.0s query+download, I/O 0.0s) using 50 resolve and 20 extract workers
$ ls -l /tmp/example/example_single_own_modules/modules/test/hieradata/fqdn/
ls: cannot access '/tmp/example/example_single_own_modules/modules/test/hieradata/fqdn/': No such file or directory

Can you check your g10k yaml config again if you're missing the 'puppetfile' in purge_level?
https://github.com/puppetlabs/r10k/blob/master/doc/dynamic-environments/configuration.mkd#purge_levels

[xorpaul]

@walkleyn or @bassonj Did I miss anything that you do in your workflow, so that I maybe can reproduce your bug?

Can you show me your g10k config yaml file, with your git URL hidden?

[walkamongus]

I'm running into this issue currently as well -- I'm not yet able to get cleanup of removed controlrepo files using v0.8.8.

My config looks like this:

:cachedir: '/tmp/g10k'
use_cache_fallback : true

deploy:
  purge_levels: [ 'deployment', 'environment', 'puppetfile' ]

sources:
    puppet:
        remote  : ssh://[email protected]/controlrepo.git
        basedir : /etc/puppetlabs/code/staging/environments
        exit_if_unreachable : true
        invalid_branches : correct

[walkamongus]

interestingly -- I just noticed my master branch is getting cleaned up, however my other branch that has its name corrected is not -- possibly issue with purging and corrected branch names?

[xorpaul]

This could be a possibility that I'm missing the autocorrected Puppet environment names.

Although I don't know if the issue creator is also having his branch name corrected as well.

I'll have a look at this later

[mdrobnak]

Something to note here: If you had the purge settings incorrect and then updated them, you've missed the boat.
The purge settings must be correct on the commit you are removing content. If it's not, it will never be noticed as unmanaged.

IE: I rolled my 'production' branch back to before a large delete as 'delete_test' - with purging turned on as I want. It did the right thing and generated types stayed around, etc.

Then I merged in production, which then deleted a bunch of files. g10k removed those files.
However, in the local copy of production, they still exist , because g10k is not aware they disappeared. (The "environment" option to purge_levels was added to my config after the change was already deployed live)

I haven't looked at the code recently, so I'm not sure what the correct fix to this is.