Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix FilterFS when skipping lazy parent directories #183

Merged
merged 2 commits into from
Feb 21, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions filter.go
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,7 @@ func (fs *filterFS) Walk(ctx context.Context, target string, fn gofs.WalkDirFunc
includeMatchInfo patternmatcher.MatchInfo
excludeMatchInfo patternmatcher.MatchInfo
calledFn bool
skipFn bool
}

// used only for include/exclude handling
Expand Down Expand Up @@ -333,6 +334,9 @@ func (fs *filterFS) Walk(ctx context.Context, target string, fn gofs.WalkDirFunc
}
}
for i, parentDir := range parentDirs {
if parentDir.skipFn {
return filepath.SkipDir
}
if parentDir.calledFn {
continue
}
Expand All @@ -352,15 +356,21 @@ func (fs *filterFS) Walk(ctx context.Context, target string, fn gofs.WalkDirFunc
}
if fs.mapFn != nil {
result := fs.mapFn(parentStat.Path, parentStat)
if result == MapResultSkipDir || result == MapResultExclude {
if result == MapResultExclude {
continue
} else if result == MapResultSkipDir {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: don't need else if after a continue.

parentDirs[i].skipFn = true
return filepath.SkipDir
}
}

if err := fn(parentStat.Path, &DirEntryInfo{Stat: parentStat}, nil); err != nil {
parentDirs[i].calledFn = true
if err := fn(parentStat.Path, &DirEntryInfo{Stat: parentStat}, nil); err == filepath.SkipDir {
parentDirs[i].skipFn = true
return filepath.SkipDir
} else if err != nil {
return err
}
parentDirs[i].calledFn = true
}
if err := fn(stat.Path, &DirEntryInfo{Stat: stat}, nil); err != nil {
return err
Expand Down
51 changes: 51 additions & 0 deletions filter_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,30 @@ file bar/foo
`), b.String())
}

func TestWalkerIncludeReturnSkipDir(t *testing.T) {
d, err := tmpDir(changeStream([]string{
"ADD foo dir",
"ADD foo/x dir",
"ADD foo/y dir",
"ADD foo/x/a.txt file",
"ADD foo/y/b.txt file",
}))
assert.NoError(t, err)
defer os.RemoveAll(d)

found := []string{}

err = Walk(context.Background(), d, &FilterOpt{
IncludePatterns: []string{"**/*.txt"},
}, func(path string, info gofs.FileInfo, err error) error {
found = append(found, path)
return filepath.SkipDir
})
assert.NoError(t, err)

assert.Equal(t, []string{"foo"}, found)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would this return anything at all. "foo" does not match the IncludePatterns.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume because the Walk recurses into foo to see if anything matches the IncludePatterns...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Walk finds a file that matches the include patterns, then starts calling Walk on parent directories, and the first Walk callback func returns SkipDir...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly that, yup.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand how this makes sense. IncludePatterns is an option to Walk. Callback should not be called on paths that don't match IncludePatterns.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's not included there then it isn't hooked up to Send/Recv - which means that we don't send the file stats for parent directories.

e.g. foo as this parent should have it's permissions/etc transferred through these methods. Potentially Send shouldn't use Walk at all, but that seems out of scope for this IMO, as it's a larger refactor and architectural change (not really convinced by it either).

}

func TestWalkerExclude(t *testing.T) {
d, err := tmpDir(changeStream([]string{
"ADD bar file",
Expand Down Expand Up @@ -308,6 +332,33 @@ file includeDir/a.txt
assert.Equal(t, []string{"excludeDir", "includeDir", filepath.FromSlash("includeDir/a.txt")}, walked)
}

func TestWalkerMapSkipDirWithPattern(t *testing.T) {
d, err := tmpDir(changeStream([]string{
"ADD x dir",
"ADD x/a.txt file",
"ADD y dir",
"ADD y/b.txt file",
}))
assert.NoError(t, err)
defer os.RemoveAll(d)

b := &bytes.Buffer{}
err = Walk(context.Background(), d, &FilterOpt{
IncludePatterns: []string{"**/*.txt"},
Map: func(_ string, s *types.Stat) MapResult {
if filepath.Base(s.Path) == "x" {
return MapResultSkipDir
}
return MapResultKeep
},
}, bufWalk(b))
assert.NoError(t, err)

assert.Equal(t, filepath.FromSlash(`dir y
file y/b.txt
`), b.String())
}

func TestWalkerPermissionDenied(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("os.Chmod not fully supported on Windows")
Expand Down
Loading