ResponseParser return type for github_graphql next version

[vvshulga]

What should be the correct returned object in ResponseParser in

incubator-devlake/backend/plugins/github_graphql/tasks/pr_collector.go

Line 177 in c316619

return nil, nil

if I want to change some data in response?
I added Files param to get information about changed paths and files, and if ResponseParser want to process this paths and save just unique paths for each PR.

[dosubot[bot]]

Hey there, @vvshulga! I'm here to help you with any bugs, questions, or contributions while you wait for a human maintainer. Let me know how I can assist you!

To modify the ResponseParser function to process and save unique file paths for each pull request, you would need to adjust the function to parse the response data and extract the file paths. Then, you can store these paths in a suitable data structure, such as a map or set, to ensure uniqueness. Finally, you would return this processed data in the desired format.

Currently, the ResponseParser function returns a slice of json.RawMessage ^[1]. To handle unique file paths, you might need to:

1. Parse the response to extract file paths.
2. Use a map or set to filter out duplicate paths.
3. Convert the unique paths into the desired return format.

Here's a conceptual example of how you might implement this:

ResponseParser: func(res *http.Response) ([]json.RawMessage, errors.Error) {
    var responseData struct {
        Files []struct {
            Path string `json:"path"`
        } `json:"files"`
    }
    err := helper.UnmarshalResponse(res, &responseData)
    if err != nil {
        return nil, err
    }

    uniquePaths := make(map[string]struct{})
    for _, file := range responseData.Files {
        uniquePaths[file.Path] = struct{}{}
    }

    var uniquePathsList []json.RawMessage
    for path := range uniquePaths {
        uniquePathsList = append(uniquePathsList, json.RawMessage(`"`+path+`"`))
    }

    return uniquePathsList, nil
},

This example assumes the response includes a files field with a path for each file. Adjust the structure according to the actual response format.

To continue talking to Dosu, mention @dosu.

[vvshulga]

@dosu in version that I'm using return type is ([]interface{}, error) and I can not update my devlake GitHub graphql plugin in the nearest time.

[dosubot[bot]]

To modify the ResponseParser function to handle unique file paths and return a ([]interface{}, error) type, you can adapt the function as follows:

1. Parse the response to extract file paths.
2. Use a map or set to filter out duplicate paths.
3. Convert the unique paths into a slice of interface{}.

Here's an example implementation:

ResponseParser: func(res *http.Response) ([]interface{}, errors.Error) {
    var responseData struct {
        Files []struct {
            Path string `json:"path"`
        } `json:"files"`
    }
    err := helper.UnmarshalResponse(res, &responseData)
    if err != nil {
        return nil, err
    }

    uniquePaths := make(map[string]struct{})
    for _, file := range responseData.Files {
        uniquePaths[file.Path] = struct{}{}
    }

    var uniquePathsList []interface{}
    for path := range uniquePaths {
        uniquePathsList = append(uniquePathsList, path)
    }

    return uniquePathsList, nil
},

This code assumes the response includes a files field with a path for each file. Adjust the structure according to the actual response format you are working with.

To continue talking to Dosu, mention @dosu.

[vvshulga]

@dosu I'm using graphql version of ResponseParser that is used then here - https://github.com/apache/incubator-devlake/blob/next/backend/helpers/pluginhelper/api/graphql_collector.go

[dosubot[bot]]

For the GraphQL version of the ResponseParser function used in graphql_collector.go, you can adapt the function to handle unique file paths by following these steps:

1. Parse the GraphQL response to extract the file paths.
2. Use a map or set to ensure the paths are unique.
3. Convert the unique paths into a slice of json.RawMessage or interface{}, depending on your requirements.

Here's an example implementation:

ResponseParser: func(queryWrapper interface{}) ([]interface{}, errors.Error) {
    query := queryWrapper.(*YourGraphqlQueryType) // Replace with your actual query type
    uniquePaths := make(map[string]struct{})

    for _, node := range query.Nodes {
        for _, file := range node.Files {
            uniquePaths[file.Path] = struct{}{}
        }
    }

    var uniquePathsList []interface{}
    for path := range uniquePaths {
        uniquePathsList = append(uniquePathsList, path)
    }

    return uniquePathsList, nil
},

Make sure to replace YourGraphqlQueryType and the structure of the loop with the actual types and fields from your GraphQL response. This example assumes that the response includes a Nodes field with a Files field containing Path for each file. Adjust the structure according to your actual response format.

To continue talking to Dosu, mention @dosu.

[vvshulga]

@dosu Method signature is following:
ResponseParser: func(iQuery interface{}, variables map[string]interface{}) ([]interface{}, error) {
result should be an array of struct that has RawDataOrigin and is some db table/Model. Looks like it should be RawData, but it's unclear.