Kedro command to profile projects

[astrojuanlu]

I'd like to see us add a CLI command which users can run to produce a flamegraph. It would massively reduce the guesswork here.

kedro profile {kedro command} -> .html / .bin

Originally posted by @datajoely in #3893 (comment)

In my mind, it would run the whole command as normal, but also generate the profiling data.

Perhaps if we were to take this seriously, a full on memray integration would incredible.

Originally posted by @datajoely in #3893 (comment)

I think this is a promising idea. Challenges:

• What format to output. Definitely an option to provide unstructured (HTML) as well as structured (whatever Spedscope supports https://github.com/jlfwong/speedscope?tab=readme-ov-file#supported-file-formats)
• What kinds of profiling to support
  • Runtime
    • Line profiling
    • Function profiling
    • ?
  • Memory
    • Memory profiling (memory-profiler, memray)
  • Disk usage
  • ?

[datajoely]

So I'm not sure what @ankatiyar used for the recent performance analysis #3866 , #1476 but that library would be a good start. Perhaps I can best frame some outcomes as user stories:

• As a Kedro user I want to understand why my pipeline is taking so long to run so that I can make the necessary tweaks to improve things.
• As a Kedro maintainer I want to ask the user a provide the results of a profiling operation so that I can help diagnose issues without the risk of sharing any sensitive data or incurring the overhead of making a minimum reproducible example.

In the first story you probably need a UI, in the second a binary file format that could be added to an issue / slack post etc would be sufficient.

[ankatiyar]

For the record, I used pyinstrument with visualisation in speedscope.app

While working on #1476, I was also thinking that we could build a profiler for Kedro pipelines as a plugin (kedro-profiler?), there's useful hooks such as before_node_run and after_node_run for us to use.

[datajoely]

For QB folks this is what one internal team built for runtime performance