Big Raster Prediction

[NikolaCenic]

I am trying to do inference(using predict_tile) on rasters of different sizes, that sometimes can not fit in the available RAM memory. Until the last release, the predict_tile was throwing an exception when this happens. In those cases I split the raster in N windows and perform inference on each of them. After the latest release this exception is not thrown anymore, and the process is getting killed. Do you have a suggestion of a workaround/solution for this issue?

[bw4sz]

Can you paste me some code so I can consider what is happening here. Its not obvious to me how any code we changed would change this behavior. If you give me a tile that used to throw the error, I will install versions going backwards until I find the commit, but my initial sense that this is a dependency issue, and not a deepforest source, since we don't to my knowledge ever check if tiles fit in memory.

At the same time, I'm not sure I ever want predict_tile to throw a out of memory error. That feels like a bug in of itself, let's also try to address that root error if we can reproduce it. The entire point of predict_tile is to cut tiles into manageable pieces, so why would you then need to do anything else. Maybe you are not using the patch_size argument properly, or there is something we can do to make that better (as a percentage of the tile size for example). Let's investigate together.

[ethanwhite]

I agree that the change from the exception to the crash is likely upstream, but I think the core issue is on our on end. I looked at the code yesterday and ran a quick test and it looked like (at least for small files) that we were using 2x file size in RAM (because we first loaded the entire file and then made a dataset out of it and stored it in a new variable). Unless rasterio is doing something fancy under the surface at large file sizes I think we probably need a re-write to handle on-disk manipulation for > memory scale tiles. We haven't encountered see this since most sources that get large are tiled.

[bw4sz]

okay, let's write an issue to memory profile the predict_tile method with the goal of reducing memory consumption.

[ethanwhite]

Oh, and @NikolaCenic - the quick fix is to cut the large raster into smaller pieces. One way to do this is using GDAL and there are instructions here if you're interested:
https://wiki.weecology.org/docs/computers-and-programming/geospatial-computing/#splitting-rasters-using-gdal

[bw4sz]

#830