Poor Performance on NAIP Imagery - Worse than just wrangling NDVI thresholding

[zackaryleady]

Hello,

I tested this on multiple tiles of NAIP imagery by removing the NIR band on 60 cm pixel size (higher res NAIP).

Even with using a high overlap and many different patch_sizes the model performance is basically worse than just using NDVI. If I use a very small patchsize=64 then I atleast get a large number of trees where atleast a small box is found near the canopy centroid, but the entire tree or multiple connected tree crowns won't really be detected and dozens of false negatives on water pixels. IF there is an area in the tile with a large density of trees and another part of the tile has low density trees it just doesn't find the low density ones using a larger patchsize. Anything from 256-1024 really doesn't find hardly any trees even though there are hundreds-thousands of trees to find. I did check and see that you are / 255 to normalize uint8 which is what NAIP is. Also you permute the bands to BGR internally so I didn't do that beforehand, but I did make it channels last.

I used this code:

def test_forest():
  model = main.deepforest()
  model.use_release()  
  with rasterio.open(r"C:\Users\zacka\Desktop\Relabel_20241007\T2\naip_w1641n1402.tif") as src:
  #with rasterio.open(r"C:\Users\zacka\Desktop\Relabel_20241007\T1\naip_w1693n1346.tif") as src:
      input_array = src.read([1, 2, 3]) # RGB

  # channels last
  input_array = input_array.transpose((1,2,0))

  #raster_path = get_data(r"C:\Users\zacka\Desktop\Relabel_20241007\T1\naip_w1693n1346.tif")
  #print(raster_path.shape)
  # Window size of 300px with an overlap of 25% among windows for this small tile.
  predicted_raster = model.predict_tile(image=input_array, iou_threshold=0.95, return_plot=True, patch_size=512 ,patch_overlap=0.90)

  # View boxes overlayed when return_plot=True, when False, boxes are returned.
  plt.imshow(predicted_raster)
  plt.show()
if __name__ == "__main__":
    test_forest()

512 patchsize / 0.9 overlap (tile is 1666x1666 pixels)

64 patchsize / 0.5 overlap (tile is 1666x1666 pixels)

Did I miss something internally or with a different function that significantly improves performance?
I'm using mamba and version 1.3.3 of deepforest. It basically seems that NDVI, with a low threshold, then derive centroids from polygons/bounding boxes, then provide to SAM2 via samgeo, would actually outperform the performance I have seen so far. Any feedback on what I am doing wrong would be greatly appreciated.

[ethanwhite]

Hi @zackaryleady - welcome and thanks for the questions! We haven't done much work on data at this coarse a resolution ourselves, but others have seen good results with NAIP data. They perform two of our recommended approaches to improve on the out-of-the-box predictions, so let me start by asking:

1. Have you tried fine tuning using some hand labeled imagery for your specific sensor/region? This is the best and easiest way to get the model to behave well with data that differs from what it has been trained on: https://deepforest.readthedocs.io/en/latest/user_guide/11_training.html
2. Have you tried more systematic hyperparameter tuning? Unfortunately the above paper doesn't list the hyperparameters they found to work well on NAIP but I think a more systematic exploration of patch size (including smaller patch sizes) and iou_threshold (which you have set to be higher than we've ever used) should also help based on the paper linked above.

If you still aren't seeing decent performance at that point we can be of more help if you can provide a sample image for us to play with.

[jonathanventura]

Hi, as @ethanwhite mentioned we did get good results by fine-tuning DeepForest on our NAIP data. Our data only has point annotations so we simply put a fixed size bounding box around each point and found the best box side length to be 9.6 m (16 pixels).

Since our dataset consists of 256x256 NAIP crops, we used predict_image instead of predict_tile and so we did not make use of or tune the iou_thresh or patch_size parameters. Fine-tuning the model had the largest effect, but we did see a further boost in performance by tuning the score_thresh parameter -- our best value was 0.302.

Note that our dataset covers urban areas and so the trees are generally more sparsely laid out.

[zackaryleady]

Hello Jonathan,

Thank you for your reply. Would you be able to save the model weights checkpoint on your fine-tuned model? OR did I miss it on your github?

~Zack

[ethanwhite]

any ETA when the NAIP dataset from WRI will be made available? Or do they have it somewhere already hosted? A google search didn't provide a direct link.

Spring 2025 is our ETA at the moment. A lot of the data we're compiling isn't openly available yet. Folks have been sending us data with the understanding that it will become open, but we still have to work through all of the details with individual contributors related to licensing, hosting, authorship, etc. Both @bw4sz and I are super over-committed at the moment, but it's possible that when things slow down a bit in December/January that we could try to run a fine tune for you on that data and send you the weights. Feel free to ping us in December to see if that's a possibility.

[jonathanventura]

@zackaryleady I put the DeepForest fine-tune checkpoint file here on OneDrive -- let me know if you have any issues with it.

[bw4sz]

Thanks @jonathanventura, I continue to appreciate your commitment to open data and helping out.