Integrated alert testing tile cache URL (1a)

[k-kresek]

This is the testing URL for the GFW Integrated Deforestation Alert layer, which combines the GLAD-S, GLAD and RADD alert layers.

Tile cache URL: https://dev-tiles.globalforestwatch.org/gfw_integrated_alerts/latest/default/{z}/{x}/{y}.png
Note that's a link to a dev environment tile cache, it's not in production yet.

[k-kresek]

Here are some data set specific notes to be aware of:

While the Red, Green, and Blue channels are the same as for RADD/GLADL/GLADS2, we are also making use of the alpha channel now. The front end needed to know if the alert had been detected by multiple systems, so we provide that, with an important caveat: It's slightly fake. Here the explanation I put in the code that generates the alpha channel:

# The front end is ALSO expecting the confidences of all the alerts in the
# original alert systems packed into an 8-bit value as the Alpha channel
# according to the following scheme:
# Alpha = (gladl_conf << 6) | (glads2_conf << 4) | (radd_conf << 2)
#
# Where gladl_conf, glads2_conf, and radd_conf are 2 for high confidence,
# 1 for low confidence, and 0 for not detected.
# Thus, the last 2 bits are currently unused.
#
# But it doesn't really NEED all that, all it REALLY needs to know is if
# the combined alert is low, high, or highest confidence. So we slimmed-
# down and saved just that fact by encoding it in the raster tile set as a 40k+ alert value.
# However rather than change the encoding as presented to the front end
# we create a fake combined confidence value. We accomplish that by
# setting RADD's confidence to 1 for low conf, 2 for high conf, and
# doing a bitwise OR with 4 (indicating a low confidence alert in GLAD-S2)
alpha = "((1 * (A.data >= 20000) + 1 * (A.data >= 30000)) | (4 * (A.data >= 40000))) << 2"

[davidlstarr]

prod tile url: https://tiles.globalforestwatch.org/gfw_integrated_alerts/latest/default/{z}/{x}/{y}.png

[davidlstarr]

@k-kresek I setup a new Dataset and Layer but it looks like I need the metadata for the Integrated alert layer. For example, tree cover uses https://production-api.globalforestwatch.org/v1/gfw-metadata/tree_cover. Whenever you get a chance is this something you could please setup for me?

[k-kresek]

Yes! We will need to connect to the integrated alerts metadata using the metadata key - I will check and see what this is.

[k-kresek]

For integrated alerts, we should the 'gfw_integrated_alerts' metadata key.

[davidlstarr]

@k-kresek Would you happen to have an example of what the integrated layer should look like in the map? This is what I'm seeing on the map when I bring the integrated alert layer in, is this correct?

?

[k-kresek]

Here is a pre-production version of the integrated alerts: https://preproduction.globalforestwatch.org/map/?map=eyJjZW50ZXIiOnsibGF0IjotMTYuMDc4MjQ1MzE4MDQzODQsImxuZyI6LTQwLjU1ODU5Mzc0OTk5ODY5fSwiZGF0YXNldHMiOlt7ImRhdGFzZXQiOiJpbnRlZ3JhdGVkLWRlZm9yZXN0YXRpb24tYWxlcnRzLThiaXQiLCJvcGFjaXR5IjoxLCJ2aXNpYmlsaXR5Ijp0cnVlLCJsYXllcnMiOlsiaW50ZWdyYXRlZC1kZWZvcmVzdGF0aW9uLWFsZXJ0cy04Yml0Il19LHsiZGF0YXNldCI6InBvbGl0aWNhbC1ib3VuZGFyaWVzIiwibGF5ZXJzIjpbImRpc3B1dGVkLXBvbGl0aWNhbC1ib3VuZGFyaWVzIiwicG9saXRpY2FsLWJvdW5kYXJpZXMiXSwib3BhY2l0eSI6MSwidmlzaWJpbGl0eSI6dHJ1ZX1dfQ%3D%3D

There are several different filters that we would like to maintain, such as the date slider, the ability to show select alert systems, and the confidence identification.

[k-kresek]

Here is some additional encoding info from Daniel:

d = days since 2014/12/31
c = confidence level (high confidence = 1 or low confidence = 0)
i = intensity value (0 - 55)

R: floor (d / 255)
G: d % 255
B: ((c + 1) * 100) + i

To extract date, confidence and intensity

d = R * 255 + G
c = floor(B/100) - 1
i = B % 100

[dmannarino]

Here's the link I posted in the chat, to Vizzuality's decode JS: https://github.com/Vizzuality/gfw/blob/develop/providers/datasets-provider/config.js#L225-L286
Briefly, you want to pull the date (relative to 2014-DEC-31) out of the Red and Green channels, the intensity out of the Blue, and the combined confidence out of the Alpha (the confidence value in Blue is obsoleted by the combined confidence in the Alpha channel). Then set the RGB values of the result pixel according to whether the combined confidence is low, high, or highest, and the Alpha (opacity) to the intensity * 50 (max 255). Oh, and possibly apply a filter based on the decoded date (remember it's relative) or desired confidence.

[davidlstarr]

@k-kresek I have been working through this layer and have run into an issue with the tiles coming across. The tiles for https://tiles.globalforestwatch.org/gfw_integrated_alerts/latest/default/{z}/{x}/{y}.png are coming across blank. Since we extract rgb values from the canvas and interpolate them on the front-end, I'm unable to generate the correct outputs such as date, confidence level and intensity. Here is an example of one of the tiles, https://tiles.globalforestwatch.org/gfw_integrated_alerts/v20220205/default/3/3/3.png. Tiles that should be populated with rgb values are showing up as blank. For example, for the glad alerts the tiles are coming back with values. Here is an example of one of those tiles, https://tiles.globalforestwatch.org/glad_prod/tiles/3/2/4.png.

[dmannarino]

@davidlstarr Hi David, I think I can solve one mystery for you: The tiles aren't actually BLANK, they're just LIGHT. If you look really closely you can see the shadow of an image there. It would appear darker but we're using the Alpha channel (interpreted as opacity) to store data, and the values are mostly small. This means if you look at the images in an image viewer expecting a standard PNG it will appear highly transparent (/light). But the Red, Green, and Blue channels have data which can be decoded.
The GLAD tiles appear at full opacity, BTW, because we didn't need to use the Alpha channel (and it defaults to full opacity).

[davidlstarr]

@dmannarino I see what you are saying but I’m not seeing any red values come across on my end. This doesn’t make sense to me. The glad alerts do not act that way and there are red values that come across.

I manually set the red value just so I could see a color come across on the tile and these are the results...I used https://tiles.globalforestwatch.org/gfw_integrated_alerts/v20220205/default/3/3/3.png as the test tile. There are no red values coming across for that tile.

This is what shows in the map, if I manually set the red value

Every single r value coming across is 0....something is def off. I'm just not sure what it is...

Is there a JS developer on your team that could help with the matter?

[davidlstarr]

@dmannarino Also, I manually set the alpha value to full opacity and left the other values as-is (rgb) and I'm still not seeing anything come across onto the canvas. The only way I'm seeing graphics in the map is by setting the red values manually.

[dmannarino]

@davidlstarr Hi David, sorry for the delay in replying.
Hmm, something's not right if you set the alpha and still get a blank canvas. As you can see in your output there should be plenty of green/blue pixels displaying. As to why you're consistently getting 0s for the red channel, I'm not sure. It IS possible with the data (because red is part of the date component), but I found plenty of pixels with (small) red values. I opened the tile in question in Python (can share the code if interested) and printed out the first 50 pixels with any non-zero value and got the following:
Sample pixels:
(9, 124, 200, 8)
(9, 228, 100, 4)
(6, 177, 200, 8)
(6, 99, 200, 8)
(9, 246, 100, 4)
(9, 246, 100, 4)
(4, 177, 200, 8)
(9, 134, 100, 4)
(4, 81, 200, 8)
(4, 202, 200, 8)
(5, 171, 100, 4)
(9, 191, 100, 4)
(9, 143, 100, 4)
(9, 175, 100, 4)
(9, 223, 100, 4)
(8, 78, 200, 8)
(4, 122, 200, 8)
(9, 143, 100, 4)
(4, 81, 200, 8)
(4, 81, 200, 8)
(7, 125, 200, 8)
(7, 142, 200, 8)
(4, 195, 200, 8)
(6, 37, 200, 8)
(4, 123, 200, 8)
(5, 100, 200, 8)
(5, 34, 200, 8)
(6, 147, 200, 8)
(9, 198, 100, 4)
(8, 174, 200, 8)
(9, 161, 100, 4)
(6, 214, 200, 8)
(6, 206, 200, 8)
(8, 64, 200, 8)
(8, 40, 200, 8)
(6, 102, 200, 8)
(4, 139, 200, 8)
(9, 229, 100, 4)
(9, 248, 100, 4)
(9, 161, 100, 4)
(9, 169, 100, 4)
(9, 185, 100, 4)
(9, 161, 100, 4)
(9, 161, 100, 4)
(8, 41, 200, 8)
(8, 1, 200, 8)
(4, 149, 200, 8)
(9, 236, 100, 4)
(9, 236, 100, 4)
(9, 236, 100, 4)

[dmannarino]

Here's what I get when I display the tile without the Alpha channel.

[davidlstarr]

@dmannarino When you get a chance could you provide us with the code you used to spit out the rgba values for https://tiles.globalforestwatch.org/gfw_integrated_alerts/v20220205/default/3/3/3.png? I appreciate it!

[dmannarino]

@dmannarino When you get a chance could you provide us with the code you used to spit out the rgba values for https://tiles.globalforestwatch.org/gfw_integrated_alerts/v20220205/default/3/3/3.png? I appreciate it!

This should do it. Let me know if you have any problems with it:

# Say you want to sample some raster tile cache tiles (which
# are relatively small PNGs).
# To run this you'll need to `pip install pillow requests`
import requests
from PIL import Image


for alert_uri in [
    # GLAD
    # "https://tiles.globalforestwatch.org/glad_prod/tiles/3/3/3.png",
    # Integrated alerts:
    "https://tiles.globalforestwatch.org/gfw_integrated_alerts/latest/default/3/3/3.png",
]:
    print(f"URI: {alert_uri}")
    response_data = requests.get(alert_uri, stream=True).raw
    image_obj = Image.open(response_data)

    pixel_list = list(image_obj.getdata())

    non_zero = [pixel for pixel in pixel_list if pixel[3] != 0]
    print(f"Number of alert pixels: {len(non_zero)}")

    print("Sample alert pixels:")
    for pixel in non_zero[:50]:
        print(f"{pixel}")

    # Display without the alpha channel to make it easier to see/compare
    # integrated alerts with the individual alerts systems
    # image_without_alpha = Image.new("RGBA", (256, 256))
    image_without_alpha = Image.new("RGB", (256, 256))
    image_without_alpha.putdata(pixel_list)
    image_without_alpha.show()

[davidlstarr]

I was able to successfully bring in the correct rgba pixels, filter out the specified date range and confidence levels. I'm currently working on the layer selector, timeslider, date range picker, and confidence toggles.

Here is the current test link to see my progress: https://alpha.blueraster.io/gfw-mapbuilder/pull-requests/rw-layers-fix/

[davidlstarr]

@k-kresek When you get an opportunity could you please test out the new functionality for the gfw layers on https://alpha.blueraster.io/gfw-mapbuilder/pull-requests/integrated-alert-layer/? Thanks!

[k-kresek]

Hey David! Thank you so much for your work on this so far! Here are a couple of initial comments I have:

1. There is a blue outline around the integrated deforestation alerts - when you zoom out certain areas actually start to look blue or purple. This is a bit visually confusing because there is no purple in the legend. Do you think we could remove the outline or match the color of the interior of the pixels?

2. On the time slider, it is hard to identify the date in the middle - could we make it larger like the end dates?

3. There is something going on with the transparency sliders for these layers. It looks like just the integrated alerts transparency slider works but only with some of the alerts (see screenshot). The other layers don't seem to have transparency enabled.

4. This is super small, but is there a way to make the dates all on one line? I just think this looks better since the other two dates fit on one line.

Let me know what you think!

Thanks,
Kai

[davidlstarr]

@k-kresek I completed 3/4 items. I wasn't quite sure on item 2. Any further info would be helpful. Thanks!

[k-kresek]

Hey David! Thank you, I will test! For item 2, I was thinking that the midpoint circle should be the same size as the two endpoint circles so that people can navigate the slider a bit easier. I kind of see now that the slider circle moves with the slider and isn't necessarily always marking the two endpoints. I think it's alright to leave it the way it is - however now all the dates that are on the bottom of the slider are 3-21-2022 which doesn't match what the top slider dates are when you switch the dates.

So, I think we can leave the circle slider style the same, but fix the dates that are shown on the bottom so they match the date shown when you hover over that part of the slider.

[davidlstarr]

I'm not seeing that issue you stated on my end. Could you provide a screenshot?

This is what I see...

[k-kresek]

Oooh! I just saw that it is different years. I think in that case it is alright, we can leave it as is.