Skip/Omit empty tiles?

[spatialsparks]

Hi QTiles team
Your plugin works great, thanks for the work! I wondered if there is any way to skip empty tiles? I am generating heatmaps out of millions of points and then exporting the heatmaps as tiles using your plugin. Now, due to the rather clustered nature of the data, alot of tiles are empty/blank. This means that at high zoom levels, most of the tiles will be empty/blank. With your plugin, I can set transparency so that these tiles dont show up on my webmap, but it would save a large amount of storage space to just skip these tiles entirely, is something like this possible?
Best regards

[simgislab]

thinking out loud, wouldn't it be easier to delete them after creation after sorting by size

[spatialsparks]

Hm, I guess that would not be all that clever because: imagine zoom level 12 for example where 80% of the tiles are nodata. For the world this would mean generating 13421773 tiles (16'777'216 tiles * 0.8 ;; http://wiki.openstreetmap.org/wiki/Zoom_levels ) which store no data, meaning that you are generating 53 GB (13'421'773 tiles * 0.004mb per empty tile). That seems quite alot of storage read/write for nothing.

Another important part of deleting AFTER creation would be that one would have to be absolutely sure of the filesize. It looks like the generated file is a PNG where every pixel has a value of rgba 0,0,0,0. But what if a tile has rgba values of 1,2,3,4 for every pixel (meaning the tile might be a uniform tile in a larger polygon or something where no edges are contained in the tile. The tile still has spatial information)?

Maybe I got something wrong here, just my ithought aswell :)

[simgislab]

you are right of course

We have a portion of code proposed that checks before rendering if there is any data at all in the tile and not to render the tile if there isn't. It sounds like it can solve your point 1, but not 2.

And it still waits to be integrated...

[spatialsparks]

That sounds great! Unluckily, I have no experience in QGIS plugin development, but might look into it, if I find time. Skipping empty tiles would already save alot of storage and especially render time, so implementing this and having an option to activate/deactive rendering of empty tiles would be an enormous performance boost I guess.

I can imagine this plugin iterates over a given raster or extent and interpolates the values to the current zoom level? Then one could check if the area of the original input which is needed to generate the 256x256 pixels of 1 tile sums up to "no_data". If so, you can skip the area and store the boundingbox of the area in a hashmap or something similar because if a zoomlevel 2 tile is "no_data" then all of the 4 zoomlevel 3 tiles within said tile will also be "no_data" and so on (all 16 tiles of zoomlevel 4, all 64 tiles of zoomlevel 5... ). So storing the highest zoomlevel bounding box of no_data values could greatly increase performance again I presume?

Again, just my two cents, because as mentioned, I have no expereince with QGIS plugin development :)

Ps. looking forward to your development approaches.

[simgislab]

consider sponsoring or pull-requests, this is the only sure way to move it forward

[luis-puhl]

@JoranBeaufort I've been working with some tiles and mounted a script to remove the empty tiles and change it's background color to transparent if they have more than one.
This does not fix the time wasted rendering the tiles but saves the disk (and file list). Working with road tiles this script cut my tile count from 1 500 000 to 250 000.

https://gist.github.com/luis-puhl/e31b74c80aa12478aa7c0d3690b18a56

But as you said, it does not know if a tile is empty (one color) because of a real feature (e.g. a lake) or not.

[srctaha]

To eliminate empty/blank tiles, I have modified DirectoryWriter class defined in ~/.qgis2/python/plugins/qtiles/writers.py as follows:

import numpy as np

class DirectoryWriter:
    def __init__(self, outputPath, rootDir):
        self.output = outputPath
        self.rootDir = rootDir
        
    def get_sumr(self, image):
        w = image.width()
        h = image.height()
        c = 4 if image.hasAlphaChannel() else 3
        s = image.bits().asstring(h*w*c)
        arr = np.fromstring(s, dtype=np.uint8).reshape((h,w,c)) 
        return arr.sum()
        
    def writeTile(self, tile, image, format, quality):
        path = '%s/%s/%s' % (self.rootDir, tile.z, tile.x)
        dirPath = '%s/%s' % (self.output.absoluteFilePath(), path)
        if self.get_sumr(image) > 0:
            QDir().mkpath(dirPath)
            image.save('%s/%s.%s' % (dirPath, tile.y, format.lower()), format, quality)
        
    def finalize(self):
        pass

Newly added get_sumr function checks total intensity. Within the writeTile function, the tile gets written into a file if and only if get_sumr returns a value greater than zero.