Proposal: Support registering external functions to be used in expressions in the style

[MikkoSteerpath]

Design Proposal: External style functions

Motivation

Some things, such as converting unix timestamps to be shown in a human readable way in for the end users locale can be extremely complex to accomplish using style expressions. Another case is implementing for example thousands separator for numbers. These things depend on the app user's locale settings and potentially even timezones.

While it could be theoretically possible to extend the style specification with every helper function app developers would need, or write extremely long and complex expressions, in many cases it would be much more convenient for everyone if there was a standard way for applications to register their own utility functions that could be used from the style.

Another example in the past was that I would have liked to use regular expressions in the string handling in the style. This proposal and js implementation were never accepted, because regular expression engines are different on different platforms, and there was a risk of introducing cross platform incompatibilities in the spec (iirc). Meanwhile, as a developer, I had no reasonable route forward (except forking the entire project, on 3 platforms), even if I was willing to accept minor differences between platforms.

The feature could also be used for prototyping new functionality and showcasing possibilities of functionality that could be included in the specification. For example one could hack together a function which returns the icon-offset based on the camera transformation matrix, to correctly place the POI in 3d space, even though this feature is not yet available.

If the value of the parameter could be updated based on animation frames, it would even be possible to make icons bounce by returning a different value for icon-offset each frame.

Proposed Change

Extend the style specification with a new expressions exec which takes two parameters, a string type parameter for the name of the external function to execute, and an array type parameter with the the parameters passed to the external function.

Examples:
["exec", "formatDate", [ ["get", "timestamp"], ["get", "poi_timezone"]] ]

API Modifications

Define a new type that is a catch all for all exec functions.

Add a new API function to the maplibre programmatic api to set and unset the catch all handler.

The new types and API could be:

type ExecStyleFunctionResult<T> = {
  cacheHint: 'until-animation-frame' | 'until-camera-move' | 'until-style-update' | 'forever';
  value: T;
}

type ExecStyleFunctionHandler = (fnName: string, params: unknown[]) => ExecStyleFunctionResult<unknown>;

function setExecStyleFunctionHandler(cb: ExternalStyleFunctionHandler | null);

Internally the engine would keep a map of the exec functions used, the previous value and

Migration Plan and Compatibility

This does not replace existing functionality but allows easy prototyping of new style expressions prior to rolling them into the spec.

Rejected Alternatives

The reason for a catch all function rather than registering individual handlers are:

• Minimal additional api
• Allow error handling on app side for incompatible styles, this is important for implementing application forward compatibility handling. It is much easier to inform the user with a prompt like "This map uses a map style that is not yet supported, please update your application" or return some default value, than to do so from a style parsing error.
• It is possible by the developer to create a more friendly api like registerFunction(funcName, funcImpl) on top of this single handler, but if maplibre implements this friendlier api, it is not possible to implement a catchall api.

The reason for passing params in a single array rather than passing params indivdually:

• Allows validating the expression's parameter count, even when the external function's parameter count is unknown.
• Catch-all would likely use array of params anyway, so this mirrors that implementation making the mapping 1-to-1 between the engine API signature and the style spec signature.

The reason why the catch all cannot just return the value:

• In order to allow the engine to optimize re-renders and re-evaluating the data there is a need to somehow tell the engine how long the returned value is valid. This is returned in the form of a cacheHint.
• If some other data is later needed, just returning the value would make it impossible to extend the current api without breaking backwards compatibility.
• Use case for returning the cacheHint is for example perhaps the function needs to fetch a translation asynchronously, but as the map needs a value now, the function returns a placeholder value with a cacheHint of until-animattion-frame and then once the fetch has resolved, the final value can be returned with a cacheHint of forever.

[sjg-wdw]

I see the need, but I wonder if just catching the data as it's being parsed might be simpler.
Extending the style spec to incorporate out of band functions is probably more logical in Javascript than it is on mobile. Where messing with the data after it's been parsed probably meshes better on both major toolkits.

[wipfli]

I agree with @sjg-wdw that it would be better to add way for the user to get all attributes of a point/line/polygon after parsing, and then being able to freely modify the attributes with a user-defined function. This means one can update/add/delete properties...

[MikkoSteerpath]

I can see that modifying the attribute data could be quite interesting as well, it has some challenges such as how to define which objects to modify, how to update the previous modification when the users locale changes and how to track when the data with the relevant properties is loaded.

In particular considering clusters, where the data is generated on the fly using the style rules, I don't think this approach will be easy to implement but feel free to prove me wrong.

As for this approach being less logical on natives I would like to give some counter arguments:

• On all platforms, there is a separate parsing context and evaluation context, and this would be implemented to happen in the evaluation context. Parsing would proceed as normal.
• On iOS for annotation styling, the approach could be considered callback based, as the developer is expected to provide annotation views for annotation objects when ever the engine asks. Using the delegate pattern on ios would feel quite natural as an ios api of this.
• On android the engine makes a callback to the OnStyleImageMissingListener when ever an image is missing. Granted, the listener pattern is not used for providing data, as there can be typically multiple java listeners for events, but in other non-maplibre libraries setting providers is a common pattern in addition to the listener pattern.

In summary, the evaluation inside maplibre is already dynamic and takes into account many global and runtime variables, the handler for custom code would just be another one. I would also argue that from a developer/library user point of view, setting a handler/delegate/provider is a common pattern on all platform.

[HarelM]

I guess the next question, assuming this is possible, might be the API signature.
You mentioned the javascript API, that can be very "loose", what would be the API for the other platforms, given that they are more strict in terms of types etc?
Would sending Object[] as the parameters in Android makes sense?

[sjg-wdw]

For Native I'd run this by the team and see what they think. Might be something really easy to do, but at least they're familiar with that part of the system.

[neodescis]

I agree with @sjg-wdw that it would be better to add way for the user to get all attributes of a point/line/polygon after parsing, and then being able to freely modify the attributes with a user-defined function. This means one can update/add/delete properties...

It would be great if we could modify the GeoJSON shapes too, in addition to properties.

This got me thinking about a potentially related use case that I am going to have to support soon: creating ellipses on the fly. Basically I have a set of points with semi-major and semi-minor axes, but I need to render them as ellipses (or polygonal approximations of them, given GeoJSON constraints). These will be delivered to the front-end as features in vector tiles. I have control over the vector tiles, but the size of the (potentially many) features coming across the wire would dramatically increase if I have to create the polygon approximations on the server. It would be ideal if I could handle this transformation client-side.

[MikkoSteerpath]

This idea has been explored before in maplibre/maplibre-gl-js#1295 and there is a bit of a summary in a comment about the challenges.

[wipfli]

@neodescis if we allow geometry-on-the-fly we could also do bezier curves for rounded lines...

[hiddewie]

I was referenced here to discuss a feature request for MapLibre GL JS, written in maplibre/maplibre-gl-js#4964.

In summary, I would like to have a way to specify (global) map state that can somehow be used in the style. The use case is to define a map theme, and allow the style to dynamically change the rendering of the style based on the chosen user theme.

My feature request would be satisfied if I could define custom functions to be used in the style specification, such that I can make some values in the style dynamic based on the map configuration in the client application.

If it makes it easier to discuss, I could also write a separate proposal just for the MapLibre style specification to allow defining and using map state in a style. There is a proposal for syntax in maplibre/maplibre-gl-js#4964, similar to how feature-state can be used.

Seeing that this proposal has been open for more than 10 months, what is the best way to move this forward?

[HarelM]

I'm pretty sure I've seen something similar related to dark theme and light theme that can benefit from "global" variables, but I can't find it... :-(
You can open a separate design proposal for the spec for something that you think can facilitate for this.
Other approach that I've seen people do it generate the style.json from parameters instead of using a style.json file, this obviously has all the flexibility of any programming language you decide to use.
Generating the json is fairly straightforward.

[hiddewie]

Thanks, I split off this proposal into #886 where it can be discussed in depth.