[ts] Add first event audio implementation to webplayer

[davidetan]

This is a first attempt to implement audio following events in the webplayer as requested in #1620.
You can have a try here (caveat: Safari works only on the walk animation since it does not support .ogg audio file and I left like that as example).

The following implementation is a draft and I just want to know if the approach I used is the right one.
I extended AssetManager to load audio.
I added a new configuration variable audioFolderUrl to specify the base path where audio files are placed.
Audio assets are loaded only when Skeleton data is parsed since audio file name is within the skeleton.

Main question is if the approach I used is correct. I attached a listener to the animationState event event and the sound is played within event callback. Is this ok to use such high level API?

Open point:

• Is the approach correct?
• Right now, Playing sounds does not pause/stop when player is stopped. -> Implemented for play/pause button, not when animation is changed. Should audio be stopped when animation is changed?
• Right now, Audio will never play when UI is disabled due to browser Autoplay Policy (read next message to discuss possible solutions)
• What to do when audio decode fails? Right now I stop the player with an error.
• Balance works only on recent versions of Safari. There is another API for audio balancing with a wider support, but little trickier to use. -> Used the older API to implement balance.
• Is audioFolderUrl configuration correct or a dictionary with file audio file name and url would be better (right now it is not possible to use an "inline base64 audio file"
• Why asset loading has differentiation for webworkers?
• Icons are not very consistent with the existing ones.

[davidetan]

I forgot to mention an important aspect of webaudio API.
Audio and Video Media cannot play automatically in webpages (aka as Autoplay Policy) and a user interaction is needed.
I tackled this problem by making the audio off as default, so that the user has to click on the "audio on" button.

This made me think right now that if the UI is disabled by configuration, the audio on/off button will never be shown and the user cannot enable the audio.
So, we have to think the best approach to enable audio when UI is disabled. Some ideas:

• If UI is disabled, an overlay appears stating that some audio will be played and the user has to click in order to make the player start
• If UI is disabled, an little overlay button in some corner with the audio disabled icon will appear and the audio will be played when the user clicks the button. In such a case the button can then disappear of be a permanent toggle audio button.
• Audio won't work if UI is disabled

Do you have any thought about this?

[davidetan]

Today I implemented:

• Pause/Play sound when player is paused/resumed. Should sound be stopped when animation is changed?
• Balance should work also on older browser now since I used an older, but wider supported API.

I think I will wait for your feedback before proceeding further with the other open points since there can be multiple possible solutions :

[badlogic]

Sorry, I overlooked the PR. Without having looked to much into the implementation, this is going to need testing across all supported platforms/browsers. Sadly, the Audio API is a bit flakey across some (older) browsers and you already found the trouble with supporting various audio file formats. I'll do a proper review sometime this week!

[davidetan]

Thanks for the reply :)
Is there any list of supported browsers by the spine player? I can do some tests on older browser too.
I will provide also a matrix of API used and browser supported version.

Last thing, these APIs are the same used by the Unity WebGL exporter. I'll try to search on their website if there is some browser compatibility declaration.

[davidetan]

All APIs, but two, are supported by browsers from 2014-2015: Chrome: >14, Edge: >12, Safari: >6, Firefox: >25, Opera: >22, IE: Not Supported.

The two APIs for pause/resume (AudioContext.resume, AudioContext.suspend) are supported from browser version from 2016-2017: Chrome: >41, Edge: >14, Safari: >9, Firefox: >40, Opera: >28, IE: Not Supported

So, I probably need to add a check for suspend/resume in order to avoid crash if called when not defined.

The only other caveat is that Safari does not support OGG.
Supported codec can be easily tested here (https://chinmay.audio/decodethis/) and using Safari you can see that OGG is not supported (moreover, on the code side it fails with a null because it is stupid and it does not follow the specification).
Unfortunately, OGG seems to not work also on Mobile Chrome, but I did not have an accurate test/research on mobile browsers I'll try next week.

Last thing, I was able to play OGG file in Safari using this library (same library used to play media on Wikipedia/media), but I don't think that bringing a dependency for this worth it.

[badlogic]

A few bugs and behavioural suggestions:

Scrubbing causes queueing up of audio sources

Reproduction

1. Load example-audio.html in your browser.
2. Enable audio by clicking the respective icon in the player.
3. Stop the animation playback via the stop button.
4. Scrub the timeline back and forth a couple of times.
5. Resume the animation playback via the play button.

Expected
Audio will only be played back when the respective event is encountered after resuming playback.

Observed
While scrubbing, every time an audio event was scrubbed over, it gets queued up. Once the play button is pressed, all queued audio events are played back instantly at the same time, resulting in a very unpleasend experience.

Suggestion
While paused, do not queue audio events.

Speaker button visibility/discoverability

The speaker button should only be visible if audioFolderUrl was given in the player configuration.

Users have to know they have to click on the player to enable audio. I propose this for better discoverability: upon loading the player, if a audioFolderPath is set, we should show the audio icon in the top left corner for a few seconds so viewers know that 1. there is audio and 2. they have to click to enable audio. This is to be done irrespective of whether the UI of the player is disabled or not.

The button can disappear after say 5 seconds in the top left corner and will then show up in the player UI (toolbar). You can work out the details to make it nice :D

Player disposal

A player can be disposed (see dispose example). All audio should immediately stop when a player is disposed.

Testing & documentation

I've tested this in Chrome, Edge, Safari, and Firefox on Windows and macOS and it works as intended. It must also be tested at least in Chrome for Android and Safari for iOS, including older versions. We need to document the audio feature here: http://en.esotericsoftware.com/spine-player

I have commented on other things directly in the PR code.

[badlogic]

Replies to your open points:

Resolved

Is the approach correct?

Yes.

Should audio be stopped when animation is changed?

Yes.

What to do when audio decode fails? Right now I stop the player with an error.

Log to the console. This is not a critical error that requires a complete shutdown of the player.

Why asset loading has differentiation for webworkers?

That was a contribution from a user that was needed in their context. It was pretty non-invasive, so it's kept as is.

Icons are not very consistent with the existing ones.

That is something @erikari may be able to help with :)

right now it is not possible to use an "inline base64 audio file"

Not supporting inline audio data is fine.

Unresolved

Is audioFolderUrl configuration correct or a dictionary with file audio file name and url would be better

The Spine Editor only stores the audio file name in an event, but not its full path. E.g. in your modified mix-and-match project, you have a folder mixAndMatch which contains all the audio. However, the events don't store , e.g.mix-and-match/footstep1.wav, but just footstep1.wav.

As such, requiring the audioFolderUrl in the player config and also that each audio file name is unique, seems fine to me. this assumes all audio files are stored flat in that directory, even if in the Spine project there may be a folder hierarchy.

Ideally, the editor actually stores the full audio file path relative to the path given in the Audio Files node in the Editor. In the example above, the event would then store mixAndMatch/footstep1.wav instead of footstep1.wav. That'd be a breaking change in the Editor. @NathanSweet what do you think?

Related: do all browsers support at least one common compressed audio format like MP3? If not, then it might make sense to switch audioFolderUrl to a dictionary approach. An entry's key would be the audio file name as stored in the event, e.g. footstep1.ogg. An entry's value would then be a list of files in the audioFolderUrl path that represent that audio file, e.g. footstep1.wav, footstep1.ogg, footstep1.mp3. The player could then choose the best file depending on what the browser it's running in supports.

If there's one common compressed format, then we don't need this more elaborate configuration scheme.

[badlogic]

Rename the mixAndMatch folder to audio and move it to examples/mix-and-match/. Add the audio events to the examples/mix-and-match/mix-and-match-pro.spine Spine project file, make sure the Audio Files node has its path set to ./audio. Modify examples/export/runtimes.sh to copy over the .json, .atlas, .png files to spine-ts/player/example/assets/, and also make it copy everything from examples/mix-and-match/audio to spine-ts/player/example/assets/audio. Remove example-audio.html and instead add a third player to example.html that loads the mix-and-match example.

To explain why: we want a single source of truth for the Spine projects that are used in all Spine Runtimes examples. We store them in examples/. When one of these Spine projects changes, or when we release a new Spine Editor major.minor version, we run examples/export/export.sh. That will export all Spine projects to .json, .skel, and .atlas files, stored in each project's respective $PROJECT_NAME/export folder. We then run examples/export/runtimes.sh to sprinkle those files around the example projects of each runtime.

[badlogic]

See comment above. Move this to example.html as a third player. Remove example-audio.html

[badlogic]

Why do you hate for loops?! :D

[NathanSweet]

Ideally, the editor actually stores the full audio file path relative to the path given in the Audio Files node in the Editor. In the example above, the event would then store mixAndMatch/footstep1.wav instead of footstep1.wav. That'd be a breaking change in the Editor. @NathanSweet what do you think?

It already works that way. Like with Spine's images path, if your audio path is audio and the sound file footstep1.wav, the event audio path is footstep1.wav. It doesn't (and shouldn't) include the audio path folder name, ie audio/footstep1.wav. You'd need to move the audio file to a mixAndMatch subfolder for the event audio path to be mixAndMatch/footstep1.wav.

Re: inline audio data, can this be enabled via the normal rawDataURIs property?

Re: audioFolderUrl, it would be tedious to need map entries for all audio files used in the skeleton. Maybe we can use convention for browsers that need a particular audio format? Eg, a browser that doesn't support OGG would strip .ogg and replace with .mp3 (or whatever). Ideally we can provide reasonable defaults for this mapping that work everywhere. We would allow customization of the mapping if there were multiple reasonable fallback formats the user could choose.

Re: AssetManager, do we want to decode all the audio into an AudioBuffer when the skeleton data is loaded? There could be a lot of audio events, and many might not get used. I agree we only want to do decode into an AudioBuffer once, but is it worth doing lazily, the first time the AudioBuffer is needed?

[davidetan]

Lot of things to say, but first of all thanks for the suggestions and review.
I did some update to the code and you can test them here. The first player has UI, the second not so you can test the floating button. The second one is also connected to a dispose button.

Scrubbing causes queueing up of audio sources

Fixed, however audio will start only if the audio event is encountered during animation running.
For example:

• if an audio start at Tstart and finish at Tend
• the timeline is moved between Tstart and Tend
• the audio won't be played, so no Tx + 5 to Tx + 10 sound of the audio will be played

Right now I simply avoided to "queue" audio to play, as suggested. The problem above can be fixed since it is possibile to play audio from a certain offset, but it is not so easy since audio buffer source can be played once and basically they would have to be recreated for each "timeline seek".
Right now I do not want to overcomplicate the code and I prefer to address first more important problems.

What to do when audio decode fails?

Right now I stop the player with an error. Log to the console. This is not a critical error that requires a complete shutdown of the player.

Done

Player disposal

A player can be disposed (see dispose example). All audio should immediately stop when a player is disposed.

Done, but can probably be improved since cached audio buffer are not cleaned up right now. I will give a more appropriate look.

Speaker button visibility/discoverability

Done.
The button will appear in the top-left corner and disappear after 5 seconds in a dumb way.
Possible improvements:

• Button could disappear before user has the possibility to see it (eg: player is at the bottom of the page)
• Make the floating button disappear time configurable (maybe someone want to leave it longer or forever)

Last note: the button is horrible and I'll ask @erikari to help me in making it fancy.

Playback speed/rate

I noticed it was easy to change audio speed/rate accordingly to the player speed.
However this change the audio pitch which is definitely not an easy thing to control with Web Audio API.
So, let me know if the pitch change is not worth the audio speed change.

Audio Buffer caching

In the previous version, I was storing the combination of AudioBufferSourceNode (containing the reference to decoded AudioBuffer), GainNode (volume), PannerNode (balance).
Audio is decoded lazily for events triggered by the running animation and it is decoded once, not every time an animation triggers an audio event.AudioBuffer is reused by the event.
The only thing recreated every time is the AudioBufferSourceNode because as written in the MDN documentation - An AudioBufferSourceNode can only be played once.

Storing the AudioBuffer within the AssetManager was the approach I was using at the beginning.
However, the problem is that the responsible for audio decoding is the AudioContext which cannot be initialised until the user interacts with the page as for the Autoplay Policy mentioned above: https://developer.chrome.com/blog/autoplay/. So, unfortunately decoding cannot be done during asset loading and I ended up doing it outside the AssetManager

The Audio Context is initialised only when the user presses the unmute button that means that I could do the audio decoding only after that.
An option might be to do the audio decoding right after the Audio Context creation for all audio events, but this would be concurrent to the animation being played and to the request of playing audio. So, if some audio is not yet decoded, the options would be:

• skipping audio event not loaded yet
• force the audio decoding for that audio (with the risk of a more complex code to tackle concurrency)

In any case I noticed that storing the aforementioned combination of audio nodes led me to decode ArrayBuffer to AudioBuffer more than one for the same ArrayBuffer if different events have different volume/balance.

I fixed this behaviour and now for each (requested to be played) audio file the decoding operation is done only once (there is the edge case in which an audio event requires an ArrayBuffer that is currently decoding).
In this way I can also avoid the ArrayBuffer copy in place (.slice(0)) since now each file is really decoded once.

However now AudioBuffer and AudioNodes are stored in the Player, while the ArrayBuffer is stored in the AssetManager.

Inline base64 audio file

inline audio data, can this be enabled via the normal rawDataURIs property?

It can be easily done actually. I just need to give to the user another way to provide the url - so the rawDataUri - for each audio file.
I was thinking to create a new audioPaths property for the configuration where the user can basically override the events dictionary in the skeleton. In such a way the users can provide the information to get the audio file in whatever as:

• rawDataUri
• a relative URL
• an absolute URL

SpinePlayerConfig
{
	...,
	audioPaths: {
		footstep1: "data:audio/mpeg;base64,//vQZAAP8AAAaQAAAAgA....",
		footstep2: "https://my.audio/footstep2.ogg",
		footstep3: "/footstep3.mp3",
	}
}

The users can also provide a different audio file from the one used within the editor.
This might address also to the problem of not supported audio file format for certain browser because the users can freely change the audio file independently of what is written within the skeleton.

Audio format browser compatibility

MP3 seems to be supported. I converted an OGG file to MP3 using and online tool and it works on Safari and Mobile Chrome. This can be easily saw using this tool that tests the audio decode of several audio file format using the same API.
The big problem seems to be OGG.
However, giving the user the possibility to provide an audio file with a different format as suggested in the point above might address the problem.

Example project

Rename the mixAndMatch folder to audio and move it to examples/mix-and-match/. Add the audio events to the examples/mix-and-match/mix-and-match-pro.spine Spine project file, make sure the Audio Files node has its path set to ./audio. Modify examples/export/runtimes.sh to copy over the .json, .atlas, .png files to spine-ts/player/example/assets/, and also make it copy everything from examples/mix-and-match/audio to spine-ts/player/example/assets/audio. Remove example-audio.html and instead add a third player to example.html that loads the mix-and-match example.

To explain why: we want a single source of truth for the Spine projects that are used in all Spine Runtimes examples. We store them in examples/. When one of these Spine projects changes, or when we release a new Spine Editor major.minor version, we run examples/export/export.sh. That will export all Spine projects to .json, .skel, and .atlas files, stored in each project's respective $PROJECT_NAME/export folder. We then run examples/export/runtimes.sh to sprinkle those files around the example projects of each runtime.

I used a different skeleton because I had the source of that one.
Is it worth to insert a new skeleton/atlas/png, etc for this?
I can add audio events within the existing assets (raptor or spineboy) and simply add audio files.

Things I did not worked on yet

• Documentation
• Icons
• Example project
• Using for loops rather than reduce and some other code suggestions :p

[NathanSweet]

let me know if the pitch change is not worth the audio speed change.

If the player can do it, that would be neat (FWIW the editor does it), but we can leave it for later/never if it's complex.

Audio Context is initialised only when the user presses the unmute button

There could be a long running audio event (like background music) that started before the user clicked unmute. Ideally we'd seek to the appropriate position when unmuted for audio events that have already occurred, rather than play nothing until new audio events occur.

An option might be to do the audio decoding right after the Audio Context creation for all audio events

A skeleton could have many audio events. Currently we download them all to ArrayBuffers via AssetManager loadAudio when the player loads. Should we only download for audio events for the current animation(s)?

We can't assume the player has a single animation because the AnimationState can be used directly by the user to play multiple animations at once. We could use AnimationStateListener start, look at the animation's EventTimelines, and call loadAudio for every audio event found.

Doing this would mean when animations change, new audio events can't play until the audio is downloaded. Maybe that isn't great and the simplicity of just downloading all the audio up front is OK. Loading only for the current animations could be left for later/never.

decoding cannot be done during asset loading and I ended up doing it outside the AssetManager

I wonder, should AssetManager loadAudio provide ArrayBuffer or AudioBuffer? If it provided AudioBuffer, it would not consider the audio loaded until 1) the ArrayBuffer is downloaded, and 2) unmute has occurred. Only then would it decode and fire the callback. This would mean AssetManager isLoadingComplete might need a boolean to exclude audio.

I just need to give to the user another way to provide the url - so the rawDataUri - for each audio file.

The rawDataURIs setting we already have maps a path to a data URI. The user would take some path (to JSON, image, audio, etc) that would normally be downloaded, like mix-and-match/footstep.ogg, and map it to a data URI. When the AssetManager goes to download that path, it sees there's a data URI and uses that instead of downloading.

The users can also provide a different audio file from the one used within the editor.
This might address also to the problem of not supported audio file format for certain browser because the users can freely change the audio file independently of what is written within the skeleton.

I'm not a fan of this, as mentioned in my last comment. It makes sense for data URIs, but not to workaround browser limitations. Users shouldn't need to list every audio path in the skeleton. If it's common to need eg to replace .ogg with .mp3 as a browser workaround, we can do that for them.

If the user really wants to customize audio paths with their own code, they could modify the actual skeleton data just after it is loaded.

Is it worth to insert a new skeleton/atlas/png, etc for this?

It's probably best to modify an existing skeleton to have audio events. Spineboy has footstep events, but no audio. That's the default project loaded by the editor, so maybe we can leave it without audio. Adding the squeak footstep to the mix-and-match project seems fine. Be sure to base the new version on the latest project file.

[badlogic]

Scrubbing causes queueing up of audio sources

I think it's fine to stick to the current simple behaviour.

Speaker button visibility/discoverability

You can detect if a player is visible inside the browser viewport. Only start the timer if it it is.

Playback speed/rate

A pitch change would not be desireable. IIRC, the Spine Editor doesn't do pitch shifting when playback speed is increased. Let's stick to the simple behaviour (I'm also pretty sure pitch shifting in the Web audio API is broken in some implementations, cause audio is always terrible).

Audio Buffer caching

Ah, yes, I totally forgot that we need interaction before being able to get the audio context. Let's keep it as is. That said, the key is currently the audio path, balance and volume. It makes more sense to key the audio path + event name. Those are unique within the skeleton.

I think having AssetManager not know about the stupidity that is "click to get audio context" is good and should be handled by the AssetManager using code, i.e. the player.

Inline base64 audio file
Audio format browser compatibility
As @NathanSweet said, just having the user provide data uris via rawDataURIs should be enough. This is related to audio format support actually. If all browsers can playback mp3, we do not need the mapping mechanism at all. Users are simply advised to use mp3. It's not like ogg is super popular relative to mp3 anyways. Let's keep it as simple as possible.

Example project
The mix-and-match project is in the examples/ folder as well. You can simply modify the corresponding Spine project.

[badlogic]

Raw data URI support looks good!

[davidetan]

I finally have some time to work on this. Here some small updates, I'll do some others improvements today.

rawDataURIs

Raw data URI support looks good!

Yeah, I completely misunderstood how rawDataURIs worked and I indeed implemented loadAudio in AssetManager in a way that was not able to use them.
After analysing better the code, I heavily simplify it, so now rawDataURIs works also for audio.

Speaker button visibility/discoverability

You can detect if a player is visible inside the browser viewport. Only start the timer if it it is.

Done.

Speed change -> Pitch change

A pitch change would not be desireable. IIRC, the Spine Editor doesn't do pitch shifting when playback speed is increased. Let's stick to the simple behaviour (I'm also pretty sure pitch shifting in the Web audio API is broken in some implementations, cause audio is always terrible).

If the player can do it, that would be neat (FWIW the editor does it), but we can leave it for later/never if it's complex.

• In the editor the pitch change is equivalent to the one I've Implemented. Am I missing anything?
• Using Webaudio API it is not directly supported to avoid the pitch change when speed change
• Fallback would be to use HTML5 Audio that seems to support it. I have to check it, but I think it does not support panning

Example project

It's probably best to modify an existing skeleton to have audio events. Spineboy has footstep events, but no audio. That's the default project loaded by the editor, so maybe we can leave it without audio. Adding the squeak footstep to the mix-and-match project seems fine. Be sure to base the new version on the latest project file.

The mix-and-match project is in the examples/ folder as well. You can simply modify the corresponding Spine project.

Actually mix-and-match project is not in the examples folder of the player. There's only Spineboy and Raptor. See here.
I'll add the mix and match example with more appropriate sounds, maybe demonstrating both audio download and embedding audio in html.

When manually moving timeline, play also audio events already started seeking at the right point

There could be a long running audio event (like background music) that started before the user clicked unmute. Ideally we'd seek to the appropriate position when unmuted for audio events that have already occurred, rather than play nothing until new audio events occur.

I have to think the best way to implement it. Probably, when an animation start/play button is pressed, I should schedule all audio events of current active animations. I such a case I need a time schedule of these events, but I did not find a convenient way to access the EventTimelines from the Player, yet. But I did not work on this so much.

Things I did not worked on/answered yet

• When download audio assets (at the beginning, at animation start, just in time)
• Where to store AudioBuffer (decoded ArrayBuffer)
• Documentation
• Icons
• Code hints

[badlogic]

mix-and-match project

I've commented on this extensively in my initial code review. All examples are stored in the examples/ folder, e.g. examples/mix-and-match. There you'll find the project's Spine file, its individual images, as well as an export/ folder.

The Spine projects in the examples/ folder are our "canonical" projects, which we also ship with the editor. The runtime examples make use of them through two steps:

1. examples/export/export.sh, a bash script that will export each project to .json and .skel alongside atlases with different settings (with/without premultiplied alpha, etc.). This is run whenever a Spine project in examples/ changes. The changed exports are then commited to Git. Not ideal, but that's how it is at the moment.
2. examples/export/runtimes.sh, a bash script that will take the contents of each examples/<name>/export folder, and copy it to each runtimes example project. For the web player example you can find the copies here: https://github.com/EsotericSoftware/spine-runtimes/blob/4.1/examples/export/runtimes.sh#L335

What I suggested was to:

1. Modify the "canonical" mix-and-match Spine project examples/mix-and-match/mix-and-match-pro.spine, adding audio events. You can also create a folder examples/mix-and-match/audio and store the audio files for the project there.
2. Run examples/export/export.sh so the changes you made to the mix-and-match Spine project get exported.
3. Modify examples/export/runtimes.sh to copy the files examples/mix-and-match/export/mix-and-match-pro.json examples/mix-and-match/export/mix-and-match-pma.atlas, examples/mix-and-match/export/mix-and-match-pma.png, and the audio files in examples/mix-and-match/audio to spine-ts/player/example/assets.
4. Run examples/export/runtimes.sh so the exported mix-and-match files are copied over to the web player example.
5. Commit the changes in examples/mix-and-match, including the new audio/ folder, and also commit the newly added exported files in spine-ts/player/example/assets.

This is the workflow when adding a new asset to a runtime's example project. We do want to minimize manual copying and duplication of "canonical" Spine projects.

EventTimelines

Yeah, that's a bit tricky. The EventTimeline class has a field frames, which is an array of floats. Each entry is the timestamp of the animation key that triggers an event. Which event is triggered is stored in the events field. So eventTimeline.frames[0] and eventTimeline.events[0] gives you the first key on the timeline, index 1 the second key, and so on. With that information you can figure out if the current playback position is "inside" an audio event. The timestamp tells you when the audio event starts relative to the start of the animation in seconds, and the end time of the audio event relative to the start of the animation would be given as event timestamp + audio duration. If the current playback position is between the start and end time, you can seek the audio to accordingly (relative to the audio duration, not the animation duration!).

That's one part of the issue solved. The other part, detecting the scrubbing and reacting to it by manually calculating audio playback positions, is actually quite a bit harder.

I think we can postpone scrubbing support though, unless you like a challenge :)

When download audio assets (at the beginning, at animation start, just in time)

I'd say at the beginning, so it's likely to be ready when the animation starts.

Where to store AudioBuffer (decoded ArrayBuffer)

The current storage in audioBufferCache looks OK to me.

Documentation

Documentation is written in the CMS in Markdown. The audio feature would need entries in this section, analogous to the other features. http://en.esotericsoftware.com/spine-player#Configuration

Icons

Maybe Erika can help with that?

Code hints

Not sure what that is.

[NathanSweet]

For starting audio events when unmute occurs, you can use EventTimeline apply to obtain all events from lastTime to time. Pass -1 for lastTime to get all events since the start of the animation. Ignore events that have no audio path. Use Event time to know when the event would have started playing and compare the elapsed time to the audio duration.

Pseudo code:

for each track in animState
   for each event timeline in the track animation
      apply timeline to collect events
      for each event
         if the event has audio and should be playing, start it at the appropriate time

I think the only downside is if unmute occurs just after an animation loops, any audio events near the end of the animation won't be played. That's not perfect, but probably fine.

[davidetan]

Example project

I've commented on this extensively in my initial code review. All examples are stored in the examples/ folder, e.g. examples/mix-and-match. There you'll find the project's Spine file, its individual images, as well as an export/ folder.

I am so sorry, I completely forgot you already explained that to me. You should have said to read your previous message 😅
I'll let you win at ping pong next time for that :)
I'll do it later 👍

Events triggered in the past when unlock audio or timeline moved

Thanks to your suggestions and secretes about runtimes, I rewrote how audio is played so that is easier to use play and seek audio.
I modified the json of an animation and added countdown audio to make it easier to test. I'll remove them when everything work as expected.
Give it a try here as usual: click me.

Howler.js - External dependency to manage audio

I wanted to avoid the usage of an external library, but now that the code is grown I think that using and external library to simplify some stuff and to increase robustness is not a bad idea. As you can see from the stars, howler.js is the library when working on audio on browsers.
It definitely simplify audio caching, audio seek (right now I have to destroy the currently playing audio node and create a new one) and browser compatibility. It also automatically attempt to unlock the possibility to play audio.

I actually think that right know the implementation does most of is useful for us and the usage of this library would lead to another heavy refactor. But before realising this feature I want to be sure we have analysed all the possibilities.

Trackentry usage

https://github.com/EsotericSoftware/spine-runtimes/pull/2169/files#diff-7280024e5c81ae34b392ba380c256d2b9c686dd03c502902dc22312029c80b1cR1227-R1240

Am I doing it right?

Other stuff

I'll leave a list of things I still I have to work on as usual:

• Example Project
• Fallback on .mp3
• Documentation
• Icons
• Some TODO I left in the code
• Final refactor: I was thinking to create an audio class within the player since properties and methods grew

[badlogic]

Events triggered in the past when unlock audio or timeline moved

This is great and works exactly as I'd have imagined it. I do not think this needs any improvement. Good job!

Howler.js - External dependency to manage audio

I've used Howler.js in the past, it's OK. However, we strive for zero dependencies. The amount of code added to Player.ts for audio support is still < 200 LOC and I think it's pretty easy to follow. Howler would not enable us to add much more functionality, so I do not think switching to Howler is an improvement. If we considered it, it would have to be an optional dependency, which complicates the build and deployment further. So, my answer is no :)

Trackentry usage

Jupp, that looks good!

Other stuff

I don't think we need the .mp3 fallback, if MP3 is supported everywhere. We will simply advise our users to use MP3s for the web player use case.

A separate audio class seems fine.

[davidetan]

I eventually used the raptor project since it was easier to find audio for its animations (usuale external example).

I exported the project using the export.sh and then importer into spine-player examples modifying and using runtime.sh as @badlogic.
I actually had some problem during this process since more file than expected were exported and added to runtime examples.
I committed only the one related to the raptor project and only the json files.

I probably made something wrong, but that can be fixed quickly.
Most important thing is that now the code is ready to be reviewed.

Below you can find the documentation I wrote. English can obviously be improved.

Audio events

In order to play audio when audio events occur, it is necessary to provide the absolute or relative prefix URL where audio files can be downloaded.
This URL can be provided through the audioFolderPath string property.

<script>
new spine.SpinePlayer("player-container", {
 ...
 audioFolderPath: "assets/audio"
});
</script>

The player looks for all audio events within your skeleton and downloads the audio file corresponding to the combination of the audioFolderPath and the event audio path for each audio event.
rawDataURIs property can be used to embed audio files in base64 form within your HTML.

The player starts with the audio muted to respect autoplay policy. To unmute audio the user can:

• use the audio button shown in the player controls
• use the floating audio button that appears in the top left corner when the player is within the browser viewport. This button appears only once and disappears after 5 seconds. If showControls is false, this button is the only way to unmute audio.

Be aware that not all browsers support decoding all audio formats. Use MP3 for a wider compatibility.