Help with a simple multitrack player in Python + PySide6

[jonata]

Hi! I'm experimenting with MLT for a side project, and I just want to create a multitrack player, that is, I want to basically play a video file in sync with an audio file. I tried the following basic code, with no success. I just want to make sure that's not something on my system and that's really something I am missing:

from PySide6.QtWidgets import QApplication, QWidget, QVBoxLayout, QPushButton
import mlt7 as mlt
import os


class VideoPlayer(QWidget):
    def __init__(self, video_file: str, audio_file: str):
        super().__init__()
        self.setWindowTitle("MLT Video Player")
        self.setGeometry(100, 100, 800, 600)

        mlt.Factory().init()

        profile = mlt.Profile()

        self.playlist = mlt.Playlist(profile)
        self.video_producer = mlt.Producer(profile, video_file)
        self.audio_producer = mlt.Producer(profile, audio_file)
        
        self.playlist.append(self.video_producer)
        self.playlist.append(self.audio_producer)

        self.tractor = mlt.Tractor()
        self.tractor.preview = True
        self.tractor.set_track(self.playlist, 0)

        self.consumer = mlt.Consumer(profile, "sdl2")
        self.consumer.connect(self.tractor)
        
        layout = QVBoxLayout()
        self.main_player = QWidget()
        layout.addWidget(self.main_player, 1)

        self.play_button = QPushButton("Play")
        self.pause_button = QPushButton("Pause")
        self.stop_button = QPushButton("Stop")
        
        self.play_button.clicked.connect(self.play_video)
        self.pause_button.clicked.connect(self.pause_video)
        self.stop_button.clicked.connect(self.stop_video)

        layout.addWidget(self.play_button)
        layout.addWidget(self.pause_button)
        layout.addWidget(self.stop_button)
        
        self.setLayout(layout)
        
        os.putenv('SDL_WINDOWID', str(self.main_player.winId()))

        self.tractor.set_speed(1)
        self.consumer.start()

    def play_video(self):
        self.tractor.set_speed(1)
        
    def pause_video(self):
        if not self.consumer.is_stopped():
            self.tractor.pause()
        
    def stop_video(self):
        if not self.consumer.is_stopped():
            self.consumer.stop()
        
    def closeEvent(self, event):
        self.consumer.stop()
        mlt.Factory().close()
        event.accept()

if __name__ == "__main__":
    import sys

    app = QApplication(sys.argv)

    video_file = "video.mp4"
    audio_file = "audio.mp3"
    
    player = VideoPlayer(video_file, audio_file)
    player.show()
    
    sys.exit(app.exec())

It keep crashing ("Floating point exception"), no matter what I do. If I just add the audio file, the window open, but wont play anything. Otherwise, it just crash.

Versions: Ubuntu 24.04.1, Python 3.10, mlt 7.22. I compiled the latest version 7.28, but still crashing.

Thanks for your attention!

[ddennedy]

SDL_WINDOWID will not embed the sdl2 consumer into the window. That only works with the very old SDL 1.2. I do not know if that is your problem, but it could be. Just because SDL2 might still have something in this regard, but that does not mean it will work with X, Wayland, or Qt. I do not know if this is your only problem. If you want to embed the video, you have a big job ahead of you if you want it to also be efficient.
What if you comment out os.putenv('SDL_WINDOWID', str(self.main_player.winId()))?
If you want to continue I suggest to start with src/swig/python/play.rb example and slowly make changes.

[jonata]

I would eventually try to use GL texture (qglsl?), I already use this technique with mpv.

Commenting the line don't fix it.

play.py returns a Segmentation Fault. 😐

[ddennedy]

play.py uses the sdl consumer. Do you have it, or did you change it to sdl2. I do not use python much, but it worked for me after changing it to sdl2 since I do not build sdl.

I just updated the play.py example because it has some bad examples of error handling. It is not sufficient to test if p due to the nature of our C++ wrapper based on our reference counting in C. Rather, you need to use if p.is_valid(). I also added our concept of "auto-profile" to the example.

Then, I extended the play example to use a playlist and multitrack tractor as you did, and it worked

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

# Import required modules
from __future__ import print_function
import mlt7 as mlt
import time
import sys

# Start the mlt system
mlt.Factory().init()

# Establish a default (usually "dv_pal") profile
profile = mlt.Profile()

# Create the producer
p = mlt.Producer(profile, sys.argv[1])

if p.is_valid():
	# Derive a profile based on the producer
	profile.from_producer(p)

	# Reload the producer using the derived profile
	p = mlt.Producer(profile, sys.argv[1])

	# Create the consumer
	c = mlt.Consumer(profile, "sdl2")

	if not c.is_valid():
		print("Failed to open the sdl2 consumer")
		sys.exit(1)

	# Setup a multitrack
	playlist = mlt.Playlist(profile)
	playlist.append(p)
	tractor = mlt.Tractor()
	tractor.set_track(playlist, 0)

	# Connect the producer to the consumer
	c.connect(tractor)
	
	# Start the consumer
	c.start()
	
	# Wait until the user stops the consumer
	while not c.is_stopped():
		time.sleep(1)
else:
	# Diagnostics
	print("Unable to open ", sys.argv[1])
	sys.exit(1)

If you still have trouble you need to validate your build and install of MLT using melt. For example melt -query consumers tells you which consumers you have. Can you play a video with melt? I discourage trying to get new development to work from a custom build if you are new to this project due to numerous dependencies and install-sensitive (libs and data loaded at runtime are location sensitive).

[jonata]

Thank you very much! It worked.

Then I tried to embed the SDL2 into a QWidget with no success, but I made a PR (#1033) that would enable it. We need to set it as in c.set("window_id", WINDOW_ID).

Thanks again!

[ddennedy]

Ok, cool. And you confirm that embedding with Qt worked with your patch?

[jonata]

Yes!