[FEATURE] Support adding preset files and directories from ZIP archives

[kblaschke]

Your Suggested Enhancement

Add support for reading presets directly from ZIP archives, and adding individual presets or directories from a ZIP to a preset playlist.

Playlists should support mixed sources, e.g. presers from the local file system and different ZIPs in the same playlist.

ZIP references can be stored or given on the command line (in --presetPath) such as zip://path/to/package.zip?/path/to/preset.milk.

[stuaxo]

Can presetPath contain multiple paths ?

It wasn't clear from the UI in projectMSDL, but I was expecting it.

It might be worth having a look at some that are out there, I think my expectation as a user is that if a subdirectory or a subdirecrory inside a zipfile has Presets and Textures directories inside that these would he picked up.

The zip I tried was something like Isosolese Mashups 2024.

It would he great to be able to stick all these zip files inside some folder, then the system indexes them and grabs what it needs.

[kblaschke]

Can presetPath contain multiple paths ?

It wasn't clear from the UI in projectMSDL, but I was expecting it.

Via the command line or within the UI, it's currently only possible to add a single file or path. The configuration file allows to add multiple paths using sub keys. Any paths added in subkeys will always be added to whatever is passed on the command line, which is a bit confusing and will change when the playlist editor is finished.

My plan is to make it possible to provide the --presetPath argument as often as necessary and each path/file would then be added to the list, replacing any pre-configured paths or playlist. Note that the path will then also support preset playlist files, which would be merged into a larger playlist and can be mixed with directories and individual preset files.

It might be worth having a look at some that are out there, I think my expectation as a user is that if a subdirectory or a subdirecrory inside a zipfile has Presets and Textures directories inside that these would he picked up.
It would he great to be able to stick all these zip files inside some folder, then the system indexes them and grabs what it needs.

That's also a possibility. Currently, the directory parser only searches for .milk files, but could additionally look into ZIPs as well. I'd make this configurable though, as it could take a long time to recursively iterate many large ZIPs, even if they don't contain presets. Plus, it should always expose behavior the user would expect or at least can control to some degree.

Textures are a bit different though as projectM can currently only load files from the host filesystem, not from data streams or memory as with the presets. Issue projectM-visualizer/projectm#870 needs to be implemented and released beforehand.

[rootnotez]

I'm concerned this will add a lot of complexity for little value. Although I'll note that, for example the presets-cream-of-the-crop is ~130mb on disk, and ~17mb as .tar.gz; so there is significant space savings. Although with modern storage sizes, that's not a huge issue.

The complexity I see is managing additional deps, (for accessing zipped files), the complications of explaining to end-users the multiple different ways of managing presets, and then testing for blocking / io behavior when reading from zips.

And I'm not familiar with many other apps that allow you to provide directory's in zipped files.

[stuaxo]

The complexity is true, and it would definitely add one or two libraries.

I'd suggest only zip files, not other compressed formats: in zip each file is a stream, and it's easy to get a list of them and using zlib or ziplib get data that is one file inside.

Apps that support browsing inside zips or zips as files include kodi, for browsing media, many emulators (especially libretro based) can load from zips, the Java ecosystem from .jar, Android, from apk, programs that have a file format that is a renamed zip containing other files, such as Libre Office and Microsoft Office.

[kblaschke]

It also would be a frontend-only addition, as libprojectM itself doesn't have to care for where files originate from - as long as the application provides it either as a filename on disk or via data stream.

In the C++ SDL2 fontend for example, I'm using POCO for all the application stuff, and it also has a very convenient ZIP library built-in by default, so it wouldn't even add another dependency in this case.

Supporting TAR (with all compression variants) is hard, because it doesn't have a directory like ZIP, it's just a stream of concatenated files which is then compressed - so you'll have to read in and unpack the whole archive once to create an index, which can take a few seconds.

[stuaxo]

Having an offline index (I guess most things use sqlite for this now) could allow a lot of flexibility when it comes to managing large amounts of presets.. but I'm getting ahead of myself.

Thanks for the pointer to POCO, I'm a software dev but don't touch C++ much, so wouldn't have known :)