Thoughts on an infrastructure for distributing Linux application bundles

Developers
1

A system to distribute Linux application bundles and exchange metadata about them is needed, and this writeup ponders on how to create the infrastructure needed for that.

Original article at
https://plus.google.com/105493415534008524873/posts/Z43MNMyPQW8

Sebastian Kügler introduced at #QtCon the #KDE Software Store https://dot.kde.org/2016/09/03/kde-software-store, a central location to get themes, wallpapers and other non-compiled assets for applications or the desktop. What is even more exciting, he also announced that it will support “containerized apps”, more adequately known as “application bundles” in formats such as Flatpak, Snappy and/or #AppImage in the future.

In this writeup, I will outline my ideas for the KDE Software Store and for distributing #Linux application bundles in general. This writeup is inspired by various chats I had on this year’s QtCon. I can’t name you all, but thanks especially to starbuck, Matthias Klumpp, Aleix Pol, Sebastian Kügler, Nicolas Froment, and Thomas Bonte.

Linux distributions have been doing an excellent job at providing operating systems composed of many individual components. They have failed, however, to provide an easy way to sideload and run applications that are not core to the operating system. Getting the latest #GIMP for #CentOS is an experience like upgrading from Windows 7 to Windows 10 “Anniversary Update” just to get the latest version of Photoshop - no wonder the Linux market share on the desktop is still so low.
Up to very recently, even the #Qt-based application started by Linus Torvalds, Subsurface, was available on the download page for Windows and for macOS but not for Linux, and the packages in the Linux distributions were either missing or utterly outdated. This is where Linux application bundles and app stores (called so even if what they offer is free) such as https://www.linux-apps.com and app centers such as GNOME Software https://wiki.gnome.org/Apps/Software and KDE Discover https://github.com/KDE/discover come into the picture.

Let’s take a concrete example. As of today, QOwnNotes 16.09.2 is one of the latest apps on the site. If I click on the “Files” tab, it takes me to http://www.qownnotes.org/installation where I can find installers for Ubuntu, Linux Mint, openSUSE , Fedora Project, Arch Linux, KaOS Linux, Debian, Gentoo, Slackware, and CentOS. Wow, that must have been a lot of work for the QOwnNotes developer Patrizio Bekerle! Not every application project will have the time or resources to do that.

As do most open soure software projects these days, QOwnNotes uses GitHub for soure code and release management at https://github.com/pbek/QOwnNotes. It also uses Travis CI to do continuous builds. Whenever someone pushes source code to the repository, Travis CI builds and tests the code at https://travis-ci.org/pbek/QOwnNotes. After a release build, it is deployed automatically to GitHub Releases at https://github.com/pbek/QOwnNotes/releases. After each build, notifications are sent out to the qownnotes IRC channel and to a mail address of the developer.

This is a quite typical combination for independent open source software projects, and here is where I would like Linux app stores to hook in.

Imagine that the Travis CI build would also generate a generic application bundle for Linux, like it already does today for macOS using macdeployqt to produce the actual bundle. Now imagine that linuxdeployqt https://github.com/probonopd/linuxdeployqt will do the exact same thing but for Linux (although the developer also could do it by hand like some https://github.com/search?l=yaml&q=appimagetravis&type=Code&utf8=✓ do).

Now, wouldn’t it be great if the developer could add another snippet to set up a webhook notification https://docs.travis-ci.com/user/notifications/#Webhook-notifications that would inform the app store that the new version is available? For that to happen, each app store would need to have a URL at which it could be informed about new releases.

Automation is sorely needed so that information doesn’t get outdated - on the Krita page https://www.linux-apps.com/p/1126921/ you’ll find versions from 10 years ago, complete with discussions on klik (AppImage predecessor), despite that Krita provides each release as an #AppImage these days.

But wait, we don’t just want to tell one app store “here is a new release of QOwnNotes”. Ideally, we want to inform all app stores that “here is a new release of QOwnNotes, here is its description, here are it screenshots, here is the link to its homepage”, etc.

Which means a format for exchanging metadata about applications is needed. This is exactly what AppStream https://www.freedesktop.org/software/appstream/docs/sect-Metadata-Application.html is (already widely in use). The application author can provide a /usr/share/metainfo/qownnotes.appdata.xml file that would specify all of these things that describe the application, but not the binaries (e.g., download links). For those, the AppStream system mandates that a so-called AppStream Hub (also known as a “compose server”) collects the information about the available binaries and puts them into what is called “collection data”. AppImage support was recently added to the AppStream specification https://github.com/ximion/appstream/commit/1df590df8557625b14c482d5af50cb30a35d47ca.

Maybe the application author wants to provide different channels with different versions for different user groups. One type of user might want the latest released versions, while aother type of user might want every bleeding-edge continuous build. The author should be able to define channels like “release” and “continuous” which users could then select, with the default being “release”.

So, the flow should go like this: The QOwnNotes author uses Travis CI and linuxdeployqt (or an own script) to produce a cross-distro Linux bundle, e.g., an AppImage. He adds a webhook notification to an AppStream Hub, informing it about a) the location of the qownnotes.appdata.xml file and b) the location of the AppImage. The AppStream Hub would use that to update its collection data (the information about all the available software) and send that to app stores like https://www.linux-apps.com and, in the future https://store.kde.org/. If someone writes a plugin for Gnome Software and KDE Discover, it would also show up in the distribution’s native package mangager GUIs, along with the native distribution packages. The AppStream Hub could download and inspect the AppImage, e.g., to make sure it really runs on at least a defined set of distributions, and to extract further information (e.g., the download size) from the AppImage. It could also scan for outdated versions of libraries contained in the AppImgae and perform various lint tests, https://github.com/probonopd/AppImages/blob/master/appdir-lint.sh is just a starting point and could do much more.

Now, what about updating the bundled applications on the users’ systems, you might ask. AppImageUpdate can do binary delta updates of installed applications, which means that to update from one continuous build to the next only a few MB need to be downloaded, rather than the entire bundle https://github.com/probonopd/AppImageKit/blob/master/AppImageUpdate.AppDir/README.md.

What about security? The system should check for gpg signatures to make sure that the bundled applications really come from where they claim to come from.

Unlike other solutions in the making, this would not be slanted to any particular Linux distribution, store, or set of tools (yes I mean you, Mark Shuttleworth).

This should make everyone happy: The application developer can continue to use the well-known workflow consisting of GitHub, Travis CI, GitHub Releases, and Notifications. He doesn’t have to produce a gazillion of different native distribution packages anymore (but of course can choose to continue doing so). The app store gets the most recent information about new versions together with the most recent download links for cross-distro application bundles. The user can go to the app store and download software without having to deal with repositories and without having to upgrade anything in his stable base system. And all of this works entirely independent from the distribution’s package mangager.

So, here are my questions:

  • How would you like a solution like this?
  • Would it make your life easier?
  • Do you think it would make your users happier?
  • Would you be willing to use such a system?
  • Would you be willing to invest some time in setting up the infrastructure for this?
  • Are you aware of better solutions?
  • What else comes to your mind?
  • Who would be in the best position to operate the AppStream Hub?
2

I would like to expand this discussion a little bit. In addition to distribution it would be really nice to have some (optional) system to manage already installed appimages.

What I am expecting from it:

  • I would like to keep my appimages organized in single place (let’s call that dir “registry”)
  • Easily add-remove appimages from registry.
  • Added appimage automatically produces necessary desktop files (like current desktopintegration script does) AND creates a binary placeholder/alias (so I can run Krita just by typing “krita” from console).
  • Ability to have multiple versions in the registry. Ability to choose which version is active (current).

Maybe such utility already exists? If it doesn’t I will consider to try to write something simple for beginning.

1 Like
3

I would like to keep my appimages organized in single place (let’s call that dir “registry”)

/usr/bin, $HOME/bin or something like that?

Easily add-remove appimages from registry
Added appimage automatically produces necessary desktop files (like current desktopintegration script does)

This is what appimaged (the optional AppImage daemon I am working on) will do.

creates a binary placeholder/alias (so I can run Krita just by typing “krita” from console)

Just copy it on your $PATH

Ability to have multiple versions in the registry. Ability to choose which version is active (current).

You can have as many as you want, but I am not sure how to best choose between them.

4

@probono, thank you for responding. Actually, when I talk about “managing” I expect some kind of CLI to manage the installed appimages. I.e.:

appimage-manager add http://files.kde.org/krita/3/linux/krita-3.0-x86_64.appimage krita 3.0 - downloads appimage from given URL, places it into registry under “krita” subdir (that’s what I mean “organizing”), creates desktop file, sets alias to call it as “krita” (3rd argument).

In this case, calling the same with another URL will add another vw\ersion and set it as default:
appimage-manager add http://files.kde.org/krita/3/linux/devbuilds/krita-3.0.1-Alpha-4c91a18-x86_64.appimage krita 3.0.1

Then, I can list versions:
appimage-manager list krita - shows all appimages of registry marked as “krita”. Outputs:

3.0
3.0.1

Rollback to version 3.0
appimage-manager set krita 3.0

5

Actually, I have found that 0install can do something like that for me. Probably I can teach it to work with appimages. :slight_smile:

6

@morevnaproject you seem to be looking at this from a traditional distribution/package management view. Nothing wrong with that, I just think that AppImage doesn’t need this.

The model I have been developing this for is:

  1. User visits the website of his favorite application
  2. Next to the download for Windows and macOS, there is an AppImage
  3. User downloads the AppImage
  4. The optional appimaged daemon recognizes that an AppImage has been downloaded and integrates it with the system (menus etc.)
  5. At this point, the AppImage is fully usable - no installation or “management” needed
  6. If the user wants to store the AppImage in some location (e.g., on an USB stick or in /opt) he is free to do so, but can do this in the File Manager without special “management” tools
  7. To remove an application, the user just deletes it, again, no “management” tools needed (appimaged should recognize the deletion and remove any desktop integration)

If you want admin-style “management” tools, then Flatpak might be more appealing to you. It is certainly way more advanced in this area. For me, it completely misses the point however, because it doesn’t allow me to drag-and-drop a downloaded app onto a USB thumbdrive and carry it around with me.

1 Like
7

I understand your approach. Flatpak is surely not an option for me either (I am also not a fan of their concept).

One question: Suppose I will drop several krita appimages into location managed by appimaged. Which one will be my KRA files get asociated with?

8

All of them, meaning you will have multiple Krita instances to choose from. But this is an area where some conceptual thinking is required to make it even simpler. For starters, we should investigate how macOS does this.

9

Ah, sorry, I have incorrectly wrote my question. Te correct one: “Which one will be my KRA files get associated with by default?”

If I remember correctly (but I might be mistaken), the last added desktop file takes place used to be associated by default (unless user manually tweaked the order of associations). So, it might depend on the order, which appimaged uses to parse files.

10

Possibly. I don’t know. Experimentation is needed.

11

Hi

I think any SW downloaded need management. Mainly because I believe any data stored on a system need management. Backup/purge/update/cleanup.
Its just a matter of time when one forget where the SW was downloaded.
People have different PATH approach/multiversioning approach to make their daily life easier and as soon as you fetch stuff from dozen places at different time to different targets things are becoming quite messy and centralized mgmt approach is nice for long term system maintenance.

BR

12

RIght now, appimaged will register both. Which one “wins” is handled by the OS - I haven’t systematically tested yet what happens. macOS provides a way for the user to select with app a file gets opened with from the “information” dialog (similar to “Properties” on Linux desktops).

13

Have you ever used a Mac?
There it’s so easy to “manage” software in the file manager that you don’t even notice you are “managing” it.

14

No. I have not used MAC.
But for me management start already in the phase where I look for SW and all of that should be easy for end user.
Finding software
Onboarding software
Instansiate software
Update software
Decomission software

So ecosystem (store/registry/catalogue/hub) is appreciated. And application manager which can interact with ecosystem(s) and my system is helping me as user.

So I am quite positive to such infrastructure and believe would make life easier.
I am getting little bit concerned with all distro specific installers (yum, apt, dnf, pacman), environment installers (pip for python, docker pull for containers etc) so even some higher bundle agnostic management could be worthy to start to look into.

Because even if appimages dont require any specific installer or package manager. It is still yet another workflow for users when it comes from 0 to use of piece of software.

One thing that I like with android is the “memory” which applications I have installed on all my devices. In case of system crash, or HW replacement I can always redeploy those apps on my new system instance/device. Or I can push same app to many devices. So such feature is something I would expect in this infrastructure.

15

I’m not sure I fully understand the OP, but if this means that app stores can distribute updates for AppImages and developers can hook the building of AppImages into their CI and it’s all one fluid process, then this idea is excellent and should go ahead ASAP. The lack of this sort of thing is what’s tilted me towards snappy and Flatpak over AppImage so far, but this feature really would bring AppImage into its own. Dayum we’ve got a lot of good tech being made!

1 Like
16

Yes, it means that :slight_smile:

1 Like
17

I think it would be also a good solution to create plugin for regular appstores, just like gnome-software-plugin-snap and gnome-software-plugin-flatpak, it would very nicely integrate with other sources.

18

Hi @k would you like to make one? Should entirely be doable, it’s just that I am not interested enough in the desktop app stores so that someone else than me would have to write the plugins.

Some time ago, I had actually started doing one, so you could pick it up from there:

1 Like
19

Thanks for repo I will check it out (but I don’t konw if I am able to create such a plugin).

It would be good to ask people from gnome community about it, maybe it could be added and developed as an official part of gnome’s set of apps.

20

It would be good to ask people from gnome community about it

Feel free to reach out to them but based on earlier interactions I got the impression that they are biased toward pushing Flatpak at the expense of other application distribution mechanisms such as AppImage.