The future of AppImage: Type 2 and new tools

Experimenting with squashfs instead of ISO9660.
Switching from zisofs to squashfs would:

  • Significantly save space, up to 45% (e.g., 121 MB rather than 218 MB for QtCreator-5.7.0-x86_64.AppImage when using a 1 MB block size with -comp xz -b 1M but that makes application launches of large apps noticeably slower); or 27% (159 MB rather than 218 MB) with great performance using default gzip2 compression
  • Allow us to use less arcane generation tools, i.e., mksquashfs instead of xorriso
  • Significantly speed up AppImage generation time, around 40% (e.g., 1:38 instead of 2:37 for QtCreator-5.7.0-x86_64.AppImage)

Proof of concept:
https://github.com/probonopd/appimagetool
Not for productive use yet!

What do you think?

There is now also appimagetool, a convenient tool that will replace AppImageAssistant. It can be used to generate AppImages in the new experimental format.

https://github.com/probonopd/appimagetool

Coming along nicely.

Have added the type 2 image format to the spec draft.

In contrast to the earlier ISO9660-based AppImage format, the new type 2 format:

  • Is not tied to any particular filesystem format (so there might be others in the future); right now my implementation uses squashfs rather than ISO9660
  • Does not use a fixed offset to determine where the filesystem image starts, but specifies that the filesystem image starts right after the end of the ELF (however long it is). This means that the runtime does not need to fit into a predetermined amount of space, which is future-proof
  • Embeds update information in an ELF section inside the AppImage rather than at a predetermined, fixed offset
  • Allows for embedding a digital signature in an ELF section inside the AppImage rather than a .asc file next to the AppImage

Along with the capabilities of the new features comes a new set of tools that will eventually replace AppImageAssistant, AppImageExtract and friends.

appimagetool right now can:

  • Generate an AppImage from an AppDir
  • Automatically detect the architecture
  • Embed update information inside the AppImage
  • Calculate the update information string for Bintray repositories
  • Sign the AppImage with GPG and embed the signature inside the AppImage

validate right now can:

  • Check the digital signature inside an AppImage to ensure that the AppImage is coming from the original author and has not been tampered with

I am considering - let me know what you think:

  • Upload to Bintray
  • Integrate validate into AppImageUpdate

Type 2 AppImages have also learned quite a few new tricks:

  • --appimage-offset: Print the byte at which the squashfs filesystem begins, for mounting with -o loop,offset=...
  • --appimage-version: Print the version (git hash) of appimagetool which was used to generate this AppImage
  • --appimage-updateinformation: Display the update information that is embedded in the AppImage
  • --appimage-signature: Display the digital signature embedded in the AppImage
  • --appimage-mount: Only mount the AppImage without executing its payload
  • --appimage-extract: Only extract the AppImage without executing its payload
  • --appimage-icon: Only extract the icon of the AppImage without executing its payload

Note that these are not set in stone yet and are subject to change.

Want to try it out?

Generate a signed AppImage:

gpg2 --full-gen-key # Generate a signing key (once)
wget https://transfer.sh/mH0jk/appimagetool
chmod a+x ./appimagetool
./appimagetool ./XChat.AppDir --sign

Read the signature:

./XChat_IRC-x86_64.AppImage --appimage-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAABCAAGBQJX6CN9AAoJENBdKWeGw9/dsvoH/RgEggMiNTwgyA4io2Dyy1j1
6U3CQST9HVmh9PjeFKZCgFCZbHvpFz9mzhLTPlOAbczBnSmmbgqROINaLW+1tqEx
stOy67D3Z1cySzRTOhSkjiUOP5unmZL6QTNPxRHmuRkyihv7YfAlkrogXQlYbZ1h
Ilt6jU1b97GSPox/EE3Z002iZGJYQ3FfjAlp9o947goY5koA5KYqyzTCvEjhTk/L
wz1mFcjEkzHt9CaHZfrZCE3QVSBTq071wzsHCFHaJswPhA6iI0psCnFY56PPResi
uljTQr3nOBaqNyUgU3y4Tbd+36cwggSaTpGAzlhgNoalIwB1ltFSdPeRPe4Q3Qc=
=MR0w
-----END PGP SIGNATURE-----

Validate the signature:

wget https://transfer.sh/UQfeE/validate
chmod a+x ./validate
./validate ./XChat_IRC-x86_64.AppImage
(...)
gpg: Signature made Sun 25 Sep 2016 10:41:24 PM CEST using RSA key ID 86C3DFDD
gpg: Good signature from "Testkey" [ultimate]

Does it work for you?

Feedback is welcome!

This looks like great step forward to me. When I have some spare time I will definitely be trying this out with SER Player and will report back.

I had a thought that may or may not be sensible. When we create an AppImage we bundle all the libraries that may be required for any of the platforms we want to support. However, on some platforms many of these libraries are actually present and we are duplicating them on the hard drive (not such a big issue) and in memory.

Would it be possible for the AppImage to check which libraries are available on the host system and delete these duplicated libraries from the bundle so that the system versions are used instead? Though this may create problems if the system libraries are changed to different versions or removed. Anyway, just a thought.

Cheers,
Chris

Would it be possible for the AppImage to check which libraries are
available on the host system and delete these duplicated libraries from
the bundle so that the system versions are used instead?

Right now AppImages are read-only; you could unpack an AppImage, delete some libraries, and then repack it. But the result would then be closely coupled to your particular system rather than portable between different machines with different distributions and versions.

I boot into a different Linux distribution each time, and still like to use the same set of apps that I keep around as AppImages.

This has now landed in https://github.com/probonopd/AppImageKit, yay!

2 Likes

Wow, nice and sweet! ISO9660 format seemed unfit for this purpose, and AppImage format was very strange to me (but convenient, anyway!). Using squashfs and making it a true ELF (without ISO hacks) is a great leap forward! I have experimented with AppImage Type 1 and will definitely use Type 2 in my project.

Some security considerations:
In AppImage 1 you could easily extract the contents without executing the ELF part. In AppImage 2 you can extract the contents, but you must get the squashfs offset from somewhere.
So, in AppImage 1 the file manager could extract the icon using its system-wide ISO extractor (any of them).
And in AppImage 2 the file manager will have to execute the binary (potentially dangerous!).

I have experimented with AppImage Type 1 and will definitely use Type 2 in my project.

Thanks @GamePad64. Be sure to include your project on the list of AppImages.

you must get the squashfs offset from somewhere

Correct. At least 3 ways to do this:

  1. Ask the AppImage itself by executing it with the --appimage-offset option. It will print the offset and exit
  2. If you don’t trust the AppImage and don’t want to execute it, you can calculate the size of the ELF from the ELF headers, and use the calculated size of the ELF as an offset
  3. Or you could run the AppImage in a restrictive Firejail sandbox. This also mounts the AppImage

Can you point me to the documentation for the format of the AppDir when using appimagetool instead of AppImageAssistant? I have been round all the documentation multiple times but cannot find it and the AppDir that my script builds does not work with appimagetool.

Also, is appimagetool able to generate 32-bit AppImages? I can only find a 64-bit prebuilt version.

Thanks,
Chris

Hi @cgarry

AppDirs should not be different between AppImageAssistant and appimagetool. Are you getting specific error messages?

Yes, AppImageKit can be built for 32-bit too, and you’ll find such builds on https://github.com/probonopd/AppImageKit/releases (look for the i686 ones).

Thanks for the reply @probono, very reassuring as always.

This is the output I get when I try to create an AppImage from my AppDir:

cgarry@hp-mate-laptop:~/software/ser-player-clean/linux_appimage/temp$ …/appimagetool-x86_64.AppImage -v ser-player.AppDir
WARNING: appstreamcli is missing, please install it if you want to use AppStream metadata
WARNING: appstream-util is missing, please install it if you want to use AppStream metadata
Desktop file: /home/cgarry/software/ser-player-clean/linux_appimage/temp/ser-player.AppDir/ser-player.desktop
Name: SER Player
Icon: /usr/share/app-install/icons/ser-player.png
Exec: ser-player.wrapper %f
Comment: A video player and processing utility for SER files
Type: Application
Categories: Video;AudioVideo
File used for determining architecture: /home/cgarry/software/ser-player-clean/linux_appimage/temp/ser-player.AppDir/usr/lib/libXi.so.6
Arch: 86_64
App name for filename: SER_Player
dest_path: SER_Player-x86_64.AppImage
DESTINATION not specified, so assuming SER_Player-x86_64.AppImage
/home/cgarry/software/ser-player-clean/linux_appimage/temp/ser-player.AppDir should be packaged as SER_Player-x86_64.AppImage
/usr/share/app-install/icons/ser-player.png{.png,.svg,.svgz,.xpm} not present but defined in desktop file

There is no error as such, but there is no AppImage created. If it help I have put the actual AppDir here:
https://dl.dropboxusercontent.com/u/1139742/appimage/ser-player.AppDir.tar.gz

I also want to quickly mention that since I have been providing my app as an AppImage I have received many messages from Linux users thanking me for supporting their particular distribution. Thanks for all your work on AppImage which has made this possible.

Thanks,
Chris

This is the error. Please make sure that you have Icon=ser-player in the desktop file and not Icon=ser-player.png.

Thanks for that, I also had to remove the ‘/app-install/’ part from the path to make it work. I am not actually sure what that is doing there!

I am now able to build an AppImage using the AppImage version of appimagetool. However, if I try and use a version of appimagetool that I have built myself from the latest sources then it fails. I will post details of this once I am back on my Linux machine.

Thanks again for you help.
Chris