3. An overview of the XEmacs Packaging System

The XEmacs Packaging System is a system for administering the installation, upgrade, and removal of Lisp libraries. For the end user, it provides facilities for determining availability of packages and which versions at remote sites. It will download and automatically install a package, ensuring that any old files from previous versions of the package are removed first. By providing a standard set of hierarchies for installation, it makes configuration of XEmacs simpler. Furthermore, packages normally provide ancillary auto-autoloads and custom-loads libraries, which are automatically detected and loaded by XEmacs upon startup. This means that once installed, all facilities of package, including autoloading the library upon invocation of a command provided by the library and convenient configuration and customization, are automatically available to the user. There is no need to add autoloads or keybindings to in the init file, and structured configuration of the package is available through the Customize system even before the libraries are loaded.

All of this convenience comes at a cost. The cost of administration at the package level is negligible compared to the benefits, of course. However, the requirement that XEmacs find and load auto-autoloads and custom-loads libraries comes at a fairly large cost in startup time. In order to reduce this cost, XEmacs imposes fairly strict conditions on the structure of an installed package.

Meeting these requirements, as well as simply providing the auto-autoloads and the information about availability and so on does impose some costs on the library maintainer. The XEmacs Packaging System also provides structure and utilities to the library maintainer to make these tasks easier. This manual documents the requirements and the tools that the XEmacs Packaging System provides to ensure that a package satisfies them.

3.1 The User View

N.B. Much of the discussion in this section undoubtedly belongs elsewhere, (xemacs)Packages.

From the user’s point of view, an XEmacs binary package is simply a standard tarball (usually gzipped) containing Lisp sources, compiled Lisp, documentation, and possibly data files or supporting executables. The tarball is unpacked using standard tools such as GNU tar and gzip. The package system does impose certain requirements for automatic configuration to work.

Here the main consideration is that the tarball “expects” to be unpacked from the top of a package hierarchy. A package hierarchy is basically an image of a classic Emacs “run-in-place” tree, with ‘lisp’, ‘etc’, ‘info’, ‘man’, ‘lib-src’, and ‘pkginfo’ subdirectories of the top. The ‘pkginfo’ subdirectory is for use by the XEmacs Packaging System administration tools, and currently contains a ‘MANIFEST.package-name’ file for each package to ensure that no cruft remains when a package is removed or updated. The ‘lisp’, ‘etc’, and ‘lib-src’ subdirectories are further subdivided, with a subdirectory for each package. The ‘info’ directory obeys the usual conventions. I.e., the ‘info’ directory is flat with a(n) (optional) ‘dir’ file and one (set of) info file(s) per package. The ‘man’ subdirectory typically contains documentation sources, separated by package. (It does not contain ‘man(1)’ pages, as Emacs provides very few of them.)

There are several standard package hierarchies, and administrators can configure others at build time, while users can configure others at run time. The standard system hierarchies are all subdirectories of an XEmacs installation root, typically ‘/usr/local/lib/xemacs/’. These are the ‘xemacs-packages’, ‘mule-packages’, ‘infodock-packages’, and ‘site-packages’ hierarchies. Each has the structure described above, but the purposes differ. The ‘xemacs-packages’ is the normal place for installing “official” packages and many third-party libraries. Unfortunately, it is not yet quite possible to read libraries containing international characters with a non-Mule XEmacs, so such libraries are sequestered in the ‘mule-packages’ hierarchy. Some packages are compatible only with the Infodock development environment, and they will be installed in the ‘infodock-packages’ hierarchy. The ‘site-packages’ hierarchy is for packages not distributed by XEmacs.org, typically locally developed.

Packages are in principle supposed to be XEmacs version-independent, but if such dependencies are unavoidable, additional standard package hierarchies may be installed under version directories, e.g. ‘/usr/local/lib/xemacs-21.4.6/’.

Users who do not have sufficient privilege to install packages in the system hierarchies may install package hierarchies under ‘~/.xemacs’. At present only the ‘xemacs-packages’, ‘mule-packages’, and ‘site-packages’ hierarchies are supported, but it might make sense to extend this to support ‘infodock-packages’ hierarchies in the future.

The package hierarchies are not searched directly for libraries to be loaded; this would be very costly. Instead, the hierarchies are ordered according to certain rules, and searched for package lisp directories at invocation. These directories are added to the general load-path. As usual, it is load-path that is searched at run-time. This approach is somewhat costly at initialization, but results in a very “clean” load-path.

The order of search can be changed at build time by specifying the ‘--with-user-packages’ (an alias for ‘--with-early-packages’), ‘--with-system-packages’ (an alias for ‘--with-late-packages’), and ‘--with-legacy-packages’ (an alias for ‘--with-last-packages’) options to ‘configure’, or at run-time by specifying the EMACSEARLYPACKAGES, EMACSLATEPACKAGES, EMACSLASTPACKAGES environment variables. See (xemacs)Packages.

The default order of search is hierarchically determined. First, the roots are ordered. The early roots are the user-specific roots, typically ‘~/.xemacs’. The late roots are the system roots, typically ‘/usr/local/lib/xemacs-21.4.6’ and ‘/usr/local/lib/xemacs’, in that order. All hierarchies for a given root are searched for package Lisp directories, which are appended to load-path in the order found. Then the search proceeds to the next root, whose results will be appended to the load-path generated by previous roots.

Second, the hierarchies below each root are searched in the order ‘site-packages’, ‘infodock-packages’, ‘mule-packages’, then ‘xemacs-packages’.

In each hierarchy there should be a ‘lisp’ subdirectory, containing directories named for the packages. Each package’s Lisp libraries thus are contained in a directory of the form root/hierarchy/lisp/package/.

With such a complex search algorithm, the possibility of libraries being shadowed by another library with the same name is quite real. There are two considerations here. First, every XEmacs package contains certain libraries with constant names. These are

‘_pkg.el’

Lisp code to inform the package administration system about the package

‘auto-autoloads.el’

Lisp code to set up autoloaded functions and variables that may be needed at load time

‘custom-load.el’

definitions of configuration variables for use with the Customize system.

They are special-cased, because the way they are used prevents shadowing from being an issue.

Second, it is possible that multiple copies of some library, or different libraries with the same name, are installed in various places in the hierarchies. To detect such shadows, use list-load-path-shadows.

Finally, note that most basic Emacs functionality, including most of the Lisp API, is implemented in Lisp libraries. Because they use internal reserved APIs that are subject to change according the needs of the developers, these libraries are distributed with the XEmacs binary, and are called core Lisp libraries. Most core Lisp libraries are “preloaded” into the Emacs binary and in normal usage are never explicitly loaded. However, they can be explicitly loaded, and if so they are searched on load-path. Furthermore, functions such as locate-library will also search on the load-path. The searching takes place under somewhat different rules from those used for packaged Lisp. It is probably easiest to think of the package hierarchy searching algorithm as receiving a load-path initialized to the core Lisp directories.

3.2 The Library Maintainer View

From the library maintainer’s viewpoint, the advantages to the XEmacs Packaging System stem from the convenience to the user of installation and upgrade. Since an installed package automatically registers its entry points via autoload and its configuration variables with the Customize system, configuration FAQs are reduced. When it’s easy to upgrade, users learn to try ‘Tools | Packages | Update Installed Packages’ before posting a FAQ whose answer is “long since fixed, please upgrade.”

This comes at some cost, as the library maintainer needs to arrange that the package be installed in a directory structure that satisfies the requirements of the XEmacs Packaging System. Autoload cookies and defcustoms must also be added to existing libraries. The XEmacs Packaging System provides infrastructure to assure that all of these annoyances need only be dealt with once. The autoload cookies and defcustoms are beyond the scope of this chapter, but most maintainers of modern packages are already familiar with these mechanisms.

The XEmacs Packaging System may be divided into the infrastructure common to all packages, and the package-specific control files. The infrastructure supports global builds, installation, and generation of the “sumo” bundles of packages, as well as generation of individual packages. The package control files describe the structure of the package’s source tree and provide administrative information.

3.2.1 Infrastructure

In order to get the greatest benefit from the XEmacs Packaging System, a library maintainer should place the package sources in an appropriate place in the XEmacs source package hierarchy, and arrange to have the source package imported into the XEmacs CVS repository. (We realize that the latter requirement can be quite burdensome. We are working on ways to remove this requirement, but for the present it remains necessary.) The library maintainer must also keep sources for any packages his/her package requires. This requirement is somewhat burdensome, but unlikely to be relaxed because of the implementation of compilation of macros in Emacs Lisp. Macros cannot be called by compiled Lisp (the macro expansion, which is always known at compile time, is inlined), so the source of the macro must be loaded before compiling the called function.

The source package hierarchy may be rooted anywhere. The CVS module is called “packages,” so we will refer to the top directory of the source package hierarchy as “the ‘packages’ directory.” The ‘packages’ directory contains two source subdirectories, ‘xemacs-packages’ and ‘mule-packages’ (for convenience in segregating the packages which depend on Mule, as they will cause load-time errors in a non-Mule XEmacs). Each subdirectory contains many package source directories, whose internal structure is not specified. That structure is left up to the convenience of the library maintainers. The requirements on the top directory of an individual package source tree are given below, Control Files.

The ‘packages’ directory contains some auxiliary Lisp libraries used in the compilation and packaging process. The content of these libraries is of interest primarily to the packaging engineers, The Package Release Engineer View.

Finally, the ‘packages’, ‘packages/xemacs-packages’, and ‘packages/mule-packages’ directories contain ‘Makefile’s and include files to control the package creation process. The ‘Makefile’s in ‘packages/xemacs-packages’ and ‘packages/mule-packages’ simply define the default sets of known packages and include ‘../iterate.rules’, which implements recursive building of all target packages.

The ‘make’ infrastructure in ‘packages’ includes

‘Makefile’

controls building of individual packages, local installation, and bundling of “sumo” tarballs

‘iterate.rules’

controls recursive builds of multiple packages

‘meta-iterate.rules’

This is used by higher-level subdirectories that do not directly contain packages. Subdirectories directly containing packages should use iterate.rules instead.

‘XEmacs.rules’

provides the rules for building and packaging. Included by all package ‘Makefile’s.

‘Local.rules’

provides local configuration, such as installation targets and staging directories, as well as a number of kludges (many now obsolete) required for building packages on the Windows platform.

‘Local.rules.template’

a template for Local.rules, liberally commented

‘Local.rules.mk’

consistency checking for ‘Local.rules’, included by both the top-level ‘Makefile’ and by ‘XEmacs.rules’.

‘Local.rules.inc’

a file to include in package ‘Makefile’s to be able to get at variables in ‘Local.rules’ before including ‘XEmacs.rules’.

‘package-compile.el’

compile environment (e.g., load-path) setup.

Of these, only ‘Local.rules’ and ‘package-compile.el’ need to be modified by the library maintainer. The changes to Local.rules affect only your environment. This should need to be done only once when first preparing the source environment. The necessary modifications to ‘package-compile.el’ need to be done for each package and are discussed in the next section, Control Files.

3.2.2 Control Files

Each package source must contain a number of control files in the top-level directory. These files in general can be created and then ignored, except for a few variables that need to be updated when new versions are released. In most cases even adding, renaming, and removing library source files can be handled by generic rules.

The package control files include

‘Makefile’

Must set a few ‘make’ variables used by the administrative utilities, and defines a couple of package-building targets to depend on appropriate targets defined generically in ‘XEmacs.rules’. It may also provide various variables and rules to transform the source tree structure into that expected by the run-time system.

‘package-info.in’

Provides a template for package information to be provided to the administrative utilities. Static variables that are rarely changed (such as the package’s name) are entered as literals. Some variables are generated by the build process (build dates and MD5 checksums) and are automatically filled in. Finally, some variables that change irregularly (dependences and even version numbers) are set as ‘make’ variables in the ‘Makefile’.

‘ChangeLog’

Not strictly required, but normally a ChangeLog will be added by the XEmacs package maintainer if different from the upstream maintainer.

‘_pkg.el’

Generated. Simply does a package-provide for the package.

‘auto-autoloads.el’

Generated. Read when XEmacs is initialized, and provides autoloads for defuns and other forms in the sources that are marked with an autoload cookie (‘;;;###autoload’.

‘custom-loads.el’

Generated. Read when XEmacs is initialized, and informs the Customize subsystem how to find the defcustom forms needed to create Customization forms for the usre configuration variables of the package.

3.2.3 Obtaining the XEmacs Packaging System and Required Utilities

Currently both the infrastructure for creating XEmacs packages and the package sources themselves are available only by CVS. See http://www.xemacs.org/Develop/cvsaccess.html for more intformation.

The XEmacs Packaging System currently requires GNU ‘make’, and XEmacs, to build packages.

3.2.4 The Package Release Engineer View

The XEmacs Package Release Engineer is responsible for keeping the system coherent. The changes to ‘packages/package-compile.el’ and ‘packages/xemacs-packages/Makefile’ required to make the package available to others, and for building SUMO tarballs, etc, are done by the Package Release Engineer, not individual library maintainers.

The Package Release Engineer also maintains assorted infrastructure for actually making releases. These are generally available for inspection in the xemacs-builds module in the CVS repository.

Package Terminology:

3.2.5 Libraries and Packages

A Lisp library is a single loadable file containing Lisp code. It may be in source or byte-compiled form. A Lisp package is a set of one or more libraries, usually related to each other in some way, bundled with administrative information for convenient distribution.

3.2.6 Package Flavors

There are two main flavors of packages.

Regular Packages

A regular package is a set of Lisp libraries design to cooperate with one another. A very complex example is Gnus. One may not in general safely remove any of the component libraries.

Single-File Packages

A single-file package is a collection of thematically related but otherwise independent Lisp libraries. These libraries are bundled together for convenience of the maintainers. Usually individual libraries may be deleted at will without any loss of functionality of other libraries in the package. However, we would recommend that you follow this rule of thumb: "When in doubt, don’t delete". If it’s really that big a deal, request that the maintainers split the package into smaller aggregations.

3.2.7 Package Distributions

XEmacs Lisp packages are distributed in two ways. Binary packages are used by system administrators and end users. They are packaged in a form convenient for direct installation into an XEmacs package hierarchy. Source packages are for developers and include all files necessary for rebuilding byte-compiled lisp and creating tarballs for distribution or installation. This is all of the package author’s source code plus all of the files necessary to build distribution tarballs (Unix Tar format files, gzipped for space savings). (Occasionally sources that are not relevant to XEmacs are usually renamed to ‘file.upstream’.)

Currently, source packages are only available via CVS. See http://www.xemacs.org/Develop/cvsaccess.html for details.

The package distributions are also split according to major features required in XEmacs to support them. At present there are generic packages, which can be loaded by any XEmacs, and Mule packages, which require Mule support or they will cause errors when loaded. Note that there is no guarantee that a generic package will have any useful functionality in a minimally configured XEmacs. As long as any XEmacs can successfully load the package’s libraries (perhaps given other required Lisp libraries), it will be classified as generic. At the present time only Mule packages need be treated specially, and even those only if they contain multibyte characters.

Building Packages:

Currently, source packages are only available via anonymous CVS. See http://www.xemacs.org/Develop/cvsaccess.html for details of checking out the ‘packages’ module.

3.2.8 Prerequisites for Building Source Packages

GNU cp

GNU install

(or a BSD compatible install program).

GNU make

(3.79 or later preferred).

makeinfo

(4.2 from texinfo-4.2)

GNU tar

(or equivalent).

GNU gzip

(or equivalent).

A properly configured ‘Local.rules’ file.

Local.rules File.

And of course, XEmacs, 21.0 or higher.

3.3 What You Can Do With Source Packages

The packages CVS sources are most useful for creating XEmacs package tarballs for installation into your own XEmacs installations or for distributing to others.

It should be noted that most of the package ‘Makefile’s do not need to contain any target rules. Everything is handled from the ‘XEmacs.rules’ file, located in the toplevel directory of the packages source tree.

This document was generated by Aidan Kehoe on December 27, 2016 using texi2html 1.82.