Lisp Engine Changes

Owner: ???

Effort: ???

Dependencies: ???

Abstract: Recently there has been a great deal of talk on the XEmacs mailing lists about potential changes to the XEmacs Lisp engine. Usually the discussion has centered around the question which is better, Common Lisp or Scheme? This is certainly an interesting debate topic, but it didn't seem to have much practical relevance to me, so I vowed to stay out of the discussion. Recently, however, it seems that people are losing sight of the broader picture. For example, nobody seems to be asking the question, "Would an extension language other than Lisp or Scheme (perhaps not a Lisp variant at all) be more appropriate?" Nor does anybody seem to be addressing what I consider to be the most fundamental question, is changing the extension language a good thing to do?

I think it would be a mistake at this point in XEmacs development to begin any project involving fundamental changes to the Lisp engine or to the XEmacs Lisp language itself. It would take a huge amount of effort to complete even part of this project, and would be a major drain on the already-insufficient resources of the XEmacs development community. Most of the gains that are purported to stem from a project such as this could be obtained with far less effort by making more incremental changes to the XEmacs core. I think it would be an even bigger mistake to change the actual XEmacs extension language (as opposed to just changing the Lisp engine, making few, if any, externally visible changes). The only language change that I could possibly imagine justifying would involve switching to some ubiquitous web language, such as Java and JavaScript, or Perl. (Even among those, I think Java would be the only possibility that really makes sense).

In the rest of this document I'll present the broader issues that would be involved in changing the Lisp engine or extension language. This should make clear why I've come to believe as I do.

Is everyone clear on the difference between interface and implementation?

There seems to be a great deal of confusion concerning the difference between interface and implementation. In the context of XEmacs, changing the interface means switching to a different extension language such as Common Lisp, Scheme, Java, etc. Changing the implementation means using a different Lisp engine. There is obviously some relation between these two issues, but there is no particular requirement that one be changed if the other is changed. It is quite possible, for example, to imagine taking the underlying engine for any of the various Lisp dialects in existence, and adapting it so that it implements the same Elisp extension language that currently exists. The vast majority of the purported benefits that we would get from changing the extension language could just as easily be obtained while making minimal changes to the external Elisp interface. This way nearly all existing Elisp programs would continue to work, there would be no need to translate Elisp programs into some other language or to simultaneously support two incompatible Lisp variants, and there would be no need for users or package authors to learn a new extension language that would be just as unfamiliar to the vast majority of them as Elisp is.

Why should we change the Lisp engine?

Let's go over the possible reasons for changing the Lisp engine.

Speed. Changing the Lisp engine might make XEmacs faster. However, consider the following.
1. XEmacs will get faster over time without any development effort at all because computers will get faster.
2. Perhaps the biggest causes of the slowness of XEmacs are not related to the Lisp engine at all. It has been asserted, for example, that the slowness of XEmacs is primarily due to the redisplay mechanism, to the handling of insertion and deletion of text in a buffer, to the event loop, etc. Nobody has done any real studies to determine what the actual cause of slowness is.
3. Emacs 18 seems plenty fast enough to most people. However, Emacs 18 also had a worse Lisp engine and a worse byte compiler than XEmacs.
4. Significant speed increases in the execution of Lisp code could be achieved without too much effort by working on the existing byte code interpreter and function call mechanism a bit.
Memory usage. A new Lisp engine with a better garbage collection mechanism might make more efficient use of memory; for example, through the use of a relocating garbage collector. However, consider this:
1. A new Lisp engine would probably have a larger memory footprint, perhaps a significantly larger one.
2. The worst memory problems might not be due to Lisp object inefficiency at all. The problems could simply be due mainly to the inefficient buffer representation. Nobody has come up with any concrete numbers on where the real problem lies.
Robustness. A new Lisp engine might well be more robust. (On the other hand, it might not be. It is not always easy to tell). However, I think that the biggest problems with robustness are in the part of the C code that is not concerned with implementing the Lisp engine. The redisplay mechanism and the unexec mechanism are probably the biggest sources of robustness problems. I think the biggest robustness problems that are related to the Lisp engine concern the use of GCPRO declarations. The entire GCPRO mechanism is ill-conceived and unsafe. The only real way to make this safe would be to do conservative garbage collection over the C stack and to eliminate the GCPRO declarations entirely. But how many of the Lisp engines that are being considered have such a mechanism built into them?
Maintainability. A new Lisp engine might well improve the maintainability of XEmacs by offloading the maintenance of the Lisp engine. However, we need to make very sure that this is, in fact, the case before embarking on a project like this. We would almost certainly have to make significant modifications to any Lisp engine that we choose to integrate, and without the active and committed support and cooperation of the developers of that Lisp engine, the maintainability problem would actually get worse.
Features. A new Lisp engine might have built in support for various features that we would like to add to the XEmacs extension language, such as lexical scoping and an object system.

Why would we want to change the extension language?

Possible reasons for changing the extension language include:

More standard.

Switching to a language that is more standard and more commonly in use would be beneficial for various reasons. First of all, the language that is more commonly used and more familiar would make it easier for users to write their own extensions and in general, increase the acceptance of XEmacs. Also, an accepted standard probably has had a lot more thought put into it than any language interface created by the XEmacs developers themselves. Furthermore, if our extension language is being actively developed and supported, much of the work that we would otherwise have to do ourselves is transferred elsewhere.

However, both Scheme and Common Lisp flunk the familiarity test. Neither language is being actively used for program development outside of small research communities, and few prospective authors of XEmacs extensions will be familiar with any Lisp variant for real world uses. (I consider the argument that Scheme is often used in introductory programming courses to be irrelevant. Many existing programmers were taught Pascal in their introductory programming courses. How many of them would actually be comfortable writing a program in Pascal?) Furthermore, someone who wants to learn Lisp can't exactly go to their neighborhood bookstore and pick up a book on this topic.
Ease of use.

There are endless arguments about which language is easiest to use. In practice, this largely boils down to which languages are most familiar.
Object oriented.
The object-oriented paradigm is the dominant one in use today for new languages. User interface concepts in particular are expressed very naturally in an object-oriented system. However, neither Scheme nor Common Lisp has been designed with object orientation in mind. There is a standard object system for Common Lisp, but it is extremely complex and difficult to understand.

Ben Wing

Conform with <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
Automatically validated by PSGML