10 Working with XEmacs Source Code Repositories.

This is part 11 of the XEmacs Frequently Asked Questions list. The primary purpose is advice on use of the version control systems used to keep the history of XEmacs development.

11.0: The XEmacs repositories

11.1: Working with CVS

11.2: Working with Mercurial

Where is the most recent XEmacs development code?

The most recent XEmacs development code is kept in a Mercurial repository, hosted by the Debian project. The read-only URL, for anybody who doesn’t intend to push upstream directly, is

http://hg.debian.org/hg/xemacs/xemacs

The read-write URL for committers is

ssh://sperber-guest@hg.debian.org//hg/xemacs/xemacs

Yes, Virginia, that doubled slash is correct.

See section How do I get commit access?.

See section What is Mercurial?.

Where is the most recent XEmacs stable code?

The most recent XEmacs stable code is kept in a Mercurial repository, hosted by the Debian project. The read-only URL is

http://hg.debian.org/hg/xemacs/xemacs-21.4

If you’re not Vin, you don’t need commit access. If you are Vin, you shouldn’t need to refer to this FAQ.

See section What is Mercurial?.

Where is the most recent XEmacs package code?

The most recent XEmacs packages code is kept in a CVS repository, hosted by the Debian project. The read-only CVSROOT, for anybody who doesn’t intend to push upstream directly, is

CVSROOT=:pserver:anonymous@cvs.alioth.debian.org:/cvsroot/xemacs

The read-write CVSROOT for committers is

CVSROOT=:ext:aliothuser@cvs.alioth.debian.org:/cvsroot/xemacs

where aliothuser is your account on alioth.debian.org. Then

cvs checkout packages

as usual. For more information, see XEmacs CVS Archive on the website.

See section How do I stay cool using CVS?.

See section How do I get commit access?.

Why isn’t package available? and what to do about it.

If a package isn’t available from the Packages repository, probably nobody has shown enough interest to add it yet. (Occasionally, there is a better package already in the XEmacs repository, of course.)

The first step is to ask about it, and propose addition, on the XEmacs Contributors list.

Most regular XEmacs contributors already shoulder primary responsibility for several packages, and contribute to maintenance of the rest, so you are unlikely to get a massively enthusiastic response unless you volunteer to become the maintainer of the version packaged for XEmacs yourself. The duties are not terribly onerous if you’re an active user of the package (xemacs-devguide)XEmacs Package Maintainer.

How do I get commit access?

To get commit access to XEmacs code, write to the XEmacs Review Board and request it. Once approved, for the development code, you also need to send Michael Sperber your SSH v2 RSA key (Alioth policy; v1 and DSA keys aren’t acceptable). A CC to the XEmacs Services team is a good idea, although not absolutely necessary. You should also get an Alioth account so that you can publish branches for review.

For packages code, you must get an Alioth account. Send your account name information to the XEmacs Services team.

The stable repository is gated; only the gatekeeper (currently Vin Shelton) has commit access. Patches for the stable repository should be submitted to XEmacs Patches, as usual.

XEmacs Mercurial Archive

XEmacs CVS Archive

See section How do I stay cool using CVS?.

See section What is Mercurial?.

How do I keep cool using CVS?

You don’t. CVS is just basically and in detail un-cool.

What would be really cool is if you would help us out in moving the packages repository to Mercurial. Volunteer on the XEmacs Contributors list. What’s needed is to figure out how to provide a one step checkout for the whole package hierarchy, while restricting commits to one package at a time.

For help using CVS, Google or ask on xemacs-beta@xemacs.org. Please update this FAQ with one or two of the best references you find.

What is Mercurial?

Mercurial is a distributed version control system, or DVCS. This means that versioning information can be easily exchanged between different lines of development, even if located on different hosts. In the older centralize version control system model, when you commit a change, it is immediately reflected in the public repository. In a DVCS, each user has a local repository, and the commit operation creates a version in that repository. To communicate with the public repository, a separate push operation must be executed. The DVCS model is more appropriate for open source development.

• The VCS model mirrors the development organization, where developers tend to work independently or in very small groups.
• Users without commit access can conveniently manage their local changes.
• Developers can work, and commit changes, while disconnected from the Internet. Then they merge and push their changes later.

Use of a DVCS does require some changes in workflow, but the XEmacs developers consider that inconvenience to be far more than balanced by the advantages.

Where do I get Mercurial?

Most OS distributions (including add-on distributions like Cygwin and MacPorts) include Mercurial packages. Of course, you can get the source distribution, as well as pre-built packages for most major platforms, from the Mercurial developers.

Do I really have to waste space on history?

Yes, you do. It’s really not that much, though. In one of my current workspaces, I see

XEmacs source files (and other cruft, such as editor backups)

115464KB

Build products

49676

Mercurial control files and history

25644

That really does include all of the history available in the main XEmacs development branch, and the build products are near twice the size of all of the Mercurial-specific information.

hg diff gives bizarre output.

You may see an unreasonable diff (often large) that doesn’t seem to reflect your work.

This is usually due to using hg diff on a merge commit. That means the commit has multiple parents, and joins together two lines of development that occurred concurrently.

You’re diffing against the "wrong" one; try the other one. You get the relevant revision number or ID from hg log. In more detail:

When there is a merge in Mercurial, it will often be the case that one of the parents is the immediate predecessor of the merge commit. hg log will report something like

    changeset:   4789:56049bea9231    # revision D, below
    parent:      4788:5cca06f930ea    # your commit
    parent:      4787:6e6f7b79c1fc    # diff against this
    user:        you (or somebody else)

    changeset:   4788:5cca06f930ea    # revision B, below
    parent:      4760:217abcf015c4    # revision A, below
    user:        you

    changeset:   4787:6e6f7b79c1fc    # revision C, below
    parent:      4786:d6cfba1cc388
    user:        somebody else

Note that the divergence took place a long time ago (r4760). It’s natural to diff against (tip - 1), in the example above, hg diff -r 4788. But this will give unexpected output!

A picture of this history looks something like

      B --- D
     /     /
    A ... C

where A is the common ancestor, B is the commit you did, C is the mainline at the time of the merge, and D is the merge commit. The three dots between A and C can represent many commits, and a lot of work. Given no conflicts in the merge, hg diff -r C -r D is the same as hg diff -r A -r B, i.e., it shows your work. Similarly, hg diff -r B -r D is the same as hg diff -r A -r C. This latter diff is likely to be quite large, and it doesn’t show your work. Unfortunately, that is the typical result of diffing against the "previous" commit.

How do I recover from a bad commit? (I already pushed.)

Once upon a time, an XEmacs developer wrote:

> GAAAAK! What’s the best way to restore ChangeLog and its history?

He had just inadvertently pushed a commit which deleted ‘src/ChangeLog’! The history is still there, not to worry. (In this case, another developer had restored src/ChangeLog already.) The best way depends on a number of things. First, let’s look at the log and the state of the DAG (the graph of commits). Here’s the log, formatted somewhat differently from the usual output for compactness.

5025    anne    Restore src/ChangeLog.
5024    barb    merge
        parents: 5023 5010
5023    barb    Error-checking.
5020    barb    merge
        parents: 5019 5006 
5019    barb    Fix non-Mule build.
5011    barb    Some internals-manual updates.
        parents: 5002
5010    cary    Windows fixes for Visual Studio 6.
        parents: 5008 5009
5009    cary    Miscellaneous small fixes to Windows VS6 build.
        parents: 5006
5008    dana    Add license information.
5007    dana    Relicense emodules.texi.
5006    cary    Instantiate compile fix for nt.c.
5005    edna    Cast correctly.
5003    edna    #'union doesn't preserve relative order
5002    barb    Fix some compile bugs.

(The gaps at 5003...5005, 5011...5019, and 5020...5023 are filled with sequences of commits by the same developers.) Let’s visualize this as a graph. Time increases to the right, the leading "50" is omitted for brevity, and the dotted links indicate that several irrelevant commits were omitted, also for brevity.

                        ,------ 09 -----.
                       /                 \
02 --- 03 ... 05 --- 06 --- 07 --- 08 --- 10 --- 24 --- 25
  \                    \                        /
   `-- 11 ... 19 -------`-- 20 ... 23 ---------'

The "problem commit" is 5010, which merges 5008 with 5009, and somehow managed to "lose" ‘src/ChangeLog’. The unobvious consequence is that, although the other changes made in 5007 and 5008 were successfully merged and are present in 5010, the log entry made by Dana for 5008 "just disappeared". (The log entry for 5007 is in a different ‘ChangeLog’, so it’s safe.)

The safe and simple way for Cary

To recover state file-by-file (also for whole directories), use hg revert. This does not change the "current" version, i.e., the commit that will be the parent for your next commit.

If it’s not a merge commit, it’s simple to restore the ChangeLog. It’s best to do it before making any other commits in your own workspace, and before pulling in new commits from others. If there are a lot of such commits in your workspace already, ask for help. But in this case, there was no such problem. Just

hg revert -r 5009 src/ChangeLog
# Add Dana's log entry by hand.
hg commit -m "Restore src/ChangeLog."

5009 is the revision id of the most recent commit that had the correct version of the file. You get that from the "parent" field in hg log, or from the DAG browser (hg view, requires hgk extension enabled).

Alternatively, Cary could revert from 5008. This would leave her with her log entry for 5009 missing, and that would have to be added by hand.

Note that in the actual history, Cary didn’t realize that Dana’s log went missing, so Anne had to pick up the slack in 5025.

Recovery by another developer

Another way to recover earlier state is with hg checkout (or hg update, which is another way to spell the same command). This changes the version that hg sees as "current", as well as reverting the workspace.

A common scenario is that another developer, such as Barb in the log above, was already working on ‘src/ChangeLog’, saves her copy, then tries to merge. She would then get a modify/delete conflict. It’s tempting to just resolve that in favor of keeping the file, and commit. This often works, but an alternative way uses the VCS:

hg checkout 5010
hg revert -r 5009 src/ChangeLog
# Add Dana's log entry by hand.
hg commit -m "Restore src/ChangeLog."

to get the same effect as described above, then

hg merge

(making her changes "float to the top" of the log) or

hg checkout 5023
hg merge

(putting the Cary’s branch at the top of the log). This assumes she has no other heads in her workspace. If she does have other heads she would have to use an explicit argument to hg merge.

Note that in the actual history, Barb didn’t realize that Dana’s log went missing, so Anne (or somebody) had to pick up the slack in 5025.

The hard but accurate way

Suppose Barb did hg pull -u, but notices the problem before resolving conflicts and committing the merge. Assume Barb was fully committed before doing hg pull -u.

# Restore the ChangeLog, "covering up" the broken commit.
# Check out Cary's head.  This nukes the merged files in the workspace,
# but the history and versions in Barb's rev. 5023 are preserved
# in the repository.  The -C is necessary to overwrite files.
hg checkout -C 5010
hg revert -r 5009 src/ChangeLog
# Merge Dana's branch (yes, again).
# The repeated merge outside of src/ChangeLog should resolve to a
# no-op, but the ChangeLog probably conflicts.
# The -f is needed because revert leaves uncommitted changes.
hg merge -f 5008
hg commit -m "Re-merge Dana's branch to recover her logs."
# Merge Barb's work.
# If Barb has only two heads, which seems likely, the argument to
# merge is optional.
hg merge 5023
hg commit -m merge

Visualizing this with a graph, we have:

                        ,------ 09 -----.
                       /                 \
02 --- 03 ... 05 --- 06 --- 07 --- 08 --- 10 *** 24 --- 25
  \                    \             \          /      /
   \                    \             `--------'      /
    \                    \                           /
     `-- 11 ... 19 -------`-- 20 ... 23 ------------'

Note that the versions 5024 and 5025 in this graph denote different versions from the actual history. The starred link means that editing work (aside from resolving conflicts) was done, on top of the merge. However, the editing work is actually done by Mercurial (the revert command)!

How do I recover from a bad commit? (I haven’t pushed yet.)

If you hadn’t yet pushed the commit you now regret, and realize it before doing further commits, you can use hg strip tip. Then just redo the commit, possibly with additional changes before committing.

hg strip is dangerous; for practical purposes it destroys history, and it also reverts the files in your workspace. It’s probably possible to recover the history, but I don’t know how. And any uncommitted changes that might be lost are gone forever. However, it is useful in cases like this.

When in doubt, use the safer method How do I recover from a bad commit? (I already pushed.).

Testing patches with Mercurial Queues.

When testing a patch proposed on xemacs-beta or xemacs-patches, conflicts or new heads often appear later, when using hg pull -u.

There are both theoretical and practical reasons why this happens, and it’s unlikely to change. The current workflow of XEmacs is also unlikely to change soon; testing patches is also probably going to remain necessary. One way to avoid this issue is to use Mercurial Queues (mq), an extension distributed with Mercurial.

Enable mq by adding

    [extensions]

    hgext.mq =

to your ‘~/.hgrc’. (Yes, the right hand side is empty.) If you already have an [extensions] section, don’t repeat it. Add hgext.mq = to the existing extensions section.

When you want to test a patch, you need an hg workspace with no uncommitted changes. If you already have some uncommitted changes, you can preserve them with mq as follows:

    $ hg qnew -f -m "Preserve local changes." local-changes

The -m flag specifies the commit message for the new patch. The -f flag "forces" qnew to put all of the uncommitted local changes into an mq patch, and commits it (you will see a commit with summary "Preserve local changes." if you do an hg log now). "local-changes" is the name of the patch.

Now, create an mq patch for the test patch (which we assume was saved to ‘/tmp/xemacs.patch’):

$ hg qimport -P -n test-xemacs-patch /tmp/xemacs.patch

The -n flag specifies the name of the patch. Give it a name sufficiently explicit so you’ll know what it is later. Remember, it may take several weeks for the patch to be pushed to the public mainline. The -P flag says "apply this patch to the workspace now".

When you want to update the workspace, you need to remove the mq commits, update, and restore your local changes and the test patch. You do it this way:

    $ hg qpop --all
    $ hg pull -u                # use your usual method, hg fetch etc.
    $ hg qpush --all

hg qpop --all undoes all the mq commits, but leaves the patches in ‘.hg/patches’. hg qpush --all reapplies the patches and restores the mq commits. Of course you hope that the patch will be committed upstream. When it is, you do this:

    $ hg qpop --all
    $ hg pull -u
    $ hg qdelete test-xemacs-patch
    $ hg qpush --all

and you’re back in business with the official version of the patch you tested, and all your local changes applied.

It’s also possible to split your local changes into smaller mq patches, but that’s out of scope for this answer.

This document was generated by Aidan Kehoe on December 27, 2016 using texi2html 1.82.