TIP #453: Tcl Based Automation for tcl/pkgs
_____________________________________________________________________________
TIP:	453
Title:	Tcl Based Automation for tcl/pkgs
Version:	$Revision: 1.2 $
Authors:	Sean Woods <yoda@etoyoc.com>
State:	Draft
Type:	Project
Tcl-Version:	8.7
Vote:	Pending
Created:	Tue Sep 13 00:00:00 UTC 2016
Keywords:	Build tooling
Abstract
This TIP proposes replacing the make package process currently employed by the core with a Tcl-based build automation tool.
Background
[376] provides for the distribution of third party packages in the Tcl/Tk core distributions. To support that TIP currently requires three separate build automations: a Makefile based automation in Unix, and both an nmake and Makefile based automation for Windows. These automation systems can get out of sync and they assume that their job is to build dynamic libraries for local installation.
The Pitch
By the time make packages has run, the local Tcl interpreter has been built already. Rather than rely on delicate hacks and makefile tricks, core distributed packages could be built and installed via exec commands inside a Tcl script. In addition, this same automation could handle functions like injecting a core distributed package into a virtual file system, as well as bundling the Tcl/Tk library file system for [430].
The Implementation
For the practcl branch of tclconfig, I put together a 4000 line self-contained package and kit building library of tools. This library is TclOO based, and provides a rudimentary (but functional) system for templating C code in Tcl, as well as a build system that is capable of nesting sub-projects. It also steals the useful bits from the fileutil module of tcllib, providing implementations for concatenating files, performing file searches, and building a global package index from a soup of modules. The library also has a wrapper to download external sources from fossil. It also contains procs that can compile a static library, dynamic library, or self contained shell directly from exec calls.
I would propose that this tool system (or a new creation by the community in a similar spirit) be included in the library/ section of the tcl core. The provisional name for this tool set would be practcl. A version of this tool could also be provided in tcllib to allow 8.5 and 8.6 based cores to continue to build extensions.
In the new scheme, make packages (in all its forms) would be replaced with a call to "$(TCLSH) $srcdir/pkgs/make.tcl build". make packages-install would be replaced by a call to "$(TCLSH) $srcdir/pkgs/make.tcl install". For advanced users, these toplevel commands build and install will accept additional arguments. For instance, to install the core distributed packages into the VFS of a kit: "$(TCLSH) $srcdir/make.tcl install -destdir $MyVFS/lib".
pkgs/make.tcl
make.tcl would be maintained as part of the core, and provide the top-level control system to build, install, or repackage the core distributed extensions. That script will also provide mechanisms to populate the pkgs file system for developers who build the tcl core from fossil checkouts.
Commands:
basekit
Compile a ZipFs style basekit suitable for the wrap command.
build ?all? ?package? ?package...?
Compile the source code for core distributed packages into binary products (as applicable.) If all is given, an attempt is made to compile all packages under pkgs/. Any other argument is interpreted to be the name of an individual package to be compiled.
install ?-destdir destinationpath?
Install all core distributed packages locally. If -destdir is given, install the packages relative to destinationpath in the same way that "make DESTDIR= destdir" would. If -destdir is not given, or is an empty string, perform an install relative to the exec_prefix in tclConfig.sh
wrap exename vfspath ?dir...?
Generate a self contained executable constructed from the virtual file system amalgamated from vfspath and any other directories given as arguments. This VFS will automatically be populated with the library/ file system from Tcl.
distribution
> Download packages listed in the "packages.tcl" file, unpack their source code in the /pkgs folder, and perform any steps required to prepare those extensions for inclusion in a core snapshot for distribution.
developer
> Download packages listed in the "packages.tcl" file, unpack their source code in the parent folder to the one the core has been unpacked from, and perform any steps required to prepare those extensions to be compiled locally as part of a developer build. This is intended for developers who work from fossil checkouts of the Tcl core.
package-list
> Stream to stdout a list of all of the packages in the packages.tcl file, in a flat machine readable format.
Tk
This same mechanism will be adapted for Tk. Tk will be also provide a pkgs/ directory. Its base kits will be based on a modified wish instead of a modified tclsh.
Normal Operation
During the build/install/etc phase each directory will be scanned for either a "configure" file or a "prac.tcl" file. Standard TEA extensions will be detected by the presence of a "configure" file. The prac.tcl file is a hint to the build system that the package needs either additional instructions and guidance. The contents of the file are interpreted by the object which is implementing the extension's ambassador to the build system.
If one were to decide to bundle tcllib, and wished to exercise its SAK based installer the prac.tcl file would read:
# Implement the install routine for tcllib
#
oo::objdefine [self] 
  method install DEST 
    set pkg [my define get pkg_name [my define get name]]
    my unpack
    set prefix [string trimleft [my <project> define get prefix] /] 
    set srcdir [my define get srcdir]
    ::practcl::dotclexec [file join $srcdir installer.tcl] \par      -pkg-path [file join $DEST $prefix lib $pkg]  -no-examples -no-html -no-nroff \par      -no-wait -no-gui -no-apps

Maintaining the Package List
Each pkgs/ file system for both Tcl and Tk will also contain a file packages.tcl. This file will be human and machine readable. It contains a description of every core distributed package, where the sources can be found, as well as which fossil tags can be utilized as either development or release with this particular version of the core.
packages.txt contains a series of keywords populating a data structure.
A simple example would by the tclconfig templates from TEA:
EXTENSION tclconfig 
   tag  trunk
   fossil_url http://core.tcl.tk/tclconfig

The EXTENSION keyword is intended to take the following arguments:
name key/value-configuration-dict
Reserved keys
tag Source code management tag for the release bundled with this edition of the core
fossil_url If the extension is managed via fossil, a url that can be fed to "fossil clone". If no tag is specified "trunk" is assumed.
git_url If the extension is managed via git, a url that can be fed to "git clone". If no tag is specified "HEAD" is assumed
file_url If the extension is only available via source snapshot, a url where the file can be downloaded. Supported formats are .tar.gz and .zip.
The list is kept separate from the actual make.tcl so that users can simply steal the list for making "batteries included" distributions. It also allows the package list to remain distinct for each branch of the core.
Copyright
This document has been placed in the public domain.
_____________________________________________________________________________
Colophon
TIP AutoGenerator - written by Donal K. Fellows