Best Practices for Developers
The objective of this page is to bring together best practices for developers working on pkg, meant to enhance its supportability and usability. This document does not intend to be a comprehensive guide to the processes and tools used to assemble and deliver packages.
Table of Contents
There are two build frameworks currently in use for pkg(5): The Make framework, and the distutils framework. The Makefile provides a wrapper around the distutils setup.py, adding functionality that is specific to Solaris. Anything that is intended to be cross-platform is implemented using the distutils framework. Solaris-specific functionality (e.g. ability to convert a host of SVR4 packages into pkg(5) packages) is implemented in additional Makefiles and is then called by the top-level Makefile. Unless you have a specific reason to use the Make framework, you should use the distutils framework, even when building on Solaris.
- distutils (setup.py)
This framework is based on Python's Distutils framework, and provides a platform-independent interface for building and distributing pkg(5). To use this framework, you must have a Python 2.4 distribution installed on your build machine. The following commands assume the python command is accessible (e.g. in your $PATH).
- python setup.py --help-commands - Command help information, including the pkg(5)-specific command additions. Note that most of the generic distutils commands are useful to the pkg(5) developer.
- python setup.py build - Builds everything, including binary modules, C extensions, and front-end CLIs. The results are placed in ../proto/build_<os>.
- python setup.py install - Does a build, builds dependencies and then creates a prototype area in ../proto/root_<os> containing a hierarchy of files intended to represent what pkg(5) looks like when installed on a particular OS.
- python setup.py bdist - Creates a binary distribution of the resulting build. This is in most cases a simple zip or gzipped tarball of the relocatable pkg(5) binaries.
- python setup.py clean - Un-does what --build did: Removes binary artifacts.
- python setup.py clobber - Un-does what make --install did. This removes the prototype area and the dependencies.
- python setup.py lint - Runs the pkg(5) C Extension source code through the lint checker. In addition, runs the pkg(5) pure Python source code through pylint. It is assumed pylint has been installed into the python installation area site-packages).
- python setup.py test - Runs unit and functional tests. This should be run as root, e.g., with pfexec or sudo.
To use the make framework to build pkg(5), one must build on a machine installed with Solaris. The following main make targets are available:
- make link - Connects your local machine to the current repository's working copy's version of the commands, modules, and supporting files.
- make link-clean - Un-does what make link did
- make all - Does a setup.py build, followed by builds of Solaris-specific modules.
- make install - Does a setup.py install, followed by installs of Solaris-specific modules such as the packagemanager GUI, update notifier, brand, and the miscellaneous utilities.
- make clean - Un-does what make all did: Removes binary artifacts.
- make clobber - Un-does what make install did: Does a make clean followed by removing the prototype files
- make test - Runs unit and functional tests. This should be run as root, e.g., with pfexec or sudo.
- make test-verbose - Same as make test, only with more output useful for debugging unit tests themselves. ==== The following makefile policy is in effect
- An incremental build when nothing has changed must do nothing except traverse makefiles. Like this:
/usr/bin/python setup.py build
- This also applies to "make install" and other targets.
- If any portion of a build fails, make must exit w/ a non-zero exit status.
- clobber and clean should work properly:
- clean should delete *all* artifacts of the build that occur in the source hierarchy (like .o files).
- clobber should depend on clean, and should destroy from the proto area all components which are placed there by "install"
- Test each target with a clean workspace (i.e. hg clone your workspace to get a virgin one, and test builds there). Test install targets by deleting the proto area and rebuilding.
Multi-Platform Best Practices
pkg(5) was originally designed for use with Solaris. However, the underlying concepts and usefulness apply more broadly. This fact, coupled with the fact that pkg(5) is currently implemented using a mostly cross-platform language (python) has led the team to produce and maintain pkg(5) in a portable manner for use in projects like Update Center, which is developing tooling based on a multi-platform pkg(5). This has a non-trivial impact to pkg(5) developers, who must take care when evolving pkg(5) to do so in a way that allows pkg(5) to continue to be portable to other platforms.
This does not mean platform-specific characteristics and features cannot be used in pkg(5). For example, Solaris' ZFS is used to implement some or all of pkg(5)' rollback capability. However, it does mean the implementation must either make an attempt to implement a degraded version of rollback on when not used on Solaris, or the feature must be gracefully disabled and appropriate warnings issued when the feature would normally be employed.
Writing portable Python
The following notes and best practices aid the pkg(5) developer in keeping pkg(5) portable.
- Best Practice: Avoid non-portable APIs
Python has a number of useful libraries that can be sourced when looking for an implementation. Unfortunately, many of them, especially those at the OS/file layer, are not portable. When code containing calls to these APIs is encountered on platforms where the APIs are not available, an ugly error occurs, and system integrity may even be compromised. Therefore, it is important to avoid these APIs whenever possible. When an API is only available on a subset of major platform types, it is usually denoted in the API Documentation. For example, see the Python API Documentation for the os module for several examples of non-portable APIs. Look for notes like "Availability: Macintosh, Unix" (which is an example of an API not available on Microsoft Windows)
- Best Practice: Use pkg.portable module for platform-specific functionality
If it is imperative that you use a platform-specific API, a special OS abstraction layer is present in pkg(5) which defines, in abstract terms, operations that are typically implemented using platform-specific methods. At runtime, pkg(5) selects which underlying implementation of this interface to load, based on the runtime platform.
This allows developers to keep platform-specific code cordoned off in specific modules that are only loaded when a compatible platform is in use. Platform-neutral code (that which makes up the bulk of pkg(5)) can call these interfaces through this abstraction layer. See the documentation present in the pkg.portable module for more information.
- Best Practice: Be aware of variations in return values
Some Python operating system-level library routines have different return values on different platforms under certain error conditions. For example, when os.rmdir is called on a existing directory that contains files, Solaris returns errno.EEXIST, but Linux returns errno.ENOTEMPTY. Another example is when shutil.rmtree is called on a non-existent directory. Solaris return errno.ENOENT, but Windows returns errno.ESRCH. When calling these methods, try to copy code from another place within pkg(5) that is already doing the same thing. It probably has the error checking correct already.
- Best Practice: Avoid filesystem specifics
The concept of filesystems is pretty universal when it comes to the platforms pkg(5) is intended to support. However, there are some differences, especially between Microsoft Windows and other OS's. In particular, the concepts found in the filesystem implementations are "similar but different". For example, Windows has the concept of a drive (e.g. "C:\"), which is not found on Unix. Also, filesystem separators tend to be different (however over time Windows has evolved to recognize and properly deal with "/" in many cases).
When using python to explore the filesystem and construct paths, if you have code that deals with the root of the filesystem in any way, be sure to account for the fact that the root looks different on Windows, and that one cannot assume that "/" is the root of the filesystem. Python has the ability to discover the "drive" portion of paths, using os.path.splitdrive(). This API is available on all platforms, but returns '' (empty string) when not on Windows.
Path separators - Unix uses "/", Windows uses "\". In general, python APIs can deal with either one, regardless of platform. So, things like os.path.exists("%s/%s/%s", d1, d2, file) generally work as expected. However, be careful when comparing filesystem paths using string operations. Doing things like mypath.split("/") is bad, since mypath, if representative of a path to a file or directory, would likely be in its platform-specific representation. (use os.path.pathsep, or even better, split it up using the os.path.split API). In general, be aware that processing paths using string functions is dangerous and should be avoided.
- Best Practice: Avoid APIs with behavioral differences
Python has many cross-platform APIs that are universally available. However, many of them behave differently (usually because the underlying OS implementation is different). For example, os.unlink() deletes files, but on Windows, if a file is "in use", this API throws an exception. It's certainly possible to write code to properly deal with these, but in general it's best to avoid it, to minimize the bugs and maximize portability. Consult python documentation for more information on behavioral differences such as these.
- Best Practice: Use EnvironmentError If you look at python's built-in exceptions, there are a few that are worth mentioning in the context of writing portable code. Take a look at this code:
f = file("%s/filters" % some_path), "r")
except IOError, e:
if e.errno != errno.ENOENT:
- In this example, python's built-in file() function is used. The documentation for this API states "If the file cannot be opened, IOError is raised". However, on Windows, if the file cannot be opened, an OSError is raised. It's unfortunate that the python documentation is ambiguous in this regard, so when dealing with failures relating to files, it is best to use EnvironmentError.
- 'Best Practice: Test C Extensions on Linux'
pkg(5) currently contains some python extensions written in C: the arch extension for getting access to lower-level Solaris platform characteristics, and the elf extension for reading information from within ELF files. The elf C code is portable to other Unix operating systems which use the ELF format (e.g. Linux), using , an open source library for accessing ELF libraries. However, there are still some differences. For example, the EM_AMD64 constant is EM_X86_64 when using libelf. In addition, the routines used to calculate SHA-1 hashes come from a built-in library (libmd) on Solaris, but must use OpenSSL on other platforms. These differences manifest themselves as several #ifdefs in the C extensions. If you are making changes to this code, it is best to test on a Linux platform using this library.
- 'Best Practice: write relocatable code'
As noted earlier, pkg(5) was originally developed for use on Solaris, with some bootstrapping assumptions about its installed location. However, modern pkg(5) now makes no absolute assumptions about its installed locations, and makes references to its needed external files using relative references or paths passed in from the external user. Therefore, you should never make assumptions about the installed location of pkg(5) itself. It may be completely installed and isolated in x:\Program Files\Sun\MessageQueue\pkg, or it may be installed into /usr. pkg(5) code that requires external resources should always make relative references to its files, based on the known location of the package front-end at runtime, and its location relative to the needed files. This would also apply to any future library-level interface, using some kind of $ORIGIN interface for location discovery.
- 'Best Practice: Write unit tests to test known platform differences'
If you develop new functionality that has known differences between platforms, make sure to supply new unit or other tests that test that the expected differences have been properly dealt with.
In addition to python APIs, the use of external resources for testing purposes should be limited to those that exist across all platforms, when required. For example, do not assume the presence of "/bin/ls" for tests that test functionality that is expected to work on Windows.
As noted above, at a bare minimum, testing involves running the supplied test targets (using the appropriate build infrastructure). Ideally, this would be executed on all platforms on which pkg(5) is expected to operate. Generally, the Solaris developers that work on pkg(5) only test on Solaris. Other teams are responsible for porting pkg(5) to other platforms and for ensuring testing is performed on other platforms.