Module Life Cycle

If you have followed this series since the beginning, then you have a very good understanding of modules by now. You know what they contain, how they are discovered, how they relate to one another, and so forth. But how does a module become active in the first place? What is the path that a module follows from inception to activation?

Let us now turn our attention to the final topic in this trail, the module life cycle, which we will consider relative to development using the Avail workbench^[1]. Please turn your attention to the bottom of the following diagram. (Unless your display is 3 meters high, then very likely you will need to scroll down…)

Launch the Avail workbench.

The entire process begins when the programmer launches the workbench. Running the workbench, or some other Avail development environment, is a prerequisite for Avail development. When the workbench begins, no modules have been loaded into the Avail run-time system. This means that no code is present and no commands can be issued.

Edit some source modules.

The programmer edits some modules using her favorite editor. After she is ready to try her changes, she saves the files and then activates a modified module (or a module that is downstream of one that she has changed).

Activate a module from one of the tree views.

When the programmer is ready to run Avail code or attempt to compile some newly changed modules, then she uses one of the two tree views to select a module for activation. This is an extremely complex process for the system, but automatic for the programmer.

Are any modules loaded already?

Immediately upon activating a module, the system must decide how to proceed based upon whether any modules are already loaded. The system may need to unload these before the selected module can be fully activated. If modules are already loaded, then the system examines them in arbitrary order^[2] to see if their corresponding source modules have changed since they were loaded. If no modules are loaded, then the system begins tracing the dependencies of the selected module.

Begin processing the next loaded module.

There are still loaded modules that have not been processed yet, so begin processing the next one.

Has this loaded module been modified since it was loaded?

The programmer may have changed the source for this module since it was loaded, so determine whether this is the case. The system first checks the last modification time of the source module. If the last modification time equals the previous value (if any) recorded for this module, then the system assumes that the source module has not changed. If the last modification time is different, then produce a cryptographic hash of the source module's content. If this hash matches the one associated with the loaded module, then the system assumes that the source module has not changed.^[3] If the source module has not changed, then ignore it; see if there are more loaded modules to process. If the source module has changed, then mark it as dirty and place it into a queue of dirty loaded modules.

Mark this module as dirty.

The loaded module has changed since it was loaded, so mark it as dirty and enqueue on the dirty queue it for eventual processing.

Are there still loaded modules to process?

If there are still loaded modules to process, then move on to the next one. If not, then determine whether the dirty queue contains anything.

Are there any loaded modules in the dirty queue?

If there are any loaded modules in the dirty queue, then it will be necessary to unload these modules (and their downstream dependents) and reload or recompile them.

Begin processing the next loaded module from the dirty queue.

There are still dirty modules that have not been processed yet, so begin processing the next one.

Mark this module's immediate dependents as dirty.

Propagate the dirtiness of the loaded module to its immediate dependents, i.e., those modules that name it as an import target: mark each dependent as dirty and place it into the dirty queue iff it has not been placed into it since the selected module was activated.

Are there still loaded modules on the dirty queue?

If there are still loaded modules on the dirty queue, then dequeue the next one. If not, then topologically sort the dirty modules in preparation for unloading them.

Topologically sort the dirty modules.

The dirty queue is now empty, but there are loaded modules that are marked as dirty. Sort these into a topological order such that the modules furthest downstream will be processed first and the modules furthest upstream will be processed last. Let us call this ordering the unload queue. Now begin processing the first module in the unload queue.

Begin processing the next loaded module in the unload queue.

There are still dirty modules that have not been processed yet, so begin processing the next one.

Unload this module.

Unload the module from the Avail run-time system. Run all unload reactors registered by sends of "After the current module is unloaded,⁇do_" that were encountered during compilation of the module. The unload reactors are run in reverse registration order. Once all unload reactors have completed running, then remove all method features defined by the doomed module. This is safe, because all dependents of this module have already been unloaded, and these features were only visible in the dependents. Any values created by this module that are held transitively by upstream modules continue to exist.

Are there still loaded modules in the unload queue?

If there are still loaded modules on the unload queue, then dequeue the next one. If not, then unloading is now complete. Any module whose ancestors (including itself) did not change since the previous activation remains loaded. Now trace the ancestry of the activated module.

Trace the ancestry of the activated module.

Recursively explore the ancestry of the activated module. The ancestry of a module comprises itself and every module that is recursively upstream of it. These are the exact modules — called ancestors — upon which the activated module depends for the names and features that it uses. During tracing, only the module headers of the ancestors are examined, not the module bodies. If any errors are encountered during the trace, such as a malformed import specification or a cyclical dependency between two modules, then the trace is aborted: activation has failed and appropriate error messages are written to the workbench's transcript. If no errors are encountered, then topologically sort the ancestors in preparation for activation.

Topologically sort the ancestors.

Sort the ancestors of the selected module into a topological order such that the modules furthest upstream will be processed first and the selected module, which is furthest downstream, will be processed last. Call this ordering the activation queue. Now begin processing the first module in the activation queue.

Begin processing the next ancestor in the activation queue.

There are still modules in the activation queue, so begin processing the next one.

Is this ancestor already loaded?

If this ancestor is already loaded into the Avail run-time system, then ignore it and move along. If no, then determine whether it has a relevant existing compilation.

Is this ancestor already compiled?

Each module root has an associated binary module repository that contains compilations of its enclosed modules. Based on the timestamp and cryptographic content hash of the source module, search the appropriate repository for a prior compilation. The repository stores several prior compilations of the source module, corresponding to various versions of the content. If an appropriate binary module already exists, then load it. If not, then compile the source module.^[4]

Load an existing compilation of this ancestor.

Load the appropriate existing compilation of the ancestor. This process behaves as if each top-level statement were executed in lexical order. If any top-level statement raises an exception, then abort the load: activation has failed and appropriate error messages are written to the workbench's transcript. Otherwise, the module's services are now available to downstream modules and its entry points are now available to the programmer. Determine whether there are more modules in the activation queue.

Compile this ancestor.

Compile the ancestor. This also loads the module into the Avail run-time system. If the compiler encounters any errors while parsing, evaluating semantic restrictions, or executing top-level statements for side-effect, then abort the compilation: activation has failed and appropriate error messages are written to the workbench's transcript. Otherwise, the module's services are now available to downstream modules and its entry points are now available to the programmer. Persistently store the binary module produced by this compilation into the appropriate binary module repository.

Store the ancestor's new compilation into a repository.

Determine the appropriate binary module repository, based on the ancestor's enclosing module root, and store the binary module created by compilation into it. This will speed up subsequent activations of the ancestor, even across an exit and restart of the workbench. Then determine whether there are more modules in the activation queue.

Are there still modules in the activation queue?

If there are still modules in the activation queue, then dequeue the next one. If not, then activation has succeeded: the selected module and all of its ancestors have been activated.

The selected module has been activated.

The selected module and its ancestors have been activated. Their services are now available to future downstream modules, and their entry points are ready to be run. The programmer can now interact with the Avail run-time system to her heart's content. When she is satisfied, then she can go back to creating or editing source modules.

The selected module could not be activated.

The system encountered an error while tracing, loading, or compiling the selected module or an ancestor. The programmer can now analyze the error messages that were written to the workbench's transcript and modify her code accordingly to address the issues discovered during the activation attempt.

Unload Reactors

Executing certain top-level statements — such as sends of "_is a new test suite" and "Test_in_«,⁇if supported«on this platform»⁇,⁇»?is_«must raise_»", each defined by Avail's testing framework — may update global data structures stored in module variables. In order to ensure correct operation of all entry points when the Avail run-time system is in flux because of an explicit activation or unload, side-effects like these should be reversed during the unloading of the module that caused them.

An unload reactor is a function that is registered with "After the current module is unloaded,⁇do_". An unload reactor typically performs an operation that reverses the side effects of previous operations, so that module variables remain consistent across repeated activations and deactivations of modules. A module's unload reactors are run, in reverse registration order, strictly before any method features introduced by the module have been removed. It is therefore safe to use methods and method features defined by the doomed module.

Please be aware that the activation process is highly parallel, and several modules may undergo unloading simultaneously. If these modules have registered unload reactors that share access to common resources, such as module variables, then contention for these resources may occur. To avoid race conditions and possible data loss, access to these shared resources must be synchronized.

Consider the following example from /avail/Avail/Unit Testing/Definers:

Without delving too deeply, Remember the current module records the module that is undergoing activation into a module variable, modules.^[5] Lines 68-75 register an unload handler that removes the module, captured into the local constant currentModule, from modules. This prevents the unit testing framework from tracking the module after it is unloaded, thereby ensuring that the module is safe to reactivate.

[1] This process is equally valid from the perspective of some other Avail development tool.

[2] The system actually processes the loaded modules in parallel, since it is safe to do so in this circumstance.

[3] In the event that you encounter a cryptographic hash collision in your project, then there are much more serious issues than whether the system correctly detects whether your source module has changed!

[4] This process is an optimization, of course, because loading a binary module is much faster that compiling a source module.

[5] Just in case you were wondering, access to modules is synchronized on suite lock by the caller, so the code presented here is free of race conditions and potential data loss.