package jenga
Install
Dune Dependency
Authors
Maintainers
Sources
sha256=3ffa14642a0cfe53256560cf03eee22f3828a7b5189688c3dad3639cb8a0fa2b
md5=3941eaa77e876630bdae8969dc89db1f
CHANGES.md.html
113.33.00
Jenga's
Api
is extended withprintf
andprintf_verbose
, andverbose
is removed. These changes are in preparation for jenga/server.In jenga/server, builds will run on a daemonized server; the build trace is displayed on any clients which are
watch
ing.Api.printf
andApi.printf_verbose
allow messages from thejenga/root.ml
.The command line flag
-verbose
will be a client flag, not a server flag, allowing the choice of verbosity to be selected per-client. It therefore won't make sense to allowjenga/root.ml
to ask ifverbose
is true, so we removeverbose
from the jengaApi
.Instead
jenga/root/ml
can useprintf_verbose
for messages which are to be displayed only by a client that is watching with-verbose
.Make benchmark analysis script compute heap stats in addition to time.
Give more information about why we build
Dep.action
andDep.action_stdout
, because right now, the output of jenga sometimes doesn't contain the information (the summary line contains it, but it's not always printed, at least when using-stop-on-error
). For instance:Old jenga:
*** jenga: ERROR: (summary) a/.DEFAULT: External command failed - build a stdout + bash -e -u -o pipefail -c false - exit a stdout, 3ms, exited with code 1 *** jenga: ERROR: (summary) a/foo.ml.d: External command failed - build a stdout + .../ocamldep.opt -modules -impl foo.ml foo.ml: File "foo.ml", line 2, characters 0-0: Error: Syntax error - exit a stdout, 4ms, exited with code 2
New jenga:
*** jenga: ERROR: (summary) a/.DEFAULT: External command failed - build a .DEFAULT + bash -e -u -o pipefail -c false - exit a .DEFAULT, 5ms, exited with code 1 *** jenga: ERROR: (summary) a/foo.ml.d: External command failed - build a rule of foo.ml.d + .../ocamldep.opt -modules -impl foo.ml foo.ml: File "foo.ml", line 2, characters 0-0: Error: Syntax error - exit a rule of foo.ml.d, 6ms, exited with code 2
When
a command run by jenga forks
and that fork inherits the stdout/stderr pipes to jenga
and the fork doesn't terminate
but the initial process does terminate
we are in a situation where, jenga before this feature would consider the command is still running. This makes for some annoying debug, because which command started the stray process is hard to tell, the stdout and stderr of the initial process are not visible anywhere (they are in jenga's memory), etc. Also jenga appears stuck.
So instead, we expect the stdout/stderr to close shortly after the initial process dies, and if not, we close stdout/stderr ourselves and consider that the command failed.
To test, I put this in a jbuild: (alias ((name DEFAULT) (deps ()) (action "(echo false >&2; sleep 4) & ")))
and
JENGA_OPTIONS='((fd_close_timeout 2s))' jenga -P
complained:*** jenga: ERROR: (summary) lib/sexplib/src/.DEFAULT: External command failed - build lib/sexplib/src stdout + bash -e -u -o pipefail -c '(echo false >&2; sleep 4) & ' false - exit lib/sexplib/src stdout, 2.004s, stdout or stderr wasn't closed 2s after process exited (due to a stray process perhaps?)
Also I had hydra do a full tree build (both in jane-continuous-release and jane-release) and they both succeeded.
Remove jenga command line flag which was
-delay N
This controlled
Config.delay_for_dev
which inserted a delay of N seconds before running any job inJob.run
. I have heard this described as the most pointless command line flag of all time!Reason to fix now: Another feature
jane/jenga/api.printf
removesApi.verbose
which exposesConfig.verbose
to jenga/root.ml. Following both features, we can remove theRun.For_user
hackery.Avoid
nan
from divide-by-0 in jenga monitor'sfinish_time_estimator
.Without this fix, we may get
nan
as the value of aTime.t
which breaks an invariant of theTime
module. This may result in a crash inTime.to_ofday
.("unhandled exception" ((monitor.ml.Error_ ((exn "Option.value_exn None") (backtrace ("Raised by primitive operation at file \"option.ml\", line 59, characters 4-21" "Called from file \"zone.ml\", line 751, characters 8-104" "Called from file \"zone.ml\", line 761, characters 10-48" "Called from file \"time0.ml\", line 275, characters 31-54" "Called from file \"finish_time_estimator.ml\", line 38, characters 14-54" "Called from file \"message.ml\", line 384, characters 30-66" "Called from file \"list.ml\", line 73, characters 12-15" "Called from file \"list.ml\", line 73, characters 12-15" "Called from file \"message.ml\", line 158, characters 2-42" "Called from file \"deferred0.ml\", line 65, characters 64-69" "Called from file \"job_queue.ml\", line 160, characters 6-47" "")) (monitor (((name try_with) (here ()) (id 4) (has_seen_error true) (is_detached true)))))) ((pid 17688) (thread_id 4
Code refactoring to avoid circular module dependencies in
jenga/server
feature.There changes have no semantic effects; it's just code movement.
Extract
Message.Job_summary
as a top level module.Extract
Build.Run_reason
as a top level module.Create all
Effort.Counter
instances inProgress
.
The current cycle detection in build.ml is very hard to use correctly. Implement a robust cycle detection in Tenacious.
Additionally, this feature lets the user see what jenga is currently working on by doing
jenga diagnostics dump-tenacious-graph
.Along the way, fixed
jenga monitor
to exit non-zero when it can't discover jengaroot.Change
jenga -time
to prefix lines with absolute times. More standard & helpful.Extend jenga Api with support for registering environment variables. These can be accessed and manipulated via RPC to the running jenga.
This feature is necessary to support env-var discovery in build-manager via RPC, rather than the current hack of parsing the output from jenga.
Environment variable manipulation is exposed on the command line with:
jenga env get NAME jenga env set NAME VALUE jenga env unset NAME jenga env print
The Api supports access to either the current value of a variable, or to a dependency which responds dynamically to changes made by
setenv
.Var.peek : 'a Var.t -> 'a Dep.getenv : 'a Var.t -> 'a Dep.t
Removed
update-stream
RPC. Was never used by anything. It wouldn't have worked anyway because it contained interned paths.Removed
app/jenga/lib/rpc_intf.mli
- just unhelpful code duplication of the descriptions inrpc_intf.ml
.Error message when running on nfs was super-confusing.
Avoid double-reporting build errors originating from the action of a multi-target rule. (And in future avoid double-reporting the error to the `error-pipe' of jenga/server.)
Attribute an error to the head target ONLY.
Additional targets are treated as dependants of the
head_target
, and so (if demanded) are regarded as havingfailure' not
error' status.Always use
head-target
in the "-build" line message, instead of whatever target happens (non deterministically) to getdemanded
first.Allow user control over the which target is regarded as the
head_target
, by taking the first target listed as thehead_target
.
Couple of extra messages bracketing when GC is performed & finished.
Stop taking nfs locks, just rely on local locks, so we're finally free from the bug of Nfs_lock where it can't remove a half created lock. Also clean things up while I'm here, no change intended in special_paths.ml.
Refactor code to remove distinction between types
Progess.Need.t
andGoal.t
; removing theProgess.Need.t
type by supporting the one additional case ofNeed.jengaroot
as aGoal.t
.Allow the user to demand a `build' of (just) the jengaroot.
Add support for unsetting environment variables prior to running actions
113.24.02
Fix the build of jenga and updated the examples
113.24.00
Restructure the code in a way that allows to build binaries that statically link jenga with the rules. This is useful because some debugging/profiling tools don't work in the presence of dynamically loaded code very well.
Switch to PPX.
Change the gc info output by jenga so it shows heap size and top heap size, instead of live and heap size. The live part is not super useful given how random it is. I have seen cases where jenga was using 20GB during building and jenga reported a heap size of 13GB at the end so the top heap size avoids being tricked.
First half of the fixes no packing: sharing the structures of dependencies, so they take less space on disk (and in memory as well, when they are loaded from disk, but not really when building from scratch given the way we will use them).
The sexp format also has sharing, because it would also blow up in size otherwise (this is different from the interning of paths, where the interning saves a constant factor). And of course, it makes it possible to see the actual on-disk representation which is nice.
Also fix unhelpful error (contains no information) when the db can't be loaded.
Break the thing that avoids rerunning rules when the set of dependencies decreases. I think it was never useful anyway.
Better error on duplicate targets in the same rule.
To prevent running more than one jenga in a repository, use a local lock rather than an nfs one. We need a transition period though, so for now we use both kinds of locks. Building on nfs is slow, so I don't think there's any downside is not supporting nfs this way. And maybe inotify doesn't work. The upside is that we don't step into Lock.Nfs bugs where if a process is interrupted at the wrong time (when the two lock files are empty) the locks won't be cleaned up automatically, forcing someone to get rid of the lock files manually.
Also rename
.jenga/.jenga.*
to.jenga/*
, because all these prefixes are annoying.Added a couple of options to turn off some part of jenga, which I used to check how they impacted performance, and could still be handy later.
Optionally display additional information about much allocation was done, at the end of builds. Used it to try to improve memory usage of full tree builds without actually doing full tree builds.
Make stat'ing faster. Hash cons some tenacious that build mtimes map to avoid a huge increase of memory use.
Got rid of noise when stopping jenga.
Some changes to the implementation of Tenacious to improve memory efficiency.
Includes the following changes:
Remove
strong_refs
field ofHeart.fragile
type and instead insert links from theclients
Ring.t
to its parent usingRing.keep_alive
.Replace the
Ring.t
in thetriggers
field ofHearts.fragile
with anIvar.t
since all uses oftriggers
were producing their own equivalentIVar.t
s. Remove the functions broken by this because they were unused.Make the
Tenacious.t
type a concrete datatype, and optimize pure computations by partially evaluating this datatype directly in the pure case.Rather than building separate
Heart.t
s for cancellation and the result, split the cancellation heart into two heartscancel
anddep
and then usedep
as the result heart. This means that a tenacious is cancelled if eithercancel
ordep
is broken, and it must return a heart representing the validity of its result combined withdep
.
Cursory benchmarking indicates a 23% improvement in maximum resident set size and a 10% improvment in (user) execution time when building the lib directory from scratch.
Adding some
sexp_of
functions, since they're always missing and it's a pain when debugging.Adding direct support for
Dep.map
. Even now thatTenacious
is smarter, this still creates lessbind
s. Doesn't seem to make much of a difference (perhaps 3-5% less allocation, on a null build of lib), but if nothing else, it's much less surprising to think thatDep.Map
becomesTenacious.Map
.Added Dep.List.concat.
Make it possible to turn off the behavior where jenga rejects commands that output on stderr.
It increases slightly the footprint of the in memory db, but the difference is tiny compared to the rest of the memory usage.
Added a few tests about
Jenga_lib.Api.Reflect
113.00.00
Treat output to stderr by an action as a failure.
112.35.00
Make
.jenga.db
be format version aware, and store.jenga
files in a subdirectory.Switch md5 computation to use a C binding, fixes #10.
Move
jem.exe
tojenga.exe monitor
,jenga_offline.exe
tojenga.exe offline
,jenga.exe -cat-api
tojenga.exe cat-api
.Remove
Path.dotdot
.Adjust behaviour of
Path.Repo.is_descendant
to be more consistent.Repo-root invariance: remove
Path.Abs.the_root
,Path.Rel.to_absolute_string
,Path.Rel.create_from_absolute
, and makePath.relative Path.the_root ".."
fail.Add a flag
-sandbox-action
tojenga build
to run the action in an environment that (attempts to) detect missing dependencies or overly large sets of targets.Don't interpret paths of shape
./foo.ext
as aliases.Always interpret
Dep.path path
as the file atpath
rather than the default alias for the directory atpath
Make jenga look for the variable JENGA_OPTIONS, and use it to add a debug setting.
Fix handling of directories created during the build.
Basic support for symlink resolution.
Add an option to the monitor command to only display a single snapshot of the progress, rather than a continually updated pipe.
Fix a deadlock where File_access throttle and directory lock are obtained in the wrong order.
112.24.00
Interns strings, significantly reducing memory use.
112.17.00
Fixed byte-compile targets to avoid stale artifact deletion of
.for-byte-compile.cmt
file when compilation fails.This avoids polling jenga trigger loop.
Show
Removed stale build artifact
messages only when-act
flag is given.Extended Jenga API with
val file_existence : Path.t -> unit t
.file-existence
has same relationship tofile-exists
asglob_change
has toglob_listing
.Fixed memory leak in tenacious hearts by using weak references and finalizers.
Strip code for obsolete versions of hearts.
Strip code for
OLD_TENACIOUS=true
.Ensure
Ring
support preemptive calls todetach
. Add ring tests.Keep message showing
Live(Kb-delta)
.
Fixed curly braces in globs.
Throttled calls to
Writer.save
, to fix thetoo many open files
bug.Throttled calls to
Writer.save
fromsave_description.run
sharing the same throttle used for all FD access, with~max_concurrent_jobs:500
Added to
jenga -progress
~save=XXX
, the number of calls toWriter.save
inProgress.saves_run
.
112.06.00
Support for user control of stale-artifact deletion, by allowing specification of an artifact-determination policy.
Expose jenga's internal (and better - only quotes when necessary) definition of
Shell.escape
inApi
Removed
Action.shell
from the API, superseded byAction.process
.Changed RPC interface as needed for build manager to switch from scraping error messages to RPCs.
Fixed jenga's per-rule memo table, which mistakenly kept stale values.
Show what target is being demanded, useful for debugging rules.
Run user action when persistent format changes.
When filtering buildable targets by globs, pay attention to the kinds allowed by the glob.
Specifically, if the kinds don't include
`File
(i.e. only include`Directory
) then we should not see anybuildable_targets
in the filtered list.
112.01.00
Don't show noisy
glob..changed
messages except with-show-glob-changed
flag.Support shared build rules via
${jenga}/share
.Detect cycle in dep scheme instead of hanging.
Made standalone actions atomic, just like actions associated with target files.
Running actions and recording the result in the persistent
.jenga.db
should be performed atomically for standalone actions, as it is for actions which are associated with target files
111.31.00
Switched API to composable generator schemes.
Support
-api
flag to show the embedded API.New examples.
111.28.00
Fixed problem that caused
rule failed to generate targets
.
111.25.00
Switched to un-version-numbered API.
Renamed
Tenacious_sample_lib.Tenacious
toTenacious_sample_lib.Tenacious_sample
to avoid conflicts in the public release.Write
buildable_targets.list
(on alias.info
).
111.21.00
Introduced jenga API v3, a small cleanup of v2 which has been planned for a while.
111.08.00
Fix a hang.
Jenga could reach a state with a non-zero todo-count, but have no jobs actually running, and then hang in this state forever. The hang would be evident from a progress line with not all targets built and with
j=0+0
such as:todo: 17 (100406 / 100423) j=0+0 con=149956 act=3303, finish at: 16:20
111.06.00
Improved the error message when the same library is defined multiple times.
Fixed an issue where jenga sometimes would sometimes complain about a self cycle when
foo.ml
uses a moduleFoo
.With
-no-notifiers
, jenga doesn't useinotify
to watch for file changes. This is useful for lintingjengaroot.ml
.Allowed writing jenga rules which restrict dependencies from an initial conservative approximation to a more accurate set discovered after an action is run