package MlFront_Exec

  1. Overview
  2. Docs
package MlFront_Exec
Legend:
Page
Library
Module
Module type
Parameter
Class
Class type
Source

Module MlFront_Exec.BuildTraceStoreSource

Constructive trace loading and saving.

Binary Format

  1. for all segments: 4 byte BE length(segment) + 32 byte SHA256(segment) + N byte segment.

Segment

A segment is zero or more traces whose total binary encoding is 16MB of RAM or less. The 16MB is the limit of 32-bit OCaml strings, but it is also a practical limit for other platforms. You can have multiple segments in a single trace file, so large files are still possible.

When writing a segment, keep this as close to 16MB as possible without exceeding it. Perhaps maintain a byte counter and when bytes exceed 12MB flush the current segment and start a new one.

The more traces in a segment, the more efficient it is to parallelize.

  1. for all traces: 4 byte BE length(trace) + N byte trace.
Sourceval get_apply_aliases : config:BuildConfig.t -> (MlFront_Core.StandardModuleId.t -> MlFront_Thunk.ThunkSemver64Rep.t -> MlFront_Core.StandardModuleId.t * MlFront_Thunk.ThunkSemver64Rep.t) BuildCore.Alacarte_6_4_test.CSuspending.t
Sourceval read_traces_gracefully : preconfig:BuildConfig.preconfig -> transform_values: (MlFront_Thunk.ThunkAst.t -> [ `Modified of MlFront_Thunk.ThunkAst.t | `Unmodified ]) option -> Unix.file_descr -> BuildInstance.ValueStore.trace list

read_traces_gracefully ~preconfig ~transform_values file reads the traces in file. Each trace's values are transformed using transform_values if transform_values is Some f.

If the trace file is incomplete and/or corrupted midway (ex. the build system was killed while writing it), all valid traces up to the incomplete/corrupt trace are returned.

The return value is the list of traces read.

Sourceval visit_traces_in_file : build_pubkey:[ `PublicKey of string ] -> build_seckey:[ `SecretKey of string ] -> valuestore_get: (BuildCore.Alacarte_3_2_apparatus.K.t -> string -> MlFront_Core.FilePath.t option MlFront_Exec.BuildCore.Alacarte_xpromise_apparatus.Promise.t) -> valuestore_put: (BuildCore.Alacarte_3_2_apparatus.K.t -> string -> string -> (bool * (string * int64)) BuildCore.Alacarte_xtraces_test.EngineBackend.promise) -> Unix.file_descr -> (int -> BuildCore.Alacarte_3_2_apparatus.K.t -> (BuildCore.Alacarte_3_2_apparatus.K.t * (BuildCore.Alacarte_3_2_apparatus.K.t, BuildCore.Alacarte_3_2_apparatus.V.t) BuildCore.Alacarte_3_2_apparatus.V.key_dependent_hash) list -> BuildCore.Alacarte_3_2_apparatus.V.t -> unit) -> unit

visit_traces_in_file ~build_pubkey ~build_seckey ~valuestore_get ~valuestore_put file visit_trace visits all the traces in file and visits the values referenced in those traces.

Each trace (key, dependency_hashes, value) in file will be given to visit_trace trace_size key dependency_hashes value. Any of the following values referenced in the trace will be visited with valuestore_get key value_id:

Values ... the parsed values AST ... are not visited if the parsed AST is not compatible with the OCaml runtime running this code; see BuildCore.compatibility_tag. If the parsed AST is not compatible, the corresponding valuesfile is parsed into a new AST, and the new AST is written to the value store using build_seckey and valuestore_put key value_id contents. This is the only use for build_seckey and valuestore_put in this function.

The same key in valuestore_get key value_id and valuestore_put key value_id content may be used with different value_id. For example, if Values is incompatible with the runtime you will see a valuestore_get key value_id for a Values value_id, and then a valuestore_get key value_id for a ValuesFile value_id to do the reparsing.

Unlike read_traces_gracefully, this function does not load all traces into memory at once, and it is an error for the traces to have any corruption.

Sourceval save : config:BuildConfig.t -> BuildInstance.ValueStore.trace list -> Unix.file_descr -> unit

save ~config traces file saves the given traces to file.