package reed-solomon-erasure

  1. Overview
  2. Docs

Bookkeeper for shard by shard encoding.

This is useful for avoiding incorrect use of encode_single_typ, and encode_single_sep_typ functions.

Naming conventions

typ represents bytes, str, and bigstr, e.g. encode_typ represents encode_bytes, encode_str, and encode_bigstr.

Use cases

Shard by shard encoding is useful for streamed data encoding where you do not have all the needed data shards immediately, but you want to spread out the encoding workload rather than doing the encoding after everything is ready.

A concrete example would be network packets encoding, where encoding packet by packet as you receive them may be more efficient than waiting for N packets then encode them all at once.

Example

open Reed_solomon_erasure

let () =
  let r = ReedSolomon.make 3 2 in

  let sbs = ShardByShard.make r in

  let shards = [|Bytes.of_string "\000\001\002\003\004";
                 Bytes.of_string "\005\006\007\008\009";
                 (* say we don't have the 3rd data shard yet
                    and we want to fill it in later *)
                 Bytes.of_string "\000\000\000\000\000";
                 Bytes.of_string "\000\000\000\000\000";
                 Bytes.of_string "\000\000\000\000\000"|] in

  (* encode 1st and 2nd data shard *)
  ShardByShard.encode_bytes sbs shards;
  ShardByShard.encode_bytes sbs shards;

  (* fill in 3rd data shard *)
  Bytes.set shards.(2) 0 '\010';
  Bytes.set shards.(2) 1 '\011';
  Bytes.set shards.(2) 2 '\012';
  Bytes.set shards.(2) 3 '\013';
  Bytes.set shards.(2) 4 '\014';

  (* now do the encoding*)
  ShardByShard.encode_bytes sbs shards;

  (* above is equivalent to doing ReedSolomon.encode_bytes shards *)

  assert (ReedSolomon.verify_bytes r shards)

Ownership of shards

The shard by shard codec is just a wrapper around the core codec type reed_solomon, so all potential shard ownership issues in functions provided by ReedSolomon are carried over.

See ReedSolomon, Ownership of shards section for details.

make r

Makes a new instance of the bookkeeping type. Note that the codec is NOT thread-safe.

The original Reed-Solomon codec is still usable - shard_by_shard codec does not require exclusive ownership of the Reed-Solomon codec.

val parity_ready : shard_by_shard -> bool

parity_ready sbs

val reset : shard_by_shard -> unit

reset sbs

Resets the bookkeeping data.

You should call this when you have added and encoded all data shards, and have finished using the parity shards.

Raises RS_SBS_Error.Exn.LeftoverShards

val reset_no_exn : shard_by_shard -> (unit, RS_SBS_Error.t) result

reset sbs

Error returning variant of reset.

val reset_force : shard_by_shard -> unit

reset_force sbs

Resets the bookkeeping data without checking.

val cur_input_index : shard_by_shard -> int

cur_input_index sbs

Returns the current input shard index.

val encode_bytes : shard_by_shard -> bytes array -> unit

encode_bytes sbs shards

See encode_bigstr.

val encode_sep_bytes : shard_by_shard -> bytes array -> bytes array -> unit

encode_sep_bytes sbs data parity

See encode_sep_bigstr.

val encode_str : shard_by_shard -> string array -> unit

encode_str sbs shards

See encode_bigstr.

val encode_sep_str : shard_by_shard -> string array -> string array -> unit

encode_sep_str sbs data parity

See encode_sep_bigstr.

val encode_bigstr : shard_by_shard -> bigstring array -> unit

encode_bigstr sbs shards

Constructs the parity shards partially using the current input data shard.

Raises RS_SBS_Error.Exn.TooManyCalls when all input shards have already been filled in via any of the encode functions.

val encode_sep_bigstr : shard_by_shard -> bigstring array -> bigstring array -> unit

encode_sep_bigstr sbs data parity

Constructs the parity shards partially using the current input data shard.

Raises RS_SBS_Error.Exn.TooManyCalls when all input shards have already been filled in via any of the encode functions.

val encode_bytes_no_exn : shard_by_shard -> bytes array -> (unit, RS_SBS_Error.t) result

encode_bytes_no_exn sbs shards

Error returning variant of encode_bytes.

val encode_sep_bytes_no_exn : shard_by_shard -> bytes array -> bytes array -> (unit, RS_SBS_Error.t) result

encode_sep_bytes_no_exn sbs data parity

Error returning variant of encode_sep_bytes.

val encode_str_no_exn : shard_by_shard -> string array -> (unit, RS_SBS_Error.t) result

encode_str_no_exn sbs shards

Error returning variant of encode_str.

val encode_sep_str_no_exn : shard_by_shard -> string array -> string array -> (unit, RS_SBS_Error.t) result

encode_sep_str_no_exn sbs data parity

Error returning variant of encode_sep_str.

val encode_bigstr_no_exn : shard_by_shard -> bigstring array -> (unit, RS_SBS_Error.t) result

encode_bigstr_no_exn sbs shards

Error returning variant of encode_bigstr.

val encode_sep_bigstr_no_exn : shard_by_shard -> bigstring array -> bigstring array -> (unit, RS_SBS_Error.t) result

encode_sep_bigstr_no_exn sbs data parity

Error returning variant of encode_sep_bigstr.