package faraday

  1. Overview
  2. Docs
package faraday
Legend:
Page
Library
Module
Module type
Parameter
Class
Class type
Source

Module FaradaySource

Serialization primitives built for speed an memory-efficiency.

Faraday is a library for writing fast and memory-efficient serializers. Its core type and related operation gives the user fine-grained control over copying and allocation behavior while serializing user-defined types, and presents the output in a form that makes it possible to use vectorized write operations, such as the writev system call, or any other platform or application-specific output APIs.

A Faraday serializer manages an internal buffer and a queue of output buffers. The output bufferes may be a sub range of the serializer's internal buffer or one that is user-provided. Buffered writes such as write_string, write_char, write_bigstring, etc., copy the source bytes into the serializer's internal buffer. Unbuffered writes such as schedule_string, schedule_bigstring, etc., on the other hand perform no copying. Instead, they enqueue the source bytes into the serializer's write queue directly.

Sourcetype t

The type of a serializer.

Constructors

Sourceval create : int -> t

create len creates a serializer with a fixed-length internal buffer of length len.

Sourceval of_bigstring : bigstring -> t

of_bigstring buf creates a serializer, using buf as its internal buffer. The serializer takes ownership of buf until the serializer has been closed and flushed of all output.

Buffered Writes

Serializers manage an internal buffer for coalescing small writes. The size of this buffer is determined when the serializer is created and does not change throughout the lifetime of that serializer. If the buffer does not contain sufficient space to service the buffered writes of the caller, a new buffer of the same size will be allocated.

Sourceval write_string : t -> ?off:int -> ?len:int -> string -> unit

write_string t ?off ?len str copies str into the serializer's internal buffer.

Sourceval write_bytes : t -> ?off:int -> ?len:int -> Bytes.t -> unit

write_bytes t ?off ?len bytes copies bytes into the serializer's internal buffer. It is safe to modify bytes after this call returns.

Sourceval write_bigstring : t -> ?off:int -> ?len:int -> bigstring -> unit

write_bigstring t ?off ?len bigstring copies bigstring into the serializer's internal buffer. It is safe to modify bytes after this call returns.

Sourceval write_gen : t -> length:('a -> int) -> blit:('a -> int -> bigstring -> int -> int -> unit) -> ?off:int -> ?len:int -> 'a -> unit

write_gen t ~length ~blit ?off ?len x copies x into the serializer's internal buffer using the provided length and blit operations.

Sourceval write_char : t -> char -> unit

write_char t char copies char into the serializer's internal buffer.

Sourceval write_uint8 : t -> int -> unit

write_uint8 t n copies the lower 8 bits of n into the serializer's internal buffer.

Sourcemodule BE : sig ... end

Big endian serializers

Sourcemodule LE : sig ... end

Little endian serializers

Unbuffered Writes

Sourceval schedule_string : t -> ?off:int -> ?len:int -> string -> unit

schedule_string t ?off ?len str schedules str to be written the next time the serializer surfaces writes to the user. str is not copied in this process.

Sourceval schedule_bytes : t -> ?off:int -> ?len:int -> Bytes.t -> unit

schedule_bytes t ?off ?len bytes schedules bytes to be written the next time the serializer surfaces writes to the user. bytes is not copied in this process, so bytes should only be modified after t has been flushed.

Sourceval schedule_bigstring : t -> ?off:int -> ?len:int -> bigstring -> unit

schedule_bigstring t ?free ?off ?len bigstring schedules bigstring to be written the next time the serializer surfaces writes to the user. bigstring is not copied in this process, so bigstring should only be modified after t has been flushed.

Control Operations

Sourceval yield : t -> unit

yield t causes t to delay surfacing writes to the user, instead returning a `Yield operation with an associated continuation k. This gives the serializer an opportunity to collect additional writes before sending them to the underlying device, which will increase the write batch size. Barring any intervening calls to yield t, calling the continuation k will surface writes to the user.

Sourceval flush : t -> (unit -> unit) -> unit

flush t f registers f to be called when all prior writes have been successfully completed. If t has no pending writes, then f will be called immediately.

Sourceval close : t -> unit

close t closes t. All subsequent write calls will raise, and any pending or subsequent yield calls will be ignored. If the serializer has any pending writes, user code will have an opportunity to service them before it receives the Close operation.

Sourceval is_closed : t -> bool

is_closed t is true if close has been called on t and false otherwise. A closed t may still have pending output.

Sourceval shift : t -> int -> unit

shift t n removes the first n bytes in t's write queue. Any scheduled buffers that are contained in this span of bytes are free()'d, if necessary.

Sourceval drain : t -> int

drain t removes all pending writes from t, returning the number of bytes that were enqueued to be written and freeing any scheduled buffers in the process.

Sourceval free_bytes_in_buffer : t -> int

free_bytes_in_buffer t returns the free space, in bytes, of the serializer's write buffer. If a write_* call has a length that exceeds this value, the serializer will allocate a new buffer that will replace the serializer's internal buffer for that and subsequent calls.

Sourceval has_pending_output : t -> bool

has_pending_output t is true if t's output queue is non-empty. It may be the case that t's queued output is being serviced by some other thread of control, but has not yet completed.

Sourceval pending_bytes : t -> int

pending_bytes t is the size of the next write, in bytes, that t will surface to the caller.

Running

Sourcetype buffer = [
  1. | `String of string
  2. | `Bytes of Bytes.t
  3. | `Bigstring of bigstring
]
Sourcetype 'a iovec = {
  1. buffer : 'a;
  2. off : int;
  3. len : int;
}

A view into iovec.buffer starting at iovec.off and with length iovec.len.

Sourcetype operation = [
  1. | `Writev of buffer iovec list
    (*

    Write the ovecs, reporting the actual number of bytes written by calling shift. Failure to do so will result in the same bytes being surfaced in a `Writev operation multiple times.

    *)
  2. | `Yield
    (*

    Yield to other threads of control, waiting for additional output before procedding. The method for achieving this is application-specific, but once complete, the caller can proceed with serialization by simply making another call to operation or serialize.

    *)
  3. | `Close
    (*

    Serialization is complete. No further output will be received.

    *)
]
Sourceval operation : t -> operation

operation t is the next operation that the caller must perform on behalf of the serializer t. Users should consider using serialize before this function. See the documentation for the operation type for details on how callers should handle these operations.

Sourceval serialize : t -> (buffer iovec list -> [ `Ok of int | `Closed ]) -> [ `Yield | `Close ]

serialize t writev sufaces the next operation of t to the caller, handling a `Writev operation with writev function and performing an additional bookkeeping on the caller's behalf. In the event that writev indicates a partial write, serialize will call yield on the serializer rather than attempting successive writev calls.

Sourceval serialize_to_string : t -> string

serialize_to_string t runs t, collecting the output into a string and returning it. serialzie_to_string t immediately closes t and ignores any calls to yield on t.