Library
Module
Module type
Parameter
Class
Class type
Serialization primitives built for speed an memory-efficiency.
Faraday is a library for writing fast and memory-efficient serializers. Its core type and related operation gives the user fine-grained control over copying and allocation behavior while serializing user-defined types, and presents the output in a form that makes it possible to use vectorized write operations, such as the writev
system call, or any other platform or application-specific output APIs.
A Faraday serializer manages an internal buffer and a queue of output buffers. The output bufferes may be a sub range of the serializer's internal buffer or one that is user-provided. Buffered writes such as write_string
, write_char
, write_bigstring
, etc., copy the source bytes into the serializer's internal buffer. Unbuffered writes such as schedule_string
, schedule_bigstring
, etc., on the other hand perform no copying. Instead, they enqueue the source bytes into the serializer's write queue directly.
type bigstring =
(char, Bigarray.int8_unsigned_elt, Bigarray.c_layout) Bigarray.Array1.t
val create : int -> t
create len
creates a serializer with a fixed-length internal buffer of length len
.
of_bigstring buf
creates a serializer, using buf
as its internal buffer. The serializer takes ownership of buf
until the serializer has been closed and flushed of all output.
Serializers manage an internal buffer for coalescing small writes. The size of this buffer is determined when the serializer is created and does not change throughout the lifetime of that serializer. If the buffer does not contain sufficient space to service the buffered writes of the caller, a new buffer of the same size will be allocated.
val write_string : t -> ?off:int -> ?len:int -> string -> unit
write_string t ?off ?len str
copies str
into the serializer's internal buffer.
write_bytes t ?off ?len bytes
copies bytes
into the serializer's internal buffer. It is safe to modify bytes
after this call returns.
write_bigstring t ?off ?len bigstring
copies bigstring
into the serializer's internal buffer. It is safe to modify bytes
after this call returns.
val write_gen :
t ->
length:('a -> int) ->
blit:('a -> int -> bigstring -> int -> int -> unit) ->
?off:int ->
?len:int ->
'a ->
unit
write_gen t ~length ~blit ?off ?len x
copies x
into the serializer's internal buffer using the provided length
and blit
operations.
val write_char : t -> char -> unit
write_char t char
copies char
into the serializer's internal buffer.
val write_uint8 : t -> int -> unit
write_uint8 t n
copies the lower 8 bits of n
into the serializer's internal buffer.
module BE : sig ... end
Big endian serializers
module LE : sig ... end
Little endian serializers
val schedule_string : t -> ?off:int -> ?len:int -> string -> unit
schedule_string t ?off ?len str
schedules str
to be written the next time the serializer surfaces writes to the user. str
is not copied in this process.
schedule_bytes t ?off ?len bytes
schedules bytes
to be written the next time the serializer surfaces writes to the user. bytes
is not copied in this process, so bytes
should only be modified after t
has been flush
ed.
schedule_bigstring t ?free ?off ?len bigstring
schedules bigstring
to be written the next time the serializer surfaces writes to the user. bigstring
is not copied in this process, so bigstring
should only be modified after t
has been flush
ed.
val yield : t -> unit
yield t
causes t
to delay surfacing writes to the user, instead returning a `Yield
operation with an associated continuation k
. This gives the serializer an opportunity to collect additional writes before sending them to the underlying device, which will increase the write batch size. Barring any intervening calls to yield t
, calling the continuation k
will surface writes to the user.
val flush : t -> (unit -> unit) -> unit
flush t f
registers f
to be called when all prior writes have been successfully completed. If t
has no pending writes, then f
will be called immediately.
val close : t -> unit
close t
closes t
. All subsequent write calls will raise, and any pending or subsequent yield
calls will be ignored. If the serializer has any pending writes, user code will have an opportunity to service them before it receives the Close
operation.
val is_closed : t -> bool
is_closed t
is true
if close
has been called on t
and false
otherwise. A closed t
may still have pending output.
val shift : t -> int -> unit
shift t n
removes the first n
bytes in t
's write queue. Any scheduled buffers that are contained in this span of bytes are free()
'd, if necessary.
val drain : t -> int
drain t
removes all pending writes from t
, returning the number of bytes that were enqueued to be written and freeing any scheduled buffers in the process.
val free_bytes_in_buffer : t -> int
free_bytes_in_buffer t
returns the free space, in bytes, of the serializer's write buffer. If a write_*
call has a length that exceeds this value, the serializer will allocate a new buffer that will replace the serializer's internal buffer for that and subsequent calls.
val has_pending_output : t -> bool
has_pending_output t
is true
if t
's output queue is non-empty. It may be the case that t
's queued output is being serviced by some other thread of control, but has not yet completed.
val pending_bytes : t -> int
pending_bytes t
is the size of the next write, in bytes, that t
will surface to the caller.
A view into iovec.buffer
starting at iovec.off
and with length iovec.len
.
type operation = [
| `Writev of buffer iovec list
Write the ovecs, reporting the actual number of bytes written by calling shift
. Failure to do so will result in the same bytes being surfaced in a `Writev
operation multiple times.
| `Yield
| `Close
Serialization is complete. No further output will be received.
*) ]
serialize t writev
sufaces the next operation of t
to the caller, handling a `Writev
operation with writev
function and performing an additional bookkeeping on the caller's behalf. In the event that writev
indicates a partial write, serialize
will call yield
on the serializer rather than attempting successive writev
calls.