Specifications
LispTick Streaming Protocol
TODO
LispTick File Format
LispTick can use lots of different format as source (.csv, dataview export, InfluxDB…). But we have created our own archive file format which is both compact and very fast as it uses direct access.
All int values, int32, int64, int with variable number of bytes, are stored using Little Endian order.
File
+----------------+------------+-------------+------------------------------------------------+
| Offset | bytes Size | Value | Description |
+----------------+------------+-------------+------------------------------------------------+
| 0 | 8 | 4c 69 73 70 | "LispTick" header |
| | | 54 69 63 6b | |
+----------------+------------+-------------+------------------------------------------------+
| 8 | S1 | ... | Data Block 1 |
+----------------+------------+-------------+------------------------------------------------+
| S1+8 | S2 | ... | Data Block 2 |
+----------------+------------+-------------+------------------------------------------------+
| ... | ... | ... | Data Blocks |
+----------------+------------+-------------+------------------------------------------------+
| S1+...+Sn-1 +8 | Sn | ... | Data Block n |
+----------------+------------+-------------+------------------------------------------------+
| Coff | Joff-Coff | ... | Intrument(s) List Data Block |
+----------------+------------+-------------+------------------------------------------------+
| Joff | FS-Joff-32 | ... | Json Scheme Data Block |
+----------------+------------+-------------+------------------------------------------------+
| -32 | 8 | Coff | Offset of Intrument(s) List Data Block (int64) |
+----------------+------------+-------------+------------------------------------------------+
| -24 | 4 | Cs | Number of Instrument(s) (int32) |
+----------------+------------+-------------+------------------------------------------------+
| -20 | 8 | Joff | Offest of Json Scheme Data Block (int64) |
+----------------+------------+-------------+------------------------------------------------+
| -12 | 4 | Js | Json Scheme size encoded (int32) |
+----------------+------------+-------------+------------------------------------------------+
| -8 | 8 | 6b 63 69 54 | "kciTpsiL" footer |
| | | 70 73 69 4c | |
+----------------+------------+-------------+------------------------------------------------+
FS is file size in bytes.
Data Block
+--------+------------+--------------------------------------+
| Offset | bytes Size | Description |
+--------+------------+--------------------------------------+
| 0 | 1-4 | Data Block Header or Zstd |
+--------+------------+--------------------------------------+
| 0 or 1 | n bytes | data encoded following Header format |
+--------+------------+--------------------------------------+
If Data Block Header is 0x28 you must have those 4 bytes at start:
- 28 b5 2f fd
Which is Zstd Magic Number, this block must be directly read by a Ztd decompressor.
Data Block Header
+------------+------------------+---------------------------------------------------------+
| Bit Number | Field Name | Description |
+------------+------------------+---------------------------------------------------------+
| 0-2 | Record Size | number of bytes per record, 0 for 8 bytes, default |
+------------+------------------+---------------------------------------------------------+
| 3-5 | Compression Mode | 0 for no compression, 1 for single value, 5 for Zstd |
+------------+------------------+---------------------------------------------------------+
| 6 | Factor Flag | if set, each record must be multiplied by a factor |
+------------+------------------+---------------------------------------------------------+
| 7 | Delta Flag | if set, each record must be computed using previous one |
+------------+------------------+---------------------------------------------------------+
Examples:
- 0x00 Uncompressed data
- 0x08 Single value, data is only one 8 bytes value repeated n times
- 0x28 Zstd compressed data, this block can be directly read with a Zstd reader
Intrument(s) List Data Block data
Once Data Block Header has been read, data is as follows:
+------------+------------+------------------------------------------------------+
| bytes Size | Field Name | Description |
+------------+------------+------------------------------------------------------+
| 4 | I1Cs | Instrument 1 Code Size, string size (int32) |
+------------+------------+------------------------------------------------------+
| I1Cs | I1Name | Instrument 1 Code (no ending zero it's a Go string) |
+------------+------------+------------------------------------------------------+
| 4 | I1Fs | Instrument 1 Number of Fields (int32) |
+------------+------------+------------------------------------------------------+
| 8 | I1Foff | Offset of Instrument 1 Fields Description (int64) |
+------------+------------+------------------------------------------------------+
+------------+------------+------------------------------------------------------+
| 4 | I2Cs | Instrument 2 Code Size, string size (int32) |
+------------+------------+------------------------------------------------------+
| I2Cs | I2Name | Instrument 2 Code (no ending zero it's a Go string) |
+------------+------------+------------------------------------------------------+
| 4 | I2Fs | Instrument 2 Number of Fields (Little Endian int32) |
+------------+------------+------------------------------------------------------+
| 8 | I2Foff | Offset of Instrument 2 Fields Description (int64) |
+------------+------------+------------------------------------------------------+
+------------+------------+------------------------------------------------------+
| ... | ... | All instruments from 1 to Cs |
+------------+------------+------------------------------------------------------+
+------------+------------+------------------------------------------------------+
| 4 | ICsCs | Instrument Cs Code Size, string size (int32) |
+------------+------------+------------------------------------------------------+
| ICsCs | ICsName | Instrument Cs Code (no ending zero it's a Go string) |
+------------+------------+------------------------------------------------------+
| 4 | ICsFs | Instrument Cs Number of Fields (int32) |
+------------+------------+------------------------------------------------------+
| 8 | ICsFoff | Offset of Instrument Cs Fields Description (int64) |
+------------+------------+------------------------------------------------------+