compress
Byte buffers, compression methods from RLE through DEFLATE, gzip, bzip2, zstd and checksums.
import compress
This package is the base of all binary data work in Salam. ByteBuf, a growable byte buffer, and ByteReader, a cursor over bytes, are what packages such as encoding.binary, archive and encoding.msgpack pass around. On top of them it has gzip, bzip2, raw DEFLATE, a few teaching compressors (RLE, LZ77, LZW, Huffman, BWT), bit-level reading and writing, and the CRC-32 and Adler-32 checksums.
To pack several files into one .zip or .tar, use archive, which calls this package for the compression. To write numbers in a fixed byte order, use encoding.binary on the same buffers.
For compression, reach for gzip when the data has to be read by other programs. Use bzip2 when the result should be smaller than gzip and a slower pack is fine; it reads and writes standard .bz2 data. The other methods are mostly there to learn from and experiment with, and their output format belongs to this package alone. The examples show that for small data the overhead of Huffman and DEFLATE comes to more than the data itself.
What trips people up
- Gzip verifies its checksum.
GzipDecompresschecks the header, inflates the data, then compares both the CRC-32 and the uncompressed length in the trailer; a mismatch setsokfalse and returns an empty buffer. - Xz and LZMA writing is a fast match finder, not a tuned one. The packed size is within a few percent of
xzat the same level, never smaller. The whole input and the whole result are held in memory, and level 9 asks for a 64 MiB window. The result of a decoder is kept whole in memory too, so use the...Maxfunctions on data you did not make yourself. - Zstandard output is valid, not the smallest. The writer is simple, so
ZstdCompressLevelmakes frames usually a quarter to a half larger than thezstdtool at the same level; every reader still accepts them. Reading is complete, but onlyZstdDecompressDictandZstdDecompressExtake a dictionary, and the decoder holds the whole result in memory, so useZstdDecompressMaxon data from outside. - Bzip2 checks every checksum and can be capped.
Bzip2Decompressverifies each block CRC and the whole stream CRC and rejects broken tables and indexes, and it accepts several streams one after another and nothing else after them. A tiny file can expand to a huge one, so useBzip2DecompressMaxon data from outside. - The method is not stored.
Compressoutput does not say which method made it; keep the method number next to the data. - A
strstops at the first zero byte. Keep binary data in aByteBuf;BufToStrUnsafeis for bytes that are really text. - Buffers are not freed for you. Call
BufFreeon buffers you are done with, especially in loops and servers.
The guide below covers the jobs this package is used for day to day: gzipping and bzip2-ing text, laying out your own binary records in a ByteBuf, checking data with CRC-32, and choosing a compression method. Each section has a complete program with its output.
Gzip text for storage or transport
GzipCompressStr turns a string into a gzip stream that gzip -d, browsers and every other gzip reader understand, and GzipDecompressBuf turns it back. Gzip output is binary, so when it has to travel inside JSON, a URL or an environment variable, wrap it in Base64 with BufToBase64. On the receiving side encoding.Base64DecodeBytes writes the bytes into a buffer you have made big enough; set the buffer's len to the count it returns. It throws when the text is not Base64 or does not fit, so call it inside try:.
import compress
import encoding
func main:
mut log := ""
repeat 20:
log += "GET /api/users 200 12ms\n"
end
packed := compress.GzipCompressStr(log)
println "original:", log.Len(), "bytes"
println "gzipped: ", packed.len, "bytes"
println "ratio: ", compress.Ratio(packed.len, log.Len())
text := compress.BufToBase64(packed)
println "base64: ", text.Len(), "characters"
mut ok := false
mut wire := compress.NewByteBufCap(text.Len())
try:
wire.len = encoding.Base64DecodeBytes(text, wire.data, text.Len())
ok = true
catch e:
println "base64:", e.ToStr()
end
println "decoded: ", ok, compress.BufEquals(wire, packed)
back := compress.GzipDecompressBuf(wire, ok)
println "restored:", ok, compress.BufToStrUnsafe(back) == log
endA log with the same line twenty times shrinks to a tenth of its size. Base64 costs a third back (64 characters for 48 bytes), which is still far smaller than the original. BufToStrUnsafe copies the bytes into a string, so use it only for data that is really text: a str ends at the first zero byte.
Bzip2 for smaller files
Bzip2CompressStr, Bzip2Compress and Bzip2CompressBuf write a standard bzip2 stream (what bzip2 and Python's bz2 read), and Bzip2CompressLevel takes a level from 1 to 9: the block size in units of 100000 bytes, 9 by default. Bzip2DecompressBuf reads one or more streams, including old ones with randomised blocks, and checks all CRCs. The program packs the same text both ways and then shows the two safety nets: a cap on the unpacked size, and a flipped bit.
import compress
func main:
words := ["salam", "compress", "block", "sort", "stream", "byte", "table", "code", "bit", "run"]
mut text := ""
mut x := 7
repeat 4000:
x = (x * 75 + 74) % 65537
text += words[x % 10]
text += " "
end
gz := compress.GzipCompressStr(text)
bz := compress.Bzip2CompressStr(text)
println "original:", text.Len(), "bytes"
println "gzip: ", gz.len, "bytes"
println "bzip2: ", bz.len, "bytes"
mut ok := false
back := compress.Bzip2DecompressBuf(bz, ok)
println "restored:", ok, compress.BufToStrUnsafe(back) == text
_ := compress.Bzip2DecompressMax(bz.data, bz.len, 1000, ok)
println "capped at 1000 bytes:", ok
mut damaged := compress.BufClone(bz)
compress.BufSet(damaged, damaged.len / 2, compress.BufGet(damaged, damaged.len / 2) ^ 1)
_ := compress.Bzip2DecompressBuf(damaged, ok)
println "damaged:", ok
endOn text with many repeated words bzip2 beats gzip by a good margin, while on short or already compressed data the two are close. Bzip2 needs more time and memory per megabyte than gzip, so it suits files that are written once and read rarely. To work with .bz2 files on disk, use Bzip2CompressFile and Bzip2DecompressFile from archive.
Your own binary format with ByteBuf
A ByteBuf grows as you push into it, and a ByteReader walks over the bytes from the start. That is all you need for a small file format: a few magic bytes so that a wrong file is rejected, a count, and then each record as a length and its bytes. BufPushU32 and RdU32 write and read a 32 bit number in big-endian order; for little-endian or 64 bit numbers use encoding.binary, which works on the same buffers.
import compress
func write_names(names: Vector<str>): compress.ByteBuf:
mut b := compress.NewByteBuf()
compress.BufPushBuf(b, compress.BufFromStr("NAMS"))
compress.BufPushU32(b, names.Len())
each name in names:
compress.BufPushU32(b, name.Len())
compress.BufPushBuf(b, compress.BufFromStr(name))
end
ret b
end
func read_names(b: compress.ByteBuf):
mut r := compress.ReaderFromBuf(b)
magic := compress.RdBytes(r, 4)
if compress.BufToStrUnsafe(magic) != "NAMS":
println "not a names file"
ret
end
count := compress.RdU32(r)
repeat count in i:
n := compress.RdU32(r)
name := compress.RdBytes(r, n)
if r.bad:
println "record", i, "is cut short"
ret
end
println i, compress.BufToStrUnsafe(name)
end
end
func main:
mut names := Vector {} as Vector<str>
names.Push("Ada")
names.Push("Grace")
names.Push("Linus")
b := write_names(names)
println "bytes:", compress.BufLen(b)
read_names(b)
mut cut := compress.NewByteReader(b.data, 20)
read_names(compress.RdBytes(cut, 20))
endThe reader never reads past the end. When a read asks for more bytes than are left, the reader sets bad, returns zeros or an empty buffer, and stays bad, so it is enough to check r.bad once after a group of reads. The last call reads a copy cut to 20 bytes: the first name still fits, the second does not.
Checking data with CRC-32
A checksum tells you whether bytes arrived unchanged. CRC32Str and CRC32Buf compute the same CRC-32 that zip, gzip and PNG use; Adler32Str and Adler32Buf are faster and weaker, the one zlib uses. The program seals a message by appending its CRC, then shows that changing a single byte is caught.
import compress
func seal(payload: str): compress.ByteBuf:
mut b := compress.BufFromStr(payload)
compress.BufPushU32(b, compress.CRC32Str(payload) as int)
ret b
end
func check(b: compress.ByteBuf): bool:
if b.len < 4: ret false end
mut r := compress.NewByteReader(b.data, b.len)
body := compress.RdBytes(r, b.len - 4)
stored := compress.RdU32(r) as u32
ret compress.CRC32Buf(body) == stored
end
func gunzip_checked(g: compress.ByteBuf, ok &: bool): compress.ByteBuf:
out := compress.GzipDecompressBuf(g, ok)
if not ok: ret out end
at := g.len - 8
stored := compress.BufGet(g, at) | (compress.BufGet(g, at + 1) << 8) | (compress.BufGet(g, at + 2) << 16) | (compress.BufGet(g, at + 3) << 24)
ok = compress.CRC32Buf(out) == stored as u32
ret out
end
func main:
mut packet := seal("move 3 to slot 7")
println "length:", packet.len, "intact:", check(packet)
compress.BufSet(packet, 5, 52)
println "after one changed byte, intact:", check(packet)
g := compress.GzipCompressStr("move 3 to slot 7")
mut broken := compress.BufClone(g)
compress.BufSet(broken, broken.len - 6, compress.BufGet(broken, broken.len - 6) ^ 1)
mut ok := false
_ := compress.GzipDecompressBuf(broken, ok)
println "damaged gzip, plain decompress:", ok
_ := gunzip_checked(broken, ok)
println "damaged gzip, checked:", ok
_ := gunzip_checked(g, ok)
println "good gzip, checked:", ok
endGzipDecompress and GzipDecompressBuf check the header and the optional header CRC, inflate the data, and then compare the CRC-32 and the length in the last eight bytes of the stream. A flipped bit there, or a corruption that still decodes, is reported by ok being false, so you never see damaged data as a successful read.
Choosing a method
Compress, CompressBuf and CompressStr take a method number, and CompressBest tries each one and keeps the smallest result. The best method depends on the shape of the data, as these output sizes show:
import compress
import str
func methodName(m: int): str:
switch m:
compress.MethodRLE: ret "rle" end
compress.MethodHuffman: ret "huffman" end
compress.MethodLZ77: ret "lz77" end
compress.MethodLZW: ret "lzw" end
compress.MethodDeflate: ret "deflate" end
compress.MethodBWT: ret "bwt" end
end
ret "none (stored)"
end
func row(label: str, data: str):
b := compress.BufFromStr(data)
mut line := label + ": " + str.FromInt(b.len) + " bytes, rle " + str.FromInt(compress.CompressBuf(compress.MethodRLE, b).len)
line = line + ", lz77 " + str.FromInt(compress.CompressBuf(compress.MethodLZ77, b).len)
line = line + ", gzip " + str.FromInt(compress.GzipCompressBuf(b).len)
mut best := 0
smallest := compress.CompressBest(b.data, b.len, best)
println line
println " best:", methodName(best), "at", smallest.len, "bytes"
end
func main:
mut runs := ""
repeat 200: runs += "0" end
mut words := ""
repeat 10: words += "error at line 42; " end
row("runs", runs)
row("words", words)
row("short", "hi there")
endRLE wins on long runs of one byte and makes ordinary text bigger. LZ77 does well on repeated phrases. MethodDeflate is left out of the list because it is this package's own container, which stores its Huffman tables in full and so starts at almost 300 bytes; gzip uses real DEFLATE with compact tables and is the better choice whenever the data leaves your program. Two rules follow: the output of Compress does not say which method made it, so store the method number next to the data (the value CompressBest wrote into its last argument), and very short inputs are best stored as they are (method 0, which CompressBest reports as none for the short line above).
Read .xz and .lzma files
XzDecompress opens the .xz files that xz, tar -J and 7z a -txz write, and LzmaDecompress the older .lzma format. An xz file is checked as it is read: the headers, the index and the CRC-32, CRC-64 or SHA-256 of every block, and any mismatch gives ok false with an empty result. The decoders hold the whole result in memory, so on data you did not make yourself call XzDecompressMax or LzmaDecompressMax with the biggest size you accept; a few hundred bytes of LZMA can describe gigabytes.
import compress
const PACKED := """
fd377a585a000004e6d6b44604c05fcf022101160000000000000000bb0ccf6ce0014e00575d0029984986240afb8a34e4
81db780958f3d0144436dd3bb4ca45234051fa042e5be576402f66b72801fb0f3b421e32e3eba071037cdd425338444ede
4cb8c8c742828bc92c1b53df8ea763015d902c3d49f9c602aacf40a000005b35fff6fd65c4e300017bcf020000005e9efc
63b1c467fb020000000004595a
"""
func nib(c: int): int:
if c >= 48 and c <= 57:
ret c - 48
end
if c >= 97 and c <= 102:
ret c - 87
end
ret -1
end
func unhex(s: str): compress.ByteBuf:
mut b := compress.NewByteBuf()
bp := s as u8*
mut hi := -1
repeat s.Len() in i:
v := nib(bp[i] as int)
if v < 0:
continue
end
if hi < 0:
hi = v
else:
compress.BufPush(b, hi * 16 + v)
hi = -1
end
end
ret b
end
func main:
packed := unhex(PACKED)
mut ok := false
text := compress.XzDecompressBuf(packed, ok)
println "decoded:", ok, text.len, "bytes from", packed.len
println compress.BufToStrUnsafe(text)
mut damaged := compress.BufClone(packed)
compress.BufSet(damaged, damaged.len - 20, compress.BufGet(damaged, damaged.len - 20) ^ 1)
broken := compress.XzDecompressBuf(damaged, ok)
println "damaged file:", ok, broken.len
small := compress.XzDecompressMax(packed.data, packed.len, 100, ok)
println "capped at 100 bytes:", ok, small.len
endWrite .xz and .lzma files
XzCompressStr, XzCompress and XzCompressBuf write a standard .xz file that xz, unxz, tar -J and 7-Zip open: one LZMA2 block, a CRC-64 check, and an index. XzCompressLevel takes a level from 0 (fastest, 256 KiB window) to 9 (64 MiB window, longest search), and XzCompressCheck also picks the check: XzCheckNone, XzCheckCrc32, XzCheckCrc64 or XzCheckSha256. Data that does not shrink is stored in uncompressed chunks, so the result is only the container overhead over the input. LzmaCompressStr, LzmaCompress, LzmaCompressBuf and LzmaCompressLevel write the older .lzma format, which has no check; archive.XzCompressFile and archive.LzmaCompressFile do it for a file.
/*
* Salam Programming Language (2024-2026)
*
* +-------------------+
* | S A L A M |
* +-------------------+
*
* Designed by Seyyed Ali Mohammadiyeh and the Salam Team
* Born from a decade of language design experience (since 2018)
*
* Repository: https://github.com/SalamLang/Salam
*
*/
package main
import compress
func main:
text := "the quick brown fox jumps over the lazy dog. " * 50
mut ok := false
xz := compress.XzCompressStr(text)
back := compress.XzDecompressBuf(xz, ok)
println ".xz:", text.Len(), "to", xz.len, "bytes", ok, back.len
println "magic:", compress.BufGet(xz, 0), compress.BufGet(xz, 1), compress.BufGet(xz, 2)
fast := compress.XzCompressLevel(text as void*, text.Len(), 0)
best := compress.XzCompressLevel(text as void*, text.Len(), 9)
println "levels:", fast.len, best.len
none := compress.XzCompressCheck(text as void*, text.Len(), 6, compress.XzCheckNone)
sha := compress.XzCompressCheck(text as void*, text.Len(), 6, compress.XzCheckSha256)
println "checks:", none.len, xz.len, sha.len
small := compress.XzDecompressBuf(sha, ok)
println "sha256 stream:", ok, small.len
lz := compress.LzmaCompressStr(text)
again := compress.LzmaDecompressBuf(lz, ok)
println ".lzma:", lz.len, "bytes", ok, again.len
lz9 := compress.LzmaCompressLevel(text as void*, text.Len(), 9)
println "level 9:", lz9.len
buf := compress.LzmaCompressBuf(back)
println "buffer:", compress.BufEquals(buf, lz)
println "empty:", compress.XzCompressBuf(compress.NewByteBuf()).len
endInside 7z archives and xz blocks the compressed data comes without a container. LzmaRawDecompress and Lzma2RawDecompress read that form when you know the properties and the unpacked size, and BcjDecode and DeltaDecode undo the filters that sit in front of it. Dictionaries above 1 GiB (LzmaMaxDictSize) are refused.
Read and write .zst files
ZstdDecompress opens the .zst files that zstd, tar --zstd and 7z a -tzstd write: one frame or several in a row, any compression level, with or without a checksum, and frames written with long-distance matching or many threads. The decoder checks as it reads. A bad table, an offset that points before the data, a size that does not add up or a wrong XXH64 checksum gives ok false and an empty result; ZstdDecompressEx also says which of these it was, through the ZstdErr... codes. The whole result is kept in memory, so on data you did not make yourself call ZstdDecompressMax with the biggest size you accept.
import compress
const PACKED := """
28b52ffd6459008502009285111290cf0184e3820d4c2042136477707af2ed3a409c9edcb0e3e7730fed14fb943c8038
53dea7a9ea4f2fc3f0527dda9e972a958b464c916376e4546b1cd18bf07ca9de9831090100407fa04214ee20b3a1
"""
func nib(c: int): int:
if c >= 48 and c <= 57:
ret c - 48
end
if c >= 97 and c <= 102:
ret c - 87
end
ret -1
end
func unhex(s: str): compress.ByteBuf:
mut b := compress.NewByteBuf()
bp := s as u8*
mut hi := -1
repeat s.Len() in i:
v := nib(bp[i] as int)
if v < 0:
continue
end
if hi < 0:
hi = v
else:
compress.BufPush(b, hi * 16 + v)
hi = -1
end
end
ret b
end
func main:
packed := unhex(PACKED)
mut ok := false
text := compress.ZstdDecompressBuf(packed, ok)
println "decoded:", ok, text.len, "bytes from", packed.len
println compress.BufToStrUnsafe(text)
mut damaged := compress.BufClone(packed)
compress.BufSet(damaged, damaged.len - 20, compress.BufGet(damaged, damaged.len - 20) ^ 1)
mut status := 0
broken := compress.ZstdDecompressEx(damaged.data, damaged.len, 0, null, 0, status)
println "damaged file:", status, compress.ZstdStatusText(status), broken.len
small := compress.ZstdDecompressMax(packed.data, packed.len, 100, ok)
println "capped at 100 bytes:", ok, small.len
mine := compress.ZstdCompressLevel(text.data, text.len, 19)
again := compress.ZstdDecompressBuf(mine, ok)
println "our own frame:", mine.len, "bytes,", compress.BufEquals(again, text)
println "size recorded in the header:", compress.ZstdContentSize(mine.data, mine.len)
endZstdCompressLevel writes a standard frame at a level from 1 to 22 (0 stores the data as it is), with the content size and a checksum in the header. The writer is deliberately simple: it finds repeats with a greedy or lazy search over a hash chain and codes them with the predefined tables, so the result is usually a quarter to a half larger than the zstd tool's at the same level. Every reader accepts it. Dictionaries are supported for reading only (ZstdDecompressDict); the writer does not use one.
Unpack a big file without holding it in memory
Everything above works on a whole buffer. For a file that may not fit in memory, NewFileBitReader reads a part of a file in pieces of 64 KiB and RawInflateToSink writes the unpacked data into a FileSink, which counts the bytes, keeps a CRC-32 and stops at a cap you choose. The archive package uses exactly these calls to unpack zip, gzip and RAR entries to disk. FileSeek, FileSize and FileReadAt use 64 bit offsets, and CRC32Update lets you checksum data that arrives in parts.
import compress
import mem
func main:
text := "stream me, stream me, stream me. " * 100
packed := compress.RawDeflate(text as void*, text.Len())
mut sink := compress.SinkOpen("packed.bin", 0 as i64)
println compress.SinkWrite(sink, packed.data, packed.len), sink.written == (packed.len as i64), sink.failed
println compress.SinkClose(sink), sink.fp == null
fp := compress.FileOpenRead("packed.bin")
size := compress.FileSize(fp)
println size == (packed.len as i64)
head := mem.Allocate(4 as u64)
got := compress.FileReadAt(fp, 0 as i64, head, 4)
println got, compress.FileSeek(fp, 0 as i64)
mem.Free(head)
mut reader := compress.NewFileBitReader(fp, 0 as i64, size)
println reader.fp == fp, reader.left == size
mut out := compress.SinkOpen("text.txt", 0 as i64)
ok := compress.RawInflateToSink(reader, out, false)
compress.FileBitReaderFree(reader)
println ok, compress.SinkClose(out), out.written == (text.Len() as i64), out.crc == compress.CRC32Str(text)
mut small := compress.SinkOpen("small.txt", 100 as i64)
mut again := compress.NewFileBitReader(fp, 0 as i64, size)
println compress.RawInflateToSink(again, small, false), small.over, small.cap
compress.FileBitReaderFree(again)
compress.SinkClose(small)
mut part := compress.SinkOpen("part.bin", 0 as i64)
println compress.SinkCopyFrom(part, fp, 2 as i64, 10 as i64), part.written
compress.SinkClose(part)
compress.FileClose(fp)
first := "hello, "
second := "world"
running := compress.CRC32Update(compress.CRC32Update(0 as u32, first as void*, first.Len()), second as void*, second.Len())
println running == compress.CRC32Str(first + second)
wf := compress.FileOpenWrite("empty.bin")
println compress.FileSize(wf)
compress.FileClose(wf)
println compress.FileOpenRead("no-such-file.bin") == null
endA sink that reaches its cap sets over and refuses the rest, so a small hostile file cannot fill the disk. Check the result of SinkClose too: a full disk shows up there as a failed write.
Which function for which job
| Job | Use |
|---|---|
| Data that other programs will read | GzipCompressStr, GzipCompressBuf, GzipDecompressBuf |
Write a .xz or .lzma file | XzCompressStr, XzCompressLevel, XzCompressCheck, LzmaCompressStr, LzmaCompressLevel |
Read a .xz or .lzma file | XzDecompressBuf, LzmaDecompressBuf; the ...Max variants for data from outside |
Read or write a .zst file | ZstdCompressBuf, ZstdCompressLevel, ZstdDecompressBuf, and ZstdDecompressMax for data from outside |
Smaller data than gzip, or a .bz2 stream | Bzip2CompressStr, Bzip2CompressLevel, Bzip2DecompressBuf, and Bzip2DecompressMax for untrusted input |
| Standard DEFLATE with no header, as zip entries and zlib streams hold it | RawDeflate, and RawInflate with the original size |
| Smallest result for data only you read | CompressBest, then Decompress with the method it chose |
| Build bytes, then read them back | NewByteBuf, BufPush..., ReaderFromBuf, Rd... |
| Fields narrower than a byte | NewBitWriter, BitWriteBits, BitReadBits |
| Detect damaged data | CRC32Buf, or Adler32Buf when speed matters more |
| Binary into text | BufToBase64, and encoding.Base64DecodeBytes back |
Buffers are not freed for you. Call BufFree on a buffer you no longer need when a program makes many of them, for example in a loop or a long running server.
Types
FileSink
struct FileSink| Name | Type | Description |
|---|---|---|
fp | void* | The open file; null after SinkClose. |
written | i64 = 0 | The number of bytes written so far. |
crc | u32 = 0 | The CRC-32 of the bytes written so far. |
cap | i64 = 0 | The most bytes the sink accepts; zero means no limit. |
over | bool = false | A write would have passed cap and was refused. |
failed | bool = false | The file could not be created or a write to it failed. |
emit | func(void*, int, void*): bool | The function a sink made by SinkCallback passes each piece to; return false to stop the decoder. |
has_emit | bool = false | True when the sink calls emit. |
ctx | void* | The value handed to emit as its third argument. |
import compress
import mem
func main:
text := "stream me, stream me, stream me. " * 100
packed := compress.RawDeflate(text as void*, text.Len())
mut sink := compress.SinkOpen("packed.bin", 0 as i64)
println compress.SinkWrite(sink, packed.data, packed.len), sink.written == (packed.len as i64), sink.failed
println compress.SinkClose(sink), sink.fp == null
fp := compress.FileOpenRead("packed.bin")
size := compress.FileSize(fp)
println size == (packed.len as i64)
head := mem.Allocate(4 as u64)
got := compress.FileReadAt(fp, 0 as i64, head, 4)
println got, compress.FileSeek(fp, 0 as i64)
mem.Free(head)
mut reader := compress.NewFileBitReader(fp, 0 as i64, size)
println reader.fp == fp, reader.left == size
mut out := compress.SinkOpen("text.txt", 0 as i64)
ok := compress.RawInflateToSink(reader, out, false)
compress.FileBitReaderFree(reader)
println ok, compress.SinkClose(out), out.written == (text.Len() as i64), out.crc == compress.CRC32Str(text)
mut small := compress.SinkOpen("small.txt", 100 as i64)
mut again := compress.NewFileBitReader(fp, 0 as i64, size)
println compress.RawInflateToSink(again, small, false), small.over, small.cap
compress.FileBitReaderFree(again)
compress.SinkClose(small)
mut part := compress.SinkOpen("part.bin", 0 as i64)
println compress.SinkCopyFrom(part, fp, 2 as i64, 10 as i64), part.written
compress.SinkClose(part)
compress.FileClose(fp)
first := "hello, "
second := "world"
running := compress.CRC32Update(compress.CRC32Update(0 as u32, first as void*, first.Len()), second as void*, second.Len())
println running == compress.CRC32Str(first + second)
gen := (off: i64, dst: void*, n: int, _c: void*):
p := dst as u8*
repeat n in k:
p[k] = ((off + (k as i64)) & (255 as i64)) as u8
end
ret n
end
marker := mem.Allocate(8 as u64)
cfp := compress.CallbackOpenRead(1000 as i64, gen, marker)
println compress.FileSize(cfp), compress.CallbackContext(cfp) == marker
probe := mem.Allocate(4 as u64)
println compress.FileReadAt(cfp, 258 as i64, probe, 4), (probe as u8*)[0], (probe as u8*)[3]
compress.FileClose(cfp)
mem.Free(probe)
mem.Free(marker)
call := mem.AllocateZeroed(16 as u64) as u8*
call[0] = 0xE8 as u8
call[2] = 1 as u8
mut mask := 0
done := compress.BcjStep(compress.BcjX86, call as void*, 16, 0 as i64, mask)
println done, call[1] as int, call[2] as int, call[3] as int, call[4] as int, mask
mem.Free(call as void*)
wf := compress.FileOpenWrite("empty.bin")
println compress.FileSize(wf)
compress.FileClose(wf)
println compress.FileOpenRead("no-such-file.bin") == null
endMethod
enum Method| Name | Type | Description |
|---|---|---|
None | | No compression. |
RLE | | Run length encoding. |
Huffman | | Huffman. |
LZ77 | | LZ77. |
LZW | | LZW. |
Deflate | | LZ77 and Huffman. |
BWT | | Burrows Wheeler. |
import compress
func main:
text := "ababababab hello hello hello hello hello hello hello hello"
b := compress.BufFromStr(text)
each method in [compress.MethodNone, compress.MethodRLE, compress.MethodHuffman, compress.MethodLZ77, compress.MethodLZW, compress.MethodDeflate, compress.MethodBWT]:
mut c := compress.CompressBuf(method, b)
mut back := compress.DecompressBuf(method, c)
println method, compress.BufLen(c), compress.BufEquals(b, back)
compress.BufFree(c)
compress.BufFree(back)
end
mut best := 0
c := compress.CompressBest(b.data, b.len, best)
println best, compress.Ratio(c.len, b.len) < 1.0
s := compress.CompressStr(compress.MethodDeflate, "aaaaaaaaaa")
b3 := compress.Decompress(compress.MethodDeflate, s.data, s.len)
println compress.BufToStrUnsafe(b3)
b4 := compress.Compress(compress.MethodRLE, b.data, b.len)
println compress.BufLen(b4) > 0, compress.Method.Huffman
println compress.BufLen(compress.RLEDecompress(compress.RLECompress(b.data, b.len).data, compress.RLECompress(b.data, b.len).len)) == b.len
h := compress.HuffmanCompress(b.data, b.len)
println compress.BufEquals(compress.HuffmanDecompress(h.data, h.len), b)
l := compress.LZ77Compress(b.data, b.len)
println compress.BufEquals(compress.LZ77Decompress(l.data, l.len), b)
w := compress.LZWCompress(b.data, b.len)
println compress.BufEquals(compress.LZWDecompress(w.data, w.len), b)
bw := compress.BWTCompress(b.data, b.len)
println compress.BufEquals(compress.BWTDecompress(bw.data, bw.len), b)
mut primary := 0
t := compress.BWTTransform(b.data, 3, primary)
println primary, compress.BufToStrUnsafe(compress.BWTInverse(t.data, t.len, primary)) == "aba"
m := compress.MTFEncode(b)
println compress.BufEquals(compress.MTFDecode(m), b)
mut good := false
println compress.BufEquals(compress.RLEDecompressMax(b4.data, b4.len, 0, good), b), good
println compress.BufEquals(compress.HuffmanDecompressMax(h.data, h.len, 0, good), b), good
println compress.BufEquals(compress.LZ77DecompressMax(l.data, l.len, 0, good), b), good
println compress.BufEquals(compress.LZWDecompressMax(w.data, w.len, 0, good), b), good
println compress.BufEquals(compress.BWTDecompressMax(bw.data, bw.len, 0, good), b), good
df := compress.DeflateCompress(b.data, b.len)
println compress.BufEquals(compress.DeflateDecompressMax(df.data, df.len, 0, good), b), good
println compress.BufLen(compress.DecompressMax(compress.MethodLZ77, l.data, l.len, 1, good)), good
println compress.BufLen(compress.DecompressMax(compress.MethodLZ77, "not compressed data" as void*, 19, 0, good)), good
endByteBuf
struct ByteBuf| Name | Type | Description |
|---|---|---|
data | void* | The memory address of the bytes. |
len | int = 0 | The number of bytes. |
cap | int = 0 | The capacity. |
import compress
func main:
mut empty := compress.NewByteBuf()
println compress.BufLen(empty)
compress.BufFree(empty)
mut b := compress.NewByteBufCap(16)
compress.BufPush(b, 72)
compress.BufPush(b, 105)
mut other := compress.BufFromStr("!!")
compress.BufPushBuf(b, other)
compress.BufPushBytes(b, other.data, 1)
compress.BufPushU32(b, 10)
println compress.BufLen(b), compress.BufGet(b, 0), compress.BufToBase64(b)
compress.BufSet(b, 0, 104)
mut c := compress.BufClone(b)
println compress.BufEquals(b, c), compress.BufToStrUnsafe(compress.BufFromStr("hello"))
mut r := compress.ReaderFromBuf(b)
println compress.RdLeft(r), compress.RdByte(r), compress.RdByte(r)
three := compress.RdBytes(r, 3)
println compress.BufLen(three), compress.RdU32(r), compress.RdLeft(r), r.bad
_ := compress.RdByte(r)
println r.bad
mut r2 := compress.NewByteReader(c.data, 2)
println compress.RdByte(r2)
compress.BufReset(b)
println compress.BufLen(b)
compress.BufFree(b)
compress.BufFree(c)
compress.BufFree(other)
endByteReader
struct ByteReader| Name | Type | Description |
|---|---|---|
data | void* | The address of the data. |
len | int = 0 | The length of the data. |
pos | int = 0 | The current position. |
bad | bool = false | A read went past the end of the data. |
The example for this item is shown with the ones above: compress-Buffer.salam
BitWriter
struct BitWriter| Name | Type | Description |
|---|---|---|
buf | ByteBuf | The bytes written. |
acc | int = 0 | The bits waiting. |
nbits | int = 0 | The number of bits waiting. |
import compress
func main:
mut w := compress.NewBitWriter()
compress.BitWriteBit(w, 1)
compress.BitWriteBits(w, 5, 3)
compress.BitWriteBits(w, 255, 8)
compress.BitFlush(w)
println compress.BitWriterLen(w)
mut r := compress.BitReaderFromBuf(w.buf)
println compress.BitReadBit(r), compress.BitReadBits(r, 3), compress.BitReadBits(r, 8)
mut r2 := compress.NewBitReader(w.buf.data, 1)
println compress.BitReadBits(r2, 4)
compress.BitWriterFree(w)
endBitReader
struct BitReader| Name | Type | Description |
|---|---|---|
data | void* | The data. |
len | int = 0 | The length of the data. |
pos | int = 0 | The current byte position. |
acc | int = 0 | The bits read. |
nbits | int = 0 | The number of bits ready. |
bad | bool = false | A read went past the end of the data. |
fp | void* | The file a bit reader made by NewFileBitReader reads from; null for a reader over memory. |
left | i64 = 0 | The number of bytes of the file that are not read yet. |
foff | i64 = 0 | Where in the file the next piece is read from, when the reader was made by NewFileBitReader. |
The example for this item is shown with the ones above: compress-Bits.salam
Constants and variables
DecodeOk
const DecodeOk: i32 = 0why value 0: the data was decoded and every check passed.import compress
func name(why: int): str:
if why == compress.DecodeOk: ret "ok" end
if why == compress.DecodeTooBig: ret "too big" end
ret "corrupt"
end
func main:
raw := compress.BufFromStr("abcabcabcabcabcabcabcabcabcabcabcabc")
gz := compress.GzipCompress(raw.data, raw.len)
xz := compress.XzCompress(raw.data, raw.len)
bz := compress.Bzip2Compress(raw.data, raw.len)
lz := compress.LzmaCompress(raw.data, raw.len)
mut why := 0
_ := compress.GzipDecompressWhy(gz.data, gz.len, 10, why)
println "gzip", name(why)
_ := compress.XzDecompressWhy(xz.data, xz.len, 10, why)
println "xz", name(why)
_ := compress.Bzip2DecompressWhy(bz.data, bz.len, 10, why)
println "bzip2", name(why)
_ := compress.LzmaDecompressWhy(lz.data, lz.len, 10, why)
println "lzma", name(why)
back := compress.GzipDecompressWhy(gz.data, gz.len, 0, why)
println "no limit", name(why), back.len
_ := compress.GzipDecompressWhy(gz.data, gz.len - 6, 0, why)
println "cut", name(why)
endDecodeCorrupt
const DecodeCorrupt: i32 = 1why value 1: the data is damaged, cut short, not the expected format, or its checksum does not match.The example for this item is shown with the ones above: compress-Why.salam
DecodeTooBig
const DecodeTooBig: i32 = 2why value 2: the data is valid so far but would unpack to more than the limit you gave.The example for this item is shown with the ones above: compress-Why.salam
DecodeIo
const DecodeIo: i32 = 3import archive
import compress
import mem
import os
import str
func collect(p: void*, n: int, c: void*): bool:
compress.BufPushBytes((c as compress.ByteBuf*)[0], p, n)
ret true
end
func main:
mut text := compress.NewByteBuf()
repeat 5000 in i:
line := "line " + str.FromInt((i * 7) as i64) + "\n"
compress.BufPushBytes(text, line as void*, line.Len())
end
mut packed := compress.Bzip2Compress(text.data, text.len)
archive.WriteBinaryFileBuf("bz.tmp", packed)
fp := compress.FileOpenRead("bz.tmp")
got := mem.AllocateZeroed(sizeof(compress.ByteBuf)) as compress.ByteBuf*
got[0] = compress.NewByteBuf()
mut sink := compress.SinkCallback(0 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
st := compress.Bzip2FileToSink(fp, 0 as i64, compress.FileSize(fp), sink)
println st == compress.DecodeOk, sink.written == (text.len as i64), compress.BufEquals(got[0], text)
compress.FileClose(fp)
fp2 := compress.FileOpenRead("bz.tmp")
mut small := compress.SinkCallback(100 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
println compress.Bzip2FileToSink(fp2, 0 as i64, compress.FileSize(fp2), small) == compress.DecodeTooBig
compress.FileClose(fp2)
println compress.DecodeIo
compress.BufFree(got[0])
mem.Free(got as void*)
compress.BufFree(text)
compress.BufFree(packed)
os.Remove("bz.tmp")
endXzCheckNone
const XzCheckNone: i32 = 0/*
* Salam Programming Language (2024-2026)
*
* +-------------------+
* | S A L A M |
* +-------------------+
*
* Designed by Seyyed Ali Mohammadiyeh and the Salam Team
* Born from a decade of language design experience (since 2018)
*
* Repository: https://github.com/SalamLang/Salam
*
*/
package main
import compress
func main:
text := "the quick brown fox jumps over the lazy dog. " * 50
mut ok := false
xz := compress.XzCompressStr(text)
back := compress.XzDecompressBuf(xz, ok)
println ".xz:", text.Len(), "to", xz.len, "bytes", ok, back.len
println "magic:", compress.BufGet(xz, 0), compress.BufGet(xz, 1), compress.BufGet(xz, 2)
fast := compress.XzCompressLevel(text as void*, text.Len(), 0)
best := compress.XzCompressLevel(text as void*, text.Len(), 9)
println "levels:", fast.len, best.len
none := compress.XzCompressCheck(text as void*, text.Len(), 6, compress.XzCheckNone)
sha := compress.XzCompressCheck(text as void*, text.Len(), 6, compress.XzCheckSha256)
println "checks:", none.len, xz.len, sha.len
small := compress.XzDecompressBuf(sha, ok)
println "sha256 stream:", ok, small.len
lz := compress.LzmaCompressStr(text)
again := compress.LzmaDecompressBuf(lz, ok)
println ".lzma:", lz.len, "bytes", ok, again.len
lz9 := compress.LzmaCompressLevel(text as void*, text.Len(), 9)
println "level 9:", lz9.len
buf := compress.LzmaCompressBuf(back)
println "buffer:", compress.BufEquals(buf, lz)
println "empty:", compress.XzCompressBuf(compress.NewByteBuf()).len
endXzCheckCrc32
const XzCheckCrc32: i32 = 1The example for this item is shown with the ones above: compress-XzWrite.salam
XzCheckCrc64
const XzCheckCrc64: i32 = 4xz.The example for this item is shown with the ones above: compress-XzWrite.salam
XzCheckSha256
const XzCheckSha256: i32 = 10The example for this item is shown with the ones above: compress-XzWrite.salam
BcjX86
const BcjX86: i32 = 4import compress
import str
const RAW_LZMA := """
0029984986240afb8a34e481db780958f3d0144436dd3bb4ca45234051fa042e5be576402f66b72801fb0f3b421e32e3eb
a071037cdd425338444ede4cb8c8c742828bc92c1b53df8ea763015d902c3d49f9c602ad8d68e0bcfed44dc0
"""
const RAW_LZMA2 := """
e0014e00595d0029984986240afb8a34e481db780958f3d0144436994e538c60ac746c02dc278329cadecbb68a986c0818
f3d60fca80edf59196fa1d512909ac7b63cbe3650a7fe9860c6f9bfc85ae148860e1e0b15ff7ea8080b5235108050000
"""
func nib(c: int): int:
if c >= 48 and c <= 57:
ret c - 48
end
if c >= 97 and c <= 102:
ret c - 87
end
ret -1
end
func unhex(s: str): compress.ByteBuf:
mut b := compress.NewByteBuf()
bp := s as u8*
mut hi := -1
repeat s.Len() in i:
v := nib(bp[i] as int)
if v < 0:
continue
end
if hi < 0:
hi = v
else:
compress.BufPush(b, hi * 16 + v)
hi = -1
end
end
ret b
end
func show(label: str, b: compress.ByteBuf):
mut line := label
repeat b.len in i:
v := compress.BufGet(b, i)
line += " "
line += str.FromInt(v as i64)
end
println line
end
func main:
println "dictionary limit:", compress.LzmaMaxDictSize
props := unhex("5d00000100")
raw1 := unhex(RAW_LZMA)
mut ok := false
a := compress.LzmaRawDecompress(props.data, props.len, raw1.data, raw1.len, 335, 0, ok)
println "lzma raw:", ok, a.len
raw2 := unhex(RAW_LZMA2)
b := compress.Lzma2RawDecompress(16, raw2.data, raw2.len, 335, 0, ok)
println "lzma2 raw:", ok, compress.BufEquals(a, b)
c := compress.Lzma2RawDecompress(16, raw2.data, raw2.len, 100, 0, ok)
println "lzma2 raw, wrong size:", ok, c.len
call := unhex("e810000000909090")
show("before", call)
println compress.BcjDecode(compress.BcjX86, call.data, call.len, 0)
show("x86 ", call)
steps := unhex("01010101")
println compress.DeltaDecode(steps.data, steps.len, 1)
show("delta ", steps)
println compress.BcjDecode(99, steps.data, steps.len, 0), compress.DeltaDecode(steps.data, steps.len, 0)
endBcjPowerPC
const BcjPowerPC: i32 = 5The example for this item is shown with the ones above: compress-XzRaw.salam
BcjIA64
const BcjIA64: i32 = 6The example for this item is shown with the ones above: compress-XzRaw.salam
BcjARM
const BcjARM: i32 = 7The example for this item is shown with the ones above: compress-XzRaw.salam
BcjARMThumb
const BcjARMThumb: i32 = 8The example for this item is shown with the ones above: compress-XzRaw.salam
BcjSPARC
const BcjSPARC: i32 = 9The example for this item is shown with the ones above: compress-XzRaw.salam
BcjARM64
const BcjARM64: i32 = 10The example for this item is shown with the ones above: compress-XzRaw.salam
BcjRISCV
const BcjRISCV: i32 = 11The example for this item is shown with the ones above: compress-XzRaw.salam
Method numbers Method…
Method enum cases as plain numbers, for Compress and Decompress.| Name | Value |
|---|---|
MethodNone | 0 |
MethodRLE | 1 |
MethodHuffman | 2 |
MethodLZ77 | 3 |
MethodLZW | 4 |
MethodDeflate | 5 |
MethodBWT | 6 |
The example for this item is shown with the ones above: compress-Methods.salam
ZstdOk
const ZstdOk: i32 = 0import compress
import str
func main:
mut text := compress.NewByteBuf()
repeat 60 in i:
line := "entry " + str.FromInt(i * 37 % 101) + ": Salam writes zstd frames\n"
compress.BufPushBytes(text, line as void*, line.Len())
end
packed := compress.ZstdCompress(text.data, text.len)
println "default level:", packed.len, "bytes from", text.len
println "is zstd:", compress.IsZstd(packed.data, packed.len), compress.IsZstdBuf(packed), compress.IsZstd(text.data, text.len)
println "size in the header:", compress.ZstdContentSize(packed.data, packed.len)
mut ok := false
back := compress.ZstdDecompress(packed.data, packed.len, ok)
println "decoded:", ok, compress.BufEquals(back, text)
back2 := compress.ZstdDecompressBuf(packed, ok)
println "buffer form:", ok, back2.len
fast := compress.ZstdCompressLevel(text.data, text.len, 1)
best := compress.ZstdCompressLevel(text.data, text.len, 19)
stored := compress.ZstdCompressLevel(text.data, text.len, 0)
println "levels 1 and 19 smaller than stored:", fast.len < stored.len, best.len < stored.len
println "stored adds", stored.len - text.len, "bytes of framing"
plain := compress.ZstdCompressOpts(text.data, text.len, 3, false)
println "no checksum saves", packed.len - plain.len, "bytes"
fromstr := compress.ZstdCompressStr("hello hello hello hello")
fromstr_buf := compress.ZstdCompressBuf(text)
println "string and buffer forms:", fromstr.len, fromstr_buf.len
mut status := 0
capped := compress.ZstdDecompressEx(packed.data, packed.len, 100, null, 0, status)
println "capped at 100:", status, compress.ZstdStatusText(status), capped.len
limit := compress.ZstdDecompressMax(packed.data, packed.len, text.len, ok)
println "capped at the exact size:", ok, limit.len
bad := compress.BufFromStr("not a zstd stream")
nope := compress.ZstdDecompressEx(bad.data, bad.len, 0, null, 0, status)
println "garbage:", status, compress.ZstdStatusText(status), nope.len
println "codes:", compress.ZstdOk, compress.ZstdErrFormat, compress.ZstdErrTruncated, compress.ZstdErrCorrupt, compress.ZstdErrChecksum, compress.ZstdErrTooBig, compress.ZstdErrDict
endZstdErrFormat
const ZstdErrFormat: i32 = 1The example for this item is shown with the ones above: compress-Zstd.salam
ZstdErrTruncated
const ZstdErrTruncated: i32 = 2The example for this item is shown with the ones above: compress-Zstd.salam
ZstdErrCorrupt
const ZstdErrCorrupt: i32 = 3The example for this item is shown with the ones above: compress-Zstd.salam
ZstdErrChecksum
const ZstdErrChecksum: i32 = 4The example for this item is shown with the ones above: compress-Zstd.salam
ZstdErrTooBig
const ZstdErrTooBig: i32 = 5The example for this item is shown with the ones above: compress-Zstd.salam
ZstdErrDict
const ZstdErrDict: i32 = 6The example for this item is shown with the ones above: compress-Zstd.salam
LzmaMaxDictSize
const LzmaMaxDictSize: i32 = 1073741824ok false.The example for this item is shown with the ones above: compress-XzRaw.salam
Functions
MTFEncode
func MTFEncode(inbuf: ByteBuf): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
MTFDecode
func MTFDecode(inbuf: ByteBuf): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
BWTTransform
func BWTTransform(p: void*, n: int, primary_out &: int): ByteBufprimary_out.The example for this item is shown with the ones above: compress-Methods.salam
BWTInverse
func BWTInverse(p: void*, n: int, primary: int): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
BWTCompress
func BWTCompress(p: void*, n: int): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
BWTDecompressMax
func BWTDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBufBWTDecompress but with a cap on the output size (0 means 1 GiB) and an ok flag. Damaged, truncated or oversized input gives ok false and an empty buffer; it never reads or writes out of bounds.The example for this item is shown with the ones above: compress-Methods.salam
BWTDecompress
func BWTDecompress(p: void*, n: int): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
LZ77Compress
func LZ77Compress(p: void*, n: int): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
LZ77DecompressMax
func LZ77DecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBufLZ77Decompress but with a cap on the output size (0 means 1 GiB) and an ok flag. Damaged, truncated or oversized input gives ok false and an empty buffer; it never reads or writes out of bounds.The example for this item is shown with the ones above: compress-Methods.salam
LZ77Decompress
func LZ77Decompress(p: void*, n: int): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
ZstdCompressFileToSink
func ZstdCompressFileToSink(fp: void*, off: i64, len: i64, level: int, sink &: FileSink): boollen bytes of the open file fp from off into one zstd frame with a checksum and writes it to a sink block by block. The input is held in a buffer of about two windows (a window is 1 MiB at the default level), so the size of the file does not matter. The bytes are the same as ZstdCompressLevel would give. level below 1 stores the blocks raw.import archive
import compress
import mem
import os
import str
func collect(p: void*, n: int, c: void*): bool:
compress.BufPushBytes((c as compress.ByteBuf*)[0], p, n)
ret true
end
func main:
mut text := compress.NewByteBuf()
repeat 20000 in i:
line := "record " + str.FromInt((i * 13) as i64) + " ok\n"
compress.BufPushBytes(text, line as void*, line.Len())
end
archive.WriteBinaryFileBuf("sink.tmp", text)
fp := compress.FileOpenRead("sink.tmp")
size := compress.FileSize(fp)
out := mem.AllocateZeroed(sizeof(compress.ByteBuf)) as compress.ByteBuf*
emit := (p: void*, n: int, c: void*): ret collect(p, n, c) end
out[0] = compress.NewByteBuf()
mut bz := compress.SinkCallback(0 as i64, emit, out as void*)
println compress.Bzip2CompressFileToSink(fp, 0 as i64, size, 9, bz)
mut one := compress.Bzip2CompressLevel(text.data, text.len, 9)
println "bzip2 same bytes", compress.BufEquals(out[0], one)
mut ok := false
mut back := compress.Bzip2Decompress(out[0].data, out[0].len, ok)
println "bzip2 round trip", ok, compress.BufEquals(back, text)
compress.BufFree(back)
compress.BufFree(one)
compress.BufFree(out[0])
out[0] = compress.NewByteBuf()
mut zs := compress.SinkCallback(0 as i64, emit, out as void*)
println compress.ZstdCompressFileToSink(fp, 0 as i64, size, 3, zs)
one = compress.ZstdCompressLevel(text.data, text.len, 3)
println "zstd same bytes", compress.BufEquals(out[0], one)
back = compress.ZstdDecompress(out[0].data, out[0].len, ok)
println "zstd round trip", ok, compress.BufEquals(back, text)
compress.BufFree(back)
compress.BufFree(one)
compress.BufFree(out[0])
out[0] = compress.NewByteBuf()
mut lz := compress.SinkCallback(0 as i64, emit, out as void*)
println compress.LzmaCompressFileToSink(fp, 0 as i64, size, 6, true, lz)
back = compress.LzmaRawDecompress(out[0].data, 5, ((out[0].data as i64) + (5 as i64)) as void*, out[0].len - 5, text.len, 0, ok)
println "lzma round trip", ok, compress.BufEquals(back, text)
compress.BufFree(back)
compress.BufFree(out[0])
compress.FileClose(fp)
mem.Free(out as void*)
compress.BufFree(text)
os.Remove("sink.tmp")
endFileOpenRead
func FileOpenRead(path: str): void*The example for this item is shown with the ones above: compress-Stream.salam
FileOpenWrite
func FileOpenWrite(path: str): void*The example for this item is shown with the ones above: compress-Stream.salam
FileClose
func FileClose(fp: void*)FileOpenRead or FileOpenWrite. A null file is ignored.The example for this item is shown with the ones above: compress-Stream.salam
FileSeek
func FileSeek(fp: void*, off: i64): booloff of the file with a 64 bit offset. True when it worked.The example for this item is shown with the ones above: compress-Stream.salam
FileSize
func FileSize(fp: void*): i64The example for this item is shown with the ones above: compress-Stream.salam
FileReadAt
func FileReadAt(fp: void*, off: i64, dst: void*, n: int): intn bytes starting at byte off of the file into dst and returns how many were read; 0 when the file cannot be positioned. The offset is 64 bit, so files over 2 GB work.The example for this item is shown with the ones above: compress-Stream.salam
CallbackOpenRead
func CallbackOpenRead(total: i64, read: func(i64, void*, int, void*): int, ctx: void*): void*read(off, dst, n, ctx) instead of a file; total is what FileSize reports. FileReadAt on it calls your function, which returns the number of bytes it put in dst (0 at the end). The decoders that read a file (LzmaFileToSink, bzip2, inflate and the rest) accept it, which is how a decrypting stage sits in front of a decompressor. Close it with FileClose; the ctx stays yours.The example for this item is shown with the ones above: compress-Stream.salam
CallbackContext
func CallbackContext(fp: void*): void*ctx given to CallbackOpenRead, or null for any other handle.The example for this item is shown with the ones above: compress-Stream.salam
MultiOpenRead
func MultiOpenRead(paths: Vector<str>): void*FileReadAt, FileSize and FileClose like a single file; a read that crosses the end of one piece continues in the next. Returns null when any piece cannot be opened.import archive
import compress
import mem
import str
func collect(p: void*, n: int, c: void*): bool:
compress.BufPushBytes((c as compress.ByteBuf*)[0], p, n)
ret true
end
func main:
mut text := compress.NewByteBuf()
repeat 2000 in i:
line := "row " + str.FromInt((i * 3) as i64) + "\n"
compress.BufPushBytes(text, line as void*, line.Len())
end
packed := compress.LzmaCompress(text.data, text.len)
archive.WriteBinaryFileBuf("lz.tmp", packed)
fp := compress.FileOpenRead("lz.tmp")
got := mem.AllocateZeroed(sizeof(compress.ByteBuf)) as compress.ByteBuf*
got[0] = compress.NewByteBuf()
mut sink := compress.SinkCallback(0 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
st := compress.LzmaFileToSink(packed.data, 5, fp, 13 as i64, (packed.len - 13) as i64, text.len as i64, sink, 0 as i64)
compress.FileClose(fp)
println st, sink.written, compress.BufEquals(got[0], text)
fp2 := compress.FileOpenRead("lz.tmp")
mut small := compress.SinkCallback(0 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
println compress.LzmaFileToSink(packed.data, 5, fp2, 13 as i64, (packed.len - 13) as i64, text.len as i64, small, 100 as i64)
compress.FileClose(fp2)
compress.BufFree(got[0])
mem.Free(got as void*)
endSinkOpen
func SinkOpen(path: str, cap: i64): FileSinkcap bytes (0 for no limit). failed is true when the file cannot be created.The example for this item is shown with the ones above: compress-Stream.salam
SinkCallback
func SinkCallback(cap: i64, emit: func(void*, int, void*): bool, ctx: void*): FileSinkemit(p, n, ctx) instead of a file. The piece is counted and checksummed like any other; cap still stops it with over. emit returns false to stop the decoder.The example for this item is shown with the ones above: compress-LzmaFile.salam
SinkWrite
func SinkWrite(s &: FileSink, p: void*, n: int): booln bytes to the sink and adds them to the count and the CRC-32. False once the sink has failed or when the bytes would pass the cap.The example for this item is shown with the ones above: compress-Stream.salam
SinkClose
func SinkClose(s &: FileSink): boolThe example for this item is shown with the ones above: compress-Stream.salam
SinkCopyFrom
func SinkCopyFrom(s &: FileSink, fp: void*, off: i64, len: i64): boollen bytes from byte off of an open file into the sink, in pieces. False when the file is too short or the sink refuses the data.The example for this item is shown with the ones above: compress-Stream.salam
GzipCompress
func GzipCompress(p: void*, n: int): ByteBuf.gz files.import compress
func main:
text := "hello gzip! hello gzip! hello gzip!"
g := compress.GzipCompressStr(text)
println compress.BufGet(g, 0), compress.BufGet(g, 1)
mut ok := false
b := compress.GzipDecompressBuf(g, ok)
println ok, compress.BufToStrUnsafe(b) == text
b2 := compress.BufFromStr(text)
g2 := compress.GzipCompressBuf(b2)
g3 := compress.GzipCompress(b2.data, b2.len)
println compress.BufEquals(g2, g3)
_ := compress.GzipDecompress(b2.data, b2.len, ok)
println ok
d := compress.DeflateCompress(b2.data, b2.len)
println compress.BufEquals(compress.DeflateDecompress(d.data, d.len), b2)
raw := compress.RawDeflate(b2.data, b2.len)
back := compress.RawInflate(raw.data, raw.len, b2.len, ok)
println ok, compress.BufEquals(back, b2)
_ := compress.RawInflateMax(raw.data, raw.len, b2.len, 10, ok)
println ok
full := compress.RawInflateMax(raw.data, raw.len, b2.len, 1000, ok)
println ok, compress.BufEquals(full, b2)
_ := compress.GzipDecompressMax(g.data, g.len, 10, ok)
println ok
whole := compress.GzipDecompressMax(g.data, g.len, 1000, ok)
println ok, compress.BufToStrUnsafe(whole) == text
endGzipDecompress
func GzipDecompress(p: void*, n: int, ok &: bool): ByteBufok becomes false when the data is not gzip. The checksum inside the file is not checked.The example for this item is shown with the ones above: compress-Gzip.salam
GzipDecompressMax
func GzipDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBufGzipDecompress, but stops with ok false when the unpacked data would pass max bytes (0 means no limit); use it on data from outside so a small file cannot fill memory.The example for this item is shown with the ones above: compress-Gzip.salam
GzipDecompressWhy
func GzipDecompressWhy(p: void*, n: int, max: int, why &: int): ByteBufGzipDecompressMax, but says why it failed: why is set to DecodeOk, DecodeCorrupt (bad or cut data, wrong checksum) or DecodeTooBig (the unpacked data would pass max). On failure the result is empty.The example for this item is shown with the ones above: compress-Why.salam
GzipCompressBuf
func GzipCompressBuf(b: ByteBuf): ByteBufThe example for this item is shown with the ones above: compress-Gzip.salam
GzipCompressStr
func GzipCompressStr(s: str): ByteBufThe example for this item is shown with the ones above: compress-Gzip.salam
GzipDecompressBuf
func GzipDecompressBuf(b: ByteBuf, ok &: bool): ByteBufThe example for this item is shown with the ones above: compress-Gzip.salam
LZWCompress
func LZWCompress(p: void*, n: int): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
LZWDecompressMax
func LZWDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBufLZWDecompress but with a cap on the output size (0 means 1 GiB) and an ok flag. Damaged, truncated or oversized input gives ok false and an empty buffer; it never reads or writes out of bounds.The example for this item is shown with the ones above: compress-Methods.salam
LZWDecompress
func LZWDecompress(p: void*, n: int): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
NewByteBuf
func NewByteBuf(): ByteBufThe example for this item is shown with the ones above: compress-Buffer.salam
NewByteBufCap
func NewByteBufCap(capacity: int): ByteBufThe example for this item is shown with the ones above: compress-Buffer.salam
BufPush
func BufPush(b &: ByteBuf, v: int)The example for this item is shown with the ones above: compress-Buffer.salam
BufPushBytes
func BufPushBytes(b &: ByteBuf, p: void*, n: int)n bytes from memory.The example for this item is shown with the ones above: compress-Buffer.salam
BufPushBuf
func BufPushBuf(b &: ByteBuf, src: ByteBuf)The example for this item is shown with the ones above: compress-Buffer.salam
BufPushU32
func BufPushU32(b &: ByteBuf, v: int)The example for this item is shown with the ones above: compress-Buffer.salam
BufGet
func BufGet(b: ByteBuf, i: int): inti.The example for this item is shown with the ones above: compress-Buffer.salam
BufSet
func BufSet(b &: ByteBuf, i: int, v: int)i.The example for this item is shown with the ones above: compress-Buffer.salam
BufLen
func BufLen(b: ByteBuf): intThe example for this item is shown with the ones above: compress-Buffer.salam
BufFree
func BufFree(b &: ByteBuf)The example for this item is shown with the ones above: compress-Buffer.salam
BufReset
func BufReset(b &: ByteBuf)The example for this item is shown with the ones above: compress-Buffer.salam
BufClone
func BufClone(b: ByteBuf): ByteBufThe example for this item is shown with the ones above: compress-Buffer.salam
BufEquals
func BufEquals(a: ByteBuf, b: ByteBuf): boolThe example for this item is shown with the ones above: compress-Buffer.salam
BufFromStr
func BufFromStr(s: str): ByteBufThe example for this item is shown with the ones above: compress-Buffer.salam
BufToStrUnsafe
func BufToStrUnsafe(b: ByteBuf): strThe example for this item is shown with the ones above: compress-Buffer.salam
BufToBase64
func BufToBase64(b: ByteBuf): strThe example for this item is shown with the ones above: compress-Buffer.salam
NewByteReader
func NewByteReader(p: void*, n: int): ByteReadern bytes of memory.The example for this item is shown with the ones above: compress-Buffer.salam
ReaderFromBuf
func ReaderFromBuf(b: ByteBuf): ByteReaderThe example for this item is shown with the ones above: compress-Buffer.salam
RdLeft
func RdLeft(r: ByteReader): intThe example for this item is shown with the ones above: compress-Buffer.salam
RdByte
func RdByte(r &: ByteReader): intbad becomes true.The example for this item is shown with the ones above: compress-Buffer.salam
RdU32
func RdU32(r &: ByteReader): intThe example for this item is shown with the ones above: compress-Buffer.salam
RdBytes
func RdBytes(r &: ByteReader, n: int): ByteBufn bytes into a new buffer.The example for this item is shown with the ones above: compress-Buffer.salam
LzmaCompressLevel
func LzmaCompressLevel(p: void*, n: int, level: int): ByteBufLzmaCompress with a level from 0 to 9, as for XzCompressLevel.The example for this item is shown with the ones above: compress-XzWrite.salam
LzmaCompress
func LzmaCompress(p: void*, n: int): ByteBuf.lzma format at level 6: a 13 byte header with the properties, the dictionary size and the exact unpacked size, then the LZMA stream. The format has no checksum.The example for this item is shown with the ones above: compress-XzWrite.salam
LzmaCompressBuf
func LzmaCompressBuf(b: ByteBuf): ByteBuf.lzma format.The example for this item is shown with the ones above: compress-XzWrite.salam
LzmaCompressStr
func LzmaCompressStr(s: str): ByteBuf.lzma format.The example for this item is shown with the ones above: compress-XzWrite.salam
XzCompressCheck
func XzCompressCheck(p: void*, n: int, level: int, check: int): ByteBufXzCompressLevel, and check is the integrity check stored with the block: XzCheckNone, XzCheckCrc32, XzCheckCrc64 or XzCheckSha256. Any other number gives CRC-64.The example for this item is shown with the ones above: compress-XzWrite.salam
XzCompressLevel
func XzCompressLevel(p: void*, n: int, level: int): ByteBufXzCompress with a level from 0 (fastest) to 9 (smallest); a number outside that range is moved to the nearest end. The level sets the window (256 KiB at 0, 8 MiB at 6, 64 MiB at 9, never bigger than the data) and how hard the match finder searches.The example for this item is shown with the ones above: compress-XzWrite.salam
XzCompress
func XzCompress(p: void*, n: int): ByteBuf.xz file at level 6 with a CRC-64 check, the way xz does by default. Empty input gives a valid stream with no block.The example for this item is shown with the ones above: compress-XzWrite.salam
XzCompressBuf
func XzCompressBuf(b: ByteBuf): ByteBufThe example for this item is shown with the ones above: compress-XzWrite.salam
XzCompressStr
func XzCompressStr(s: str): ByteBufThe example for this item is shown with the ones above: compress-XzWrite.salam
LzmaCompressFileToSink
func LzmaCompressFileToSink(fp: void*, off: i64, len: i64, level: int, with_props: bool, sink &: FileSink): boollen bytes of the open file fp from off into one bare LZMA stream (no size field, no end marker) and writes it to a sink as it goes. With with_props the stream starts with the 5 property bytes that LzmaFileToSink wants. Only a window of about twice the dictionary is held in memory. The output can differ from LzmaCompressLevel by a fraction of a percent because the matcher sees the data in pieces.The example for this item is shown with the ones above: compress-CompressSink.salam
RLECompress
func RLECompress(p: void*, n: int): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
RLEDecompressMax
func RLEDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBufRLEDecompress but with a cap on the output size (0 means 1 GiB) and an ok flag. Damaged, truncated or oversized input gives ok false and an empty buffer; it never reads or writes out of bounds.The example for this item is shown with the ones above: compress-Methods.salam
RLEDecompress
func RLEDecompress(p: void*, n: int): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
RawInflateToSink
func RawInflateToSink(r &: BitReader, sink &: FileSink, d64: bool): boold64 is true). False when the data is broken or the sink refuses it because of its cap or a failed write.The example for this item is shown with the ones above: compress-Stream.salam
RawInflate
func RawInflate(p: void*, n: int, expected_size: int, ok &: bool): ByteBufexpected_size is the rough size of the output.The example for this item is shown with the ones above: compress-Gzip.salam
RawInflateMax
func RawInflateMax(p: void*, n: int, expected_size: int, max: int, ok &: bool): ByteBufRawInflate, but stops with ok false once the output would pass max bytes, so a small hostile stream cannot expand without bound. max of 0 means no limit.The example for this item is shown with the ones above: compress-Gzip.salam
RawInflate64Max
func RawInflate64Max(p: void*, n: int, expected_size: int, max: int, ok &: bool): ByteBufRawInflateMax for Deflate64, the DEFLATE variant of 7-Zip and Zip with a 64 KiB window, 16 bit long matches and two more distance codes.The example for this item is shown with the ones above: compress-Gzip.salam
ZstdStatusText
func ZstdStatusText(code: int): strThe example for this item is shown with the ones above: compress-Zstd.salam
IsZstd
func IsZstd(p: void*, n: int): bool28 B5 2F FD) or the magic number of a skippable frame. It reads four bytes and checks nothing else.The example for this item is shown with the ones above: compress-Zstd.salam
IsZstdBuf
func IsZstdBuf(b: ByteBuf): boolIsZstd on a byte buffer.The example for this item is shown with the ones above: compress-Zstd.salam
ZstdDecompressEx
func ZstdDecompressEx(p: void*, n: int, max: int, dict: void*, dn: int, status &: int): ByteBufmax is the largest output accepted (0 means just under 1 GiB). dict and dn give an optional dictionary (null and 0 for none), either a trained one or plain content. status receives one of the ZstdOk and ZstdErr... codes, and the result is an empty buffer on any failure.The example for this item is shown with the ones above: compress-Zstd.salam
ZstdDecompressMax
func ZstdDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBufZstdDecompress, but stops with ok false when the unpacked data would pass max bytes (0 means no limit). Use it on data from outside: a few bytes of Zstandard can describe gigabytes.The example for this item is shown with the ones above: compress-Zstd.salam
ZstdDecompress
func ZstdDecompress(p: void*, n: int, ok &: bool): ByteBuf.zst file: one frame or several in a row, with skippable frames passed over. It handles every block type, including Huffman and FSE compressed ones, repeated offsets and the XXH64 checksum when the frame carries one. ok is false for damaged, truncated or unsupported data, and the result is then empty. A frame that needs a dictionary fails here; use ZstdDecompressDict.The example for this item is shown with the ones above: compress-Zstd.salam
ZstdDecompressBuf
func ZstdDecompressBuf(b: ByteBuf, ok &: bool): ByteBufThe example for this item is shown with the ones above: compress-Zstd.salam
ZstdDecompressDict
func ZstdDecompressDict(p: void*, n: int, dict: void*, dn: int, ok &: bool): ByteBufzstd --train writes, or any bytes, which then serve as raw content. A frame that names a dictionary ID needs the dictionary with that ID.import compress
const FRAME := """
28b52ffd645900ed010092430c11a0ed7425bfff01a825dda80e11a94a1c014020bd51a7cf6fcc41d92e1f9a01043a9c
4edf68a4ea1867b8bed41b6e2a240200f7874039c1d515ee20b3a1
"""
func nib(c: int): int:
if c >= 48 and c <= 57:
ret c - 48
end
if c >= 97 and c <= 102:
ret c - 87
end
ret -1
end
func unhex(s: str): compress.ByteBuf:
mut b := compress.NewByteBuf()
bp := s as u8*
mut hi := -1
repeat s.Len() in i:
v := nib(bp[i] as int)
if v < 0:
continue
end
if hi < 0:
hi = v
else:
compress.BufPush(b, hi * 16 + v)
hi = -1
end
end
ret b
end
func main:
frame := unhex(FRAME)
dict := compress.BufFromStr("Salam is a programming language. ")
mut status := 0
without := compress.ZstdDecompressEx(frame.data, frame.len, 0, null, 0, status)
println "without the dictionary:", status, compress.ZstdStatusText(status), without.len
mut ok := false
text := compress.ZstdDecompressDict(frame.data, frame.len, dict.data, dict.len, ok)
println "with the dictionary:", ok, text.len, "bytes from", frame.len
println compress.BufToStrUnsafe(text)
other := compress.BufFromStr("some other bytes")
wrong := compress.ZstdDecompressEx(frame.data, frame.len, 0, other.data, other.len, status)
println "with a different dictionary:", status, wrong.len
endZstdCompressOpts
func ZstdCompressOpts(p: void*, n: int, level: int, checksum: bool): ByteBuflevel, with or without the XXH64 content checksum. The frame records the unpacked size.The example for this item is shown with the ones above: compress-Zstd.salam
ZstdCompressLevel
func ZstdCompressLevel(p: void*, n: int, level: int): ByteBufzstd tool and every other reader accepts. The writer is simple (a greedy or lazy search and the predefined entropy tables), so its output is larger than the reference tool's at the same level, but it is valid everywhere.The example for this item is shown with the ones above: compress-Zstd.salam
ZstdCompress
func ZstdCompress(p: void*, n: int): ByteBufThe example for this item is shown with the ones above: compress-Zstd.salam
ZstdCompressBuf
func ZstdCompressBuf(b: ByteBuf): ByteBufThe example for this item is shown with the ones above: compress-Zstd.salam
ZstdCompressStr
func ZstdCompressStr(s: str): ByteBufThe example for this item is shown with the ones above: compress-Zstd.salam
ZstdContentSize
func ZstdContentSize(p: void*, n: int): i64The example for this item is shown with the ones above: compress-Zstd.salam
NewBitWriter
func NewBitWriter(): BitWriterThe example for this item is shown with the ones above: compress-Bits.salam
BitWriteBit
func BitWriteBit(w &: BitWriter, bit: int)The example for this item is shown with the ones above: compress-Bits.salam
BitWriteBits
func BitWriteBits(w &: BitWriter, value: int, count: int)count bits of value.The example for this item is shown with the ones above: compress-Bits.salam
BitFlush
func BitFlush(w &: BitWriter)The example for this item is shown with the ones above: compress-Bits.salam
BitWriterLen
func BitWriterLen(w &: BitWriter): intThe example for this item is shown with the ones above: compress-Bits.salam
BitWriterFree
func BitWriterFree(w &: BitWriter)The example for this item is shown with the ones above: compress-Bits.salam
NewBitReader
func NewBitReader(p: void*, n: int): BitReaderThe example for this item is shown with the ones above: compress-Bits.salam
NewFileBitReader
func NewFileBitReader(fp: void*, off: i64, len: i64): BitReaderlen bytes of a file, starting at byte off, that reads the file in pieces of 64 KiB. bad is true when the file cannot be positioned. Free it with FileBitReaderFree.The example for this item is shown with the ones above: compress-Stream.salam
FileBitReaderFree
func FileBitReaderFree(r &: BitReader)NewFileBitReader. It does not close the file.The example for this item is shown with the ones above: compress-Stream.salam
BitReaderFromBuf
func BitReaderFromBuf(b: ByteBuf): BitReaderimport compress
func main:
mut b := compress.NewByteBuf()
compress.BufPushBytes(b, "abc" as void*, 3)
println b.len, b.cap >= b.len, b.data != null
mut r := compress.ReaderFromBuf(b)
println r.len, r.pos, r.bad, r.data == b.data
_ := compress.RdByte(r)
println r.pos
_ := compress.RdBytes(r, 5)
println r.bad
mut w := compress.NewBitWriter()
compress.BitWriteBits(w, 5, 3)
println w.nbits, w.acc, w.buf.len
compress.BitFlush(w)
println w.nbits, w.buf.len
mut br := compress.BitReaderFromBuf(w.buf)
println br.len, br.pos, br.nbits, br.acc, br.bad, br.data == w.buf.data
println compress.BitReadBits(br, 3), br.nbits, br.pos
_ := compress.BitReadBits(br, 16)
println br.bad
compress.BitWriterFree(w)
compress.BufFree(b)
methods := [compress.MethodNone, compress.MethodRLE, compress.MethodHuffman, compress.MethodLZ77, compress.MethodLZW, compress.MethodDeflate, compress.MethodBWT]
each m in methods:
out := compress.CompressStr(m, "aaaaabbbbbaaaaabbbbb")
println m, compress.BufToStrUnsafe(compress.Decompress(m, out.data, out.len))
end
endBitReadBit
func BitReadBit(r &: BitReader): intThe example for this item is shown with the ones above: compress-Bits.salam
BitReadBits
func BitReadBits(r &: BitReader, count: int): intcount bits.The example for this item is shown with the ones above: compress-Bits.salam
ZstdFileToSink
func ZstdFileToSink(fp: void*, off: i64, len: i64, sink &: FileSink, memcap: i64): intlen bytes read from fp at off into a sink block by block, with a sliding window: memory is the frame's window size (from its header) plus a few hundred kilobytes, however big the output is. A frame whose window is over memcap (0 means 1 GiB) is refused with ZstdErrTooBig, and so is one whose declared size or output passes the sink's own cap. The content checksum is verified, several frames and skippable frames are accepted, and a frame that needs a dictionary gives ZstdErrDict. When the sink itself fails, sink.failed is set.import archive
import compress
import mem
import os
import str
func collect(p: void*, n: int, c: void*): bool:
compress.BufPushBytes((c as compress.ByteBuf*)[0], p, n)
ret true
end
func main:
mut text := compress.NewByteBuf()
repeat 5000 in i:
line := "line " + str.FromInt((i * 7) as i64) + "\n"
compress.BufPushBytes(text, line as void*, line.Len())
end
mut packed := compress.ZstdCompress(text.data, text.len)
archive.WriteBinaryFileBuf("zst.tmp", packed)
fp := compress.FileOpenRead("zst.tmp")
got := mem.AllocateZeroed(sizeof(compress.ByteBuf)) as compress.ByteBuf*
got[0] = compress.NewByteBuf()
mut sink := compress.SinkCallback(0 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
st := compress.ZstdFileToSink(fp, 0 as i64, compress.FileSize(fp), sink, 0 as i64)
println st == compress.ZstdOk, sink.written == (text.len as i64), compress.BufEquals(got[0], text)
compress.FileClose(fp)
fp2 := compress.FileOpenRead("zst.tmp")
mut small := compress.SinkCallback(100 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
println compress.ZstdFileToSink(fp2, 0 as i64, compress.FileSize(fp2), small, 0 as i64) == compress.ZstdErrTooBig
compress.FileClose(fp2)
compress.BufFree(got[0])
mem.Free(got as void*)
compress.BufFree(text)
compress.BufFree(packed)
os.Remove("zst.tmp")
endBcjStep
func BcjStep(kind: int, p: void*, n: int, pos: i64, state &: int): intBcjX86, BcjPowerPC, BcjIA64, BcjARM, BcjARMThumb, BcjSPARC, BcjARM64 or BcjRISCV) over n bytes in place. pos is the position of the first byte in the whole stream and state carries the x86 history between calls. It returns how many bytes are final; hold the rest back and give them again at the front of the next piece. A step can be run on any piece size and gives the same bytes as one call over the whole data. -1 for an unknown kind.The example for this item is shown with the ones above: compress-Stream.salam
BcjDecode
func BcjDecode(kind: int, p: void*, n: int, start: int): booln bytes. kind is one of the Bcj... numbers, and start the address the first byte had (0 for xz and 7z). Returns false for an unknown kind.The example for this item is shown with the ones above: compress-XzRaw.salam
DeltaDecode
func DeltaDecode(p: void*, n: int, dist: int): booldist is the distance, 1 to 256. Returns false for any other distance.The example for this item is shown with the ones above: compress-XzRaw.salam
SwapDecode
func SwapDecode(p: void*, n: int, width: int): boolwidth 2 or 4 reverses every group of that many bytes; a trailing partial group stays as it is. Returns false for any other width.The example for this item is shown with the ones above: compress-XzRaw.salam
Compress
func Compress(method: int, p: void*, n: int): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
DecompressMax
func DecompressMax(method: int, p: void*, n: int, max: int, ok &: bool): ByteBufDecompress with a cap on the output size (0 means 1 GiB) and an ok flag; bad input gives ok false and an empty buffer.The example for this item is shown with the ones above: compress-Methods.salam
Decompress
func Decompress(method: int, p: void*, n: int): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
CompressStr
func CompressStr(method: int, s: str): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
CompressBuf
func CompressBuf(method: int, b: ByteBuf): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
DecompressBuf
func DecompressBuf(method: int, b: ByteBuf): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
CompressBest
func CompressBest(p: void*, n: int, method_out &: int): ByteBufmethod_out.The example for this item is shown with the ones above: compress-Methods.salam
Ratio
func Ratio(compressed_len: int, original_len: int): f64The example for this item is shown with the ones above: compress-Methods.salam
CRC32Update
func CRC32Update(state: u32, p: void*, n: int): u32CRC32 of all the parts joined, so a big file can be checksummed piece by piece.The example for this item is shown with the ones above: compress-Stream.salam
CRC32
func CRC32(p: void*, n: int): u32import compress
func main:
println compress.CRC32Str("123456789"), compress.Adler32Str("Wikipedia")
b := compress.BufFromStr("123456789")
println compress.CRC32(b.data, b.len) == compress.CRC32Buf(b)
println compress.Adler32(b.data, b.len) == compress.Adler32Buf(b)
println compress.CRC32Str("hello")
endCRC32Str
func CRC32Str(s: str): u32The example for this item is shown with the ones above: compress-Checksum.salam
CRC32Buf
func CRC32Buf(b: ByteBuf): u32The example for this item is shown with the ones above: compress-Checksum.salam
Adler32
func Adler32(p: void*, n: int): u32The example for this item is shown with the ones above: compress-Checksum.salam
Adler32Str
func Adler32Str(s: str): u32The example for this item is shown with the ones above: compress-Checksum.salam
Adler32Buf
func Adler32Buf(b: ByteBuf): u32The example for this item is shown with the ones above: compress-Checksum.salam
XXH64
func XXH64(p: void*, n: int, seed: u64): u64n bytes with a seed. A Zstandard frame uses the low 32 bits of the seed 0 value as its content checksum.import compress
import conv
func main:
println conv.FormatHex(compress.XXH64Str(""))
println conv.FormatHex(compress.XXH64Str("abc"))
text := compress.BufFromStr("Salam")
println conv.FormatHex(compress.XXH64Buf(text))
println conv.FormatHex(compress.XXH64(text.data, text.len, 0 as u64))
println conv.FormatHex(compress.XXH64(text.data, text.len, 1 as u64))
endXXH64Str
func XXH64Str(s: str): u64The example for this item is shown with the ones above: compress-XXH64.salam
XXH64Buf
func XXH64Buf(b: ByteBuf): u64The example for this item is shown with the ones above: compress-XXH64.salam
DeflateCompress
func DeflateCompress(p: void*, n: int): ByteBufRawDeflate or gzip.The example for this item is shown with the ones above: compress-Gzip.salam
DeflateDecompressMax
func DeflateDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBufDeflateDecompress but with a cap on the output size (0 means 1 GiB) and an ok flag. Damaged, truncated or oversized input gives ok false and an empty buffer; it never reads or writes out of bounds.The example for this item is shown with the ones above: compress-Methods.salam
DeflateDecompress
func DeflateDecompress(p: void*, n: int): ByteBufDeflateCompress.The example for this item is shown with the ones above: compress-Gzip.salam
RawDeflateTo
func RawDeflateTo(w &: BitWriter, p: void*, n: int, last: bool)RawDeflate, but writes into a BitWriter you keep, so a long input can be given in pieces: every call but the last passes last false, the last passes true, and call BitFlush at the end. Each piece is compressed on its own, without matches into earlier pieces.The example for this item is shown with the ones above: compress-Gzip.salam
RawDeflate
func RawDeflate(p: void*, n: int): ByteBufThe example for this item is shown with the ones above: compress-Gzip.salam
HuffmanCompress
func HuffmanCompress(p: void*, n: int): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
HuffmanDecompressMax
func HuffmanDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBufHuffmanDecompress but with a cap on the output size (0 means 1 GiB) and an ok flag. Damaged, truncated or oversized input gives ok false and an empty buffer; it never reads or writes out of bounds.The example for this item is shown with the ones above: compress-Methods.salam
HuffmanDecompress
func HuffmanDecompress(p: void*, n: int): ByteBufThe example for this item is shown with the ones above: compress-Methods.salam
XzDecompress
func XzDecompress(p: void*, n: int, ok &: bool): ByteBuf.xz file: the stream header and footer, every block with its filters (LZMA2, the branch filters for x86, ARM, ARM64, PowerPC, SPARC and IA-64, and delta), the index and the check (CRC-32, CRC-64 or SHA-256, whichever the file declares). Streams written one after the other are joined. ok is false for damaged, truncated or unsupported input, and the result is then empty. The result is capped at just under 1 GiB; use XzDecompressMax on data from outside to set a smaller cap.import compress
const XZ := """
fd377a585a000004e6d6b44604c05fcf022101160000000000000000bb0ccf6ce0014e00575d0029984986240afb8a34e4
81db780958f3d0144436dd3bb4ca45234051fa042e5be576402f66b72801fb0f3b421e32e3eba071037cdd425338444ede
4cb8c8c742828bc92c1b53df8ea763015d902c3d49f9c602aacf40a000005b35fff6fd65c4e300017bcf020000005e9efc
63b1c467fb020000000004595a
"""
const LZMA := """
5d00008000ffffffffffffffff0029984986240afb8a34e481db780958f3d0144436dd3bb4ca45234051fa042e5be57640
2f66b72801fb0f3b421e32e3eba071037cdd425338444ede4cb8c8c742828bc92c1b53df8ea763015d902c3d49f9c602ad
8d68e0bcfed44dc0
"""
func nib(c: int): int:
if c >= 48 and c <= 57:
ret c - 48
end
if c >= 97 and c <= 102:
ret c - 87
end
ret -1
end
func unhex(s: str): compress.ByteBuf:
mut b := compress.NewByteBuf()
bp := s as u8*
mut hi := -1
repeat s.Len() in i:
v := nib(bp[i] as int)
if v < 0:
continue
end
if hi < 0:
hi = v
else:
compress.BufPush(b, hi * 16 + v)
hi = -1
end
end
ret b
end
func main:
xz := unhex(XZ)
lz := unhex(LZMA)
mut ok := false
a := compress.XzDecompress(xz.data, xz.len, ok)
println ".xz:", ok, a.len
b := compress.XzDecompressBuf(xz, ok)
println ".xz buffer:", ok, compress.BufEquals(a, b)
c := compress.XzDecompressMax(xz.data, xz.len, a.len, ok)
println ".xz with max:", ok, c.len
d := compress.XzDecompressMax(xz.data, xz.len, a.len - 1, ok)
println ".xz one byte over max:", ok, d.len
e := compress.LzmaDecompress(lz.data, lz.len, ok)
println ".lzma:", ok, compress.BufEquals(a, e)
f := compress.LzmaDecompressBuf(lz, ok)
println ".lzma buffer:", ok, f.len
g := compress.LzmaDecompressMax(lz.data, lz.len, 50, ok)
println ".lzma with max 50:", ok, g.len
endXzDecompressMax
func XzDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBufXzDecompress, but stops with ok false when the unpacked data would pass max bytes (0 means no limit), so a tiny file cannot expand without bound.The example for this item is shown with the ones above: compress-Xz.salam
XzDecompressWhy
func XzDecompressWhy(p: void*, n: int, max: int, why &: int): ByteBufXzDecompressMax, but sets why to DecodeOk, DecodeCorrupt or DecodeTooBig.The example for this item is shown with the ones above: compress-Why.salam
XzDecompressBuf
func XzDecompressBuf(b: ByteBuf, ok &: bool): ByteBufThe example for this item is shown with the ones above: compress-Xz.salam
Bzip2CompressLevel
func Bzip2CompressLevel(p: void*, n: int, level: int): ByteBufBzip2Compress with a level from 1 to 9, the block size in units of 100000 bytes; a number outside that range is moved to the nearest end. A lower level uses less memory.import compress
func main:
text := "hello bzip2! hello bzip2! hello bzip2! hello bzip2!"
z := compress.Bzip2CompressStr(text)
println compress.BufGet(z, 0), compress.BufGet(z, 1), compress.BufGet(z, 2), compress.BufGet(z, 3)
mut ok := false
b := compress.Bzip2DecompressBuf(z, ok)
println ok, compress.BufToStrUnsafe(b) == text
src := compress.BufFromStr(text)
z2 := compress.Bzip2CompressBuf(src)
z3 := compress.Bzip2Compress(src.data, src.len)
println compress.BufEquals(z2, z3)
z4 := compress.Bzip2CompressLevel(src.data, src.len, 1)
println compress.BufGet(z4, 3)
back := compress.Bzip2Decompress(z4.data, z4.len, ok)
println ok, compress.BufEquals(back, src)
_ := compress.Bzip2Decompress(src.data, src.len, ok)
println ok
_ := compress.Bzip2DecompressMax(z.data, z.len, 10, ok)
println ok
whole := compress.Bzip2DecompressMax(z.data, z.len, 1000, ok)
println ok, compress.BufToStrUnsafe(whole) == text
endBzip2Compress
func Bzip2Compress(p: void*, n: int): ByteBuf.bz2) with the biggest block size, level 9.The example for this item is shown with the ones above: compress-Bzip2.salam
Bzip2CompressBuf
func Bzip2CompressBuf(b: ByteBuf): ByteBufThe example for this item is shown with the ones above: compress-Bzip2.salam
Bzip2CompressStr
func Bzip2CompressStr(s: str): ByteBufThe example for this item is shown with the ones above: compress-Bzip2.salam
Bzip2Decompress
func Bzip2Decompress(p: void*, n: int, ok &: bool): ByteBufok becomes false when the data is not valid bzip2, when a block or stream CRC does not match or when anything follows the last stream. A failure returns an empty buffer.The example for this item is shown with the ones above: compress-Bzip2.salam
Bzip2DecompressMax
func Bzip2DecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBufBzip2Decompress, but stops with ok false when the unpacked data would pass max bytes (0 means no limit); use it on data from outside so a small file cannot fill memory.The example for this item is shown with the ones above: compress-Bzip2.salam
Bzip2DecompressWhy
func Bzip2DecompressWhy(p: void*, n: int, max: int, why &: int): ByteBufBzip2DecompressMax, but sets why to DecodeOk, DecodeCorrupt or DecodeTooBig, so a stream over the limit is told apart from a damaged one.The example for this item is shown with the ones above: compress-Why.salam
Bzip2DecompressBuf
func Bzip2DecompressBuf(b: ByteBuf, ok &: bool): ByteBufThe example for this item is shown with the ones above: compress-Bzip2.salam
Bzip2FileToSink
func Bzip2FileToSink(fp: void*, off: i64, len: i64, sink &: FileSink): intlen bytes read from fp at off into a sink, one block at a time, so memory stays at the block (at most 900 KB of table data plus 1 MiB of output) whatever the size of the file. Several streams in a row are accepted. Every block's CRC and the stream CRC are checked. The result is DecodeOk, DecodeCorrupt, DecodeTooBig (the sink's own cap was passed) or DecodeIo (the sink could not take the data).The example for this item is shown with the ones above: compress-Bzip2File.salam
Bzip2CompressFileToSink
func Bzip2CompressFileToSink(fp: void*, off: i64, len: i64, level: int, sink &: FileSink): boollen bytes of the open file fp from off into one bzip2 stream and writes it to a sink block by block, so memory is one block (about 18 MB of working tables at level 9) plus 256 KiB of input, whatever the size of the file. level is 1 to 9. The bytes are the same as Bzip2CompressLevel would give. Returns false when the file cannot be read or the sink refuses.The example for this item is shown with the ones above: compress-CompressSink.salam
XzFileToSink
func XzFileToSink(fp: void*, off: i64, len: i64, sink &: FileSink, memcap: i64): intlen bytes read from fp at off into a sink without holding it in memory. The index at the end of each stream is read first, so the unpacked size is known (and checked against the sink's cap) before anything is written; then every block is unpacked through a window the size of its dictionary, and its CRC-32, CRC-64 or SHA-256 is checked. Several streams in a row and stream padding are accepted. A file that uses a Delta or BCJ filter is read whole and decoded in memory instead (up to 2 GB). memcap limits the window (0 for none). The result is DecodeOk, DecodeCorrupt, DecodeTooBig or DecodeIo.import archive
import compress
import mem
import os
import str
func collect(p: void*, n: int, c: void*): bool:
compress.BufPushBytes((c as compress.ByteBuf*)[0], p, n)
ret true
end
func main:
mut text := compress.NewByteBuf()
repeat 5000 in i:
line := "line " + str.FromInt((i * 7) as i64) + "\n"
compress.BufPushBytes(text, line as void*, line.Len())
end
mut packed := compress.XzCompress(text.data, text.len)
archive.WriteBinaryFileBuf("xz.tmp", packed)
fp := compress.FileOpenRead("xz.tmp")
got := mem.AllocateZeroed(sizeof(compress.ByteBuf)) as compress.ByteBuf*
got[0] = compress.NewByteBuf()
mut sink := compress.SinkCallback(0 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
st := compress.XzFileToSink(fp, 0 as i64, compress.FileSize(fp), sink, 0 as i64)
println st == compress.DecodeOk, sink.written == (text.len as i64), compress.BufEquals(got[0], text)
compress.FileClose(fp)
fp2 := compress.FileOpenRead("xz.tmp")
mut small := compress.SinkCallback(100 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
println compress.XzFileToSink(fp2, 0 as i64, compress.FileSize(fp2), small, 0 as i64) == compress.DecodeTooBig
compress.FileClose(fp2)
compress.BufFree(got[0])
mem.Free(got as void*)
compress.BufFree(text)
compress.BufFree(packed)
os.Remove("xz.tmp")
endLzma2RawDecompress
func Lzma2RawDecompress(dict_prop: int, p: void*, n: int, out_size: int, max: int, ok &: bool): ByteBufdict_prop is the one byte dictionary property (0 to 40). out_size is the exact unpacked size, or -1 when it is not known; max of 0 means no limit.The example for this item is shown with the ones above: compress-XzRaw.salam
LzmaRawDecompress
func LzmaRawDecompress(props: void*, plen: int, p: void*, n: int, out_size: int, max: int, ok &: bool): ByteBufprops is the 5 byte coder property (one byte for lc, lp and pb, then the dictionary size, little-endian), and out_size the exact unpacked size, which the stream must produce. max of 0 means no limit.The example for this item is shown with the ones above: compress-XzRaw.salam
LzmaDecompress
func LzmaDecompress(p: void*, n: int, ok &: bool): ByteBuf.lzma format (what xz --format=lzma writes), whose 13 byte header holds the properties, the dictionary size and the unpacked size, or "unknown" followed by an end marker. The format has no checksum, so only the structure is checked. Dictionary sizes above LzmaMaxDictSize are refused.The example for this item is shown with the ones above: compress-Xz.salam
LzmaDecompressMax
func LzmaDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBufLzmaDecompress, but stops with ok false when the unpacked data would pass max bytes (0 means no limit).The example for this item is shown with the ones above: compress-Xz.salam
LzmaDecompressWhy
func LzmaDecompressWhy(p: void*, n: int, max: int, why &: int): ByteBufLzmaDecompressMax, but sets why to DecodeOk, DecodeCorrupt or DecodeTooBig.The example for this item is shown with the ones above: compress-Why.salam
LzmaDecompressBuf
func LzmaDecompressBuf(b: ByteBuf, ok &: bool): ByteBufThe example for this item is shown with the ones above: compress-Xz.salam
LzmaFileToSink
func LzmaFileToSink(props: void*, plen: int, fp: void*, off: i64, len: i64, out_size: i64, sink &: FileSink, memcap: i64): intprops, then the data from off for len bytes of the open file fp) straight into a sink. Only a sliding window the size of the dictionary is kept, never the whole result, so the unpacked size can pass 2 GiB. out_size is the exact unpacked size. memcap is the most memory the window may take (0 for no limit). Returns 0 on success, 1 for bad data, 2 when the window would be over memcap or over 2 GiB, and 3 when the sink refused data.The example for this item is shown with the ones above: compress-LzmaFile.salam
Lzma2FileToSink
func Lzma2FileToSink(dict_prop: int, fp: void*, off: i64, len: i64, out_size: i64, sink &: FileSink, memcap: i64): intLzmaFileToSink for a bare LZMA2 stream; dict_prop is the one byte dictionary property (0 to 40). The window is the dictionary size, or the unpacked size when that is smaller.The example for this item is shown with the ones above: compress-LzmaFile.salam