Standard library · Encoding and data

compress

Byte buffers, compression methods from RLE through DEFLATE, gzip, bzip2, zstd and checksums.

import compress

This package is the base of all binary data work in Salam. ByteBuf, a growable byte buffer, and ByteReader, a cursor over bytes, are what packages such as encoding.binary, archive and encoding.msgpack pass around. On top of them it has gzip, bzip2, raw DEFLATE, a few teaching compressors (RLE, LZ77, LZW, Huffman, BWT), bit-level reading and writing, and the CRC-32 and Adler-32 checksums.

To pack several files into one .zip or .tar, use archive, which calls this package for the compression. To write numbers in a fixed byte order, use encoding.binary on the same buffers.

For compression, reach for gzip when the data has to be read by other programs. Use bzip2 when the result should be smaller than gzip and a slower pack is fine; it reads and writes standard .bz2 data. The other methods are mostly there to learn from and experiment with, and their output format belongs to this package alone. The examples show that for small data the overhead of Huffman and DEFLATE comes to more than the data itself.

What trips people up

  • Gzip verifies its checksum. GzipDecompress checks the header, inflates the data, then compares both the CRC-32 and the uncompressed length in the trailer; a mismatch sets ok false and returns an empty buffer.
  • Xz and LZMA writing is a fast match finder, not a tuned one. The packed size is within a few percent of xz at the same level, never smaller. The whole input and the whole result are held in memory, and level 9 asks for a 64 MiB window. The result of a decoder is kept whole in memory too, so use the ...Max functions on data you did not make yourself.
  • Zstandard output is valid, not the smallest. The writer is simple, so ZstdCompressLevel makes frames usually a quarter to a half larger than the zstd tool at the same level; every reader still accepts them. Reading is complete, but only ZstdDecompressDict and ZstdDecompressEx take a dictionary, and the decoder holds the whole result in memory, so use ZstdDecompressMax on data from outside.
  • Bzip2 checks every checksum and can be capped. Bzip2Decompress verifies each block CRC and the whole stream CRC and rejects broken tables and indexes, and it accepts several streams one after another and nothing else after them. A tiny file can expand to a huge one, so use Bzip2DecompressMax on data from outside.
  • The method is not stored. Compress output does not say which method made it; keep the method number next to the data.
  • A str stops at the first zero byte. Keep binary data in a ByteBuf; BufToStrUnsafe is for bytes that are really text.
  • Buffers are not freed for you. Call BufFree on buffers you are done with, especially in loops and servers.

The guide below covers the jobs this package is used for day to day: gzipping and bzip2-ing text, laying out your own binary records in a ByteBuf, checking data with CRC-32, and choosing a compression method. Each section has a complete program with its output.

Gzip text for storage or transport

GzipCompressStr turns a string into a gzip stream that gzip -d, browsers and every other gzip reader understand, and GzipDecompressBuf turns it back. Gzip output is binary, so when it has to travel inside JSON, a URL or an environment variable, wrap it in Base64 with BufToBase64. On the receiving side encoding.Base64DecodeBytes writes the bytes into a buffer you have made big enough; set the buffer's len to the count it returns. It throws when the text is not Base64 or does not fit, so call it inside try:.

compress-guide-gzip.salam
import compress
import encoding

func main:
    mut log := ""
    repeat 20:
        log += "GET /api/users 200 12ms\n"
    end
    packed := compress.GzipCompressStr(log)
    println "original:", log.Len(), "bytes"
    println "gzipped: ", packed.len, "bytes"
    println "ratio:   ", compress.Ratio(packed.len, log.Len())
    text := compress.BufToBase64(packed)
    println "base64:  ", text.Len(), "characters"
    mut ok := false
    mut wire := compress.NewByteBufCap(text.Len())
    try:
        wire.len = encoding.Base64DecodeBytes(text, wire.data, text.Len())
        ok = true
    catch e:
        println "base64:", e.ToStr()
    end
    println "decoded: ", ok, compress.BufEquals(wire, packed)
    back := compress.GzipDecompressBuf(wire, ok)
    println "restored:", ok, compress.BufToStrUnsafe(back) == log
end
Outputoriginal: 480 bytes gzipped: 48 bytes ratio: 0.1 base64: 64 characters decoded: true true restored: true true

A log with the same line twenty times shrinks to a tenth of its size. Base64 costs a third back (64 characters for 48 bytes), which is still far smaller than the original. BufToStrUnsafe copies the bytes into a string, so use it only for data that is really text: a str ends at the first zero byte.

Bzip2 for smaller files

Bzip2CompressStr, Bzip2Compress and Bzip2CompressBuf write a standard bzip2 stream (what bzip2 and Python's bz2 read), and Bzip2CompressLevel takes a level from 1 to 9: the block size in units of 100000 bytes, 9 by default. Bzip2DecompressBuf reads one or more streams, including old ones with randomised blocks, and checks all CRCs. The program packs the same text both ways and then shows the two safety nets: a cap on the unpacked size, and a flipped bit.

compress-guide-bzip2.salam
import compress

func main:
    words := ["salam", "compress", "block", "sort", "stream", "byte", "table", "code", "bit", "run"]
    mut text := ""
    mut x := 7
    repeat 4000:
        x = (x * 75 + 74) % 65537
        text += words[x % 10]
        text += " "
    end
    gz := compress.GzipCompressStr(text)
    bz := compress.Bzip2CompressStr(text)
    println "original:", text.Len(), "bytes"
    println "gzip:    ", gz.len, "bytes"
    println "bzip2:   ", bz.len, "bytes"
    mut ok := false
    back := compress.Bzip2DecompressBuf(bz, ok)
    println "restored:", ok, compress.BufToStrUnsafe(back) == text
    _ := compress.Bzip2DecompressMax(bz.data, bz.len, 1000, ok)
    println "capped at 1000 bytes:", ok
    mut damaged := compress.BufClone(bz)
    compress.BufSet(damaged, damaged.len / 2, compress.BufGet(damaged, damaged.len / 2) ^ 1)
    _ := compress.Bzip2DecompressBuf(damaged, ok)
    println "damaged:", ok
end
Outputoriginal: 22837 bytes gzip: 3281 bytes bzip2: 2050 bytes restored: true true capped at 1000 bytes: false damaged: false

On text with many repeated words bzip2 beats gzip by a good margin, while on short or already compressed data the two are close. Bzip2 needs more time and memory per megabyte than gzip, so it suits files that are written once and read rarely. To work with .bz2 files on disk, use Bzip2CompressFile and Bzip2DecompressFile from archive.

Your own binary format with ByteBuf

A ByteBuf grows as you push into it, and a ByteReader walks over the bytes from the start. That is all you need for a small file format: a few magic bytes so that a wrong file is rejected, a count, and then each record as a length and its bytes. BufPushU32 and RdU32 write and read a 32 bit number in big-endian order; for little-endian or 64 bit numbers use encoding.binary, which works on the same buffers.

compress-guide-record.salam
import compress

func write_names(names: Vector<str>): compress.ByteBuf:
    mut b := compress.NewByteBuf()
    compress.BufPushBuf(b, compress.BufFromStr("NAMS"))
    compress.BufPushU32(b, names.Len())
    each name in names:
        compress.BufPushU32(b, name.Len())
        compress.BufPushBuf(b, compress.BufFromStr(name))
    end
    ret b
end

func read_names(b: compress.ByteBuf):
    mut r := compress.ReaderFromBuf(b)
    magic := compress.RdBytes(r, 4)
    if compress.BufToStrUnsafe(magic) != "NAMS":
        println "not a names file"
        ret
    end
    count := compress.RdU32(r)
    repeat count in i:
        n := compress.RdU32(r)
        name := compress.RdBytes(r, n)
        if r.bad:
            println "record", i, "is cut short"
            ret
        end
        println i, compress.BufToStrUnsafe(name)
    end
end

func main:
    mut names := Vector {} as Vector<str>
    names.Push("Ada")
    names.Push("Grace")
    names.Push("Linus")
    b := write_names(names)
    println "bytes:", compress.BufLen(b)
    read_names(b)
    mut cut := compress.NewByteReader(b.data, 20)
    read_names(compress.RdBytes(cut, 20))
end
Outputbytes: 33 0 Ada 1 Grace 2 Linus 0 Ada record 1 is cut short

The reader never reads past the end. When a read asks for more bytes than are left, the reader sets bad, returns zeros or an empty buffer, and stays bad, so it is enough to check r.bad once after a group of reads. The last call reads a copy cut to 20 bytes: the first name still fits, the second does not.

Checking data with CRC-32

A checksum tells you whether bytes arrived unchanged. CRC32Str and CRC32Buf compute the same CRC-32 that zip, gzip and PNG use; Adler32Str and Adler32Buf are faster and weaker, the one zlib uses. The program seals a message by appending its CRC, then shows that changing a single byte is caught.

compress-guide-checksum.salam
import compress

func seal(payload: str): compress.ByteBuf:
    mut b := compress.BufFromStr(payload)
    compress.BufPushU32(b, compress.CRC32Str(payload) as int)
    ret b
end

func check(b: compress.ByteBuf): bool:
    if b.len < 4: ret false end
    mut r := compress.NewByteReader(b.data, b.len)
    body := compress.RdBytes(r, b.len - 4)
    stored := compress.RdU32(r) as u32
    ret compress.CRC32Buf(body) == stored
end

func gunzip_checked(g: compress.ByteBuf, ok &: bool): compress.ByteBuf:
    out := compress.GzipDecompressBuf(g, ok)
    if not ok: ret out end
    at := g.len - 8
    stored := compress.BufGet(g, at) | (compress.BufGet(g, at + 1) << 8) | (compress.BufGet(g, at + 2) << 16) | (compress.BufGet(g, at + 3) << 24)
    ok = compress.CRC32Buf(out) == stored as u32
    ret out
end

func main:
    mut packet := seal("move 3 to slot 7")
    println "length:", packet.len, "intact:", check(packet)
    compress.BufSet(packet, 5, 52)
    println "after one changed byte, intact:", check(packet)
    g := compress.GzipCompressStr("move 3 to slot 7")
    mut broken := compress.BufClone(g)
    compress.BufSet(broken, broken.len - 6, compress.BufGet(broken, broken.len - 6) ^ 1)
    mut ok := false
    _ := compress.GzipDecompressBuf(broken, ok)
    println "damaged gzip, plain decompress:", ok
    _ := gunzip_checked(broken, ok)
    println "damaged gzip, checked:", ok
    _ := gunzip_checked(g, ok)
    println "good gzip, checked:", ok
end
Outputlength: 20 intact: true after one changed byte, intact: false damaged gzip, plain decompress: false damaged gzip, checked: false good gzip, checked: true

GzipDecompress and GzipDecompressBuf check the header and the optional header CRC, inflate the data, and then compare the CRC-32 and the length in the last eight bytes of the stream. A flipped bit there, or a corruption that still decodes, is reported by ok being false, so you never see damaged data as a successful read.

Choosing a method

Compress, CompressBuf and CompressStr take a method number, and CompressBest tries each one and keeps the smallest result. The best method depends on the shape of the data, as these output sizes show:

compress-guide-choose.salam
import compress
import str

func methodName(m: int): str:
    switch m:
        compress.MethodRLE: ret "rle" end
        compress.MethodHuffman: ret "huffman" end
        compress.MethodLZ77: ret "lz77" end
        compress.MethodLZW: ret "lzw" end
        compress.MethodDeflate: ret "deflate" end
        compress.MethodBWT: ret "bwt" end
    end
    ret "none (stored)"
end

func row(label: str, data: str):
    b := compress.BufFromStr(data)
    mut line := label + ": " + str.FromInt(b.len) + " bytes, rle " + str.FromInt(compress.CompressBuf(compress.MethodRLE, b).len)
    line = line + ", lz77 " + str.FromInt(compress.CompressBuf(compress.MethodLZ77, b).len)
    line = line + ", gzip " + str.FromInt(compress.GzipCompressBuf(b).len)
    mut best := 0
    smallest := compress.CompressBest(b.data, b.len, best)
    println line
    println "  best:", methodName(best), "at", smallest.len, "bytes"
end

func main:
    mut runs := ""
    repeat 200: runs += "0" end
    mut words := ""
    repeat 10: words += "error at line 42; " end
    row("runs", runs)
    row("words", words)
    row("short", "hi there")
end
Outputruns: 200 bytes, rle 8, lz77 30, gzip 23 best: rle at 8 bytes words: 180 bytes, rle 186, lz77 44, gzip 40 best: lz77 at 44 bytes short: 8 bytes, rle 13, lz77 13, gzip 28 best: none (stored) at 12 bytes

RLE wins on long runs of one byte and makes ordinary text bigger. LZ77 does well on repeated phrases. MethodDeflate is left out of the list because it is this package's own container, which stores its Huffman tables in full and so starts at almost 300 bytes; gzip uses real DEFLATE with compact tables and is the better choice whenever the data leaves your program. Two rules follow: the output of Compress does not say which method made it, so store the method number next to the data (the value CompressBest wrote into its last argument), and very short inputs are best stored as they are (method 0, which CompressBest reports as none for the short line above).

Read .xz and .lzma files

XzDecompress opens the .xz files that xz, tar -J and 7z a -txz write, and LzmaDecompress the older .lzma format. An xz file is checked as it is read: the headers, the index and the CRC-32, CRC-64 or SHA-256 of every block, and any mismatch gives ok false with an empty result. The decoders hold the whole result in memory, so on data you did not make yourself call XzDecompressMax or LzmaDecompressMax with the biggest size you accept; a few hundred bytes of LZMA can describe gigabytes.

compress-guide-xz.salam
import compress

const PACKED := """
fd377a585a000004e6d6b44604c05fcf022101160000000000000000bb0ccf6ce0014e00575d0029984986240afb8a34e4
81db780958f3d0144436dd3bb4ca45234051fa042e5be576402f66b72801fb0f3b421e32e3eba071037cdd425338444ede
4cb8c8c742828bc92c1b53df8ea763015d902c3d49f9c602aacf40a000005b35fff6fd65c4e300017bcf020000005e9efc
63b1c467fb020000000004595a
"""

func nib(c: int): int:
    if c >= 48 and c <= 57:
        ret c - 48
    end
    if c >= 97 and c <= 102:
        ret c - 87
    end
    ret -1
end

func unhex(s: str): compress.ByteBuf:
    mut b := compress.NewByteBuf()
    bp := s as u8*
    mut hi := -1
    repeat s.Len() in i:
        v := nib(bp[i] as int)
        if v < 0:
            continue
        end
        if hi < 0:
            hi = v
        else:
            compress.BufPush(b, hi * 16 + v)
            hi = -1
        end
    end
    ret b
end

func main:
    packed := unhex(PACKED)
    mut ok := false
    text := compress.XzDecompressBuf(packed, ok)
    println "decoded:", ok, text.len, "bytes from", packed.len
    println compress.BufToStrUnsafe(text)

    mut damaged := compress.BufClone(packed)
    compress.BufSet(damaged, damaged.len - 20, compress.BufGet(damaged, damaged.len - 20) ^ 1)
    broken := compress.XzDecompressBuf(damaged, ok)
    println "damaged file:", ok, broken.len

    small := compress.XzDecompressMax(packed.data, packed.len, 100, ok)
    println "capped at 100 bytes:", ok, small.len
end
Outputdecoded: true 335 bytes from 160 Salam is a programming language. This text is packed with xz. Salam is a programming language. This text is packed with xz. Salam is a programming language. This text is packed with xz. Salam is a programming language. This text is packed with xz. Salam is a programming language. This text is packed with xz. Last line, not repeated. damaged file: false 0 capped at 100 bytes: false 0

Write .xz and .lzma files

XzCompressStr, XzCompress and XzCompressBuf write a standard .xz file that xz, unxz, tar -J and 7-Zip open: one LZMA2 block, a CRC-64 check, and an index. XzCompressLevel takes a level from 0 (fastest, 256 KiB window) to 9 (64 MiB window, longest search), and XzCompressCheck also picks the check: XzCheckNone, XzCheckCrc32, XzCheckCrc64 or XzCheckSha256. Data that does not shrink is stored in uncompressed chunks, so the result is only the container overhead over the input. LzmaCompressStr, LzmaCompress, LzmaCompressBuf and LzmaCompressLevel write the older .lzma format, which has no check; archive.XzCompressFile and archive.LzmaCompressFile do it for a file.

compress-guide-xz-write.salam
/*
 * Salam Programming Language (2024-2026)
 *
 *   +-------------------+
 *   |     S A L A M     |
 *   +-------------------+
 *
 * Designed by Seyyed Ali Mohammadiyeh and the Salam Team
 * Born from a decade of language design experience (since 2018)
 *
 * Repository: https://github.com/SalamLang/Salam
 *
 */

package main

import compress

func main:
    text := "the quick brown fox jumps over the lazy dog. " * 50
    mut ok := false

    xz := compress.XzCompressStr(text)
    back := compress.XzDecompressBuf(xz, ok)
    println ".xz:", text.Len(), "to", xz.len, "bytes", ok, back.len
    println "magic:", compress.BufGet(xz, 0), compress.BufGet(xz, 1), compress.BufGet(xz, 2)

    fast := compress.XzCompressLevel(text as void*, text.Len(), 0)
    best := compress.XzCompressLevel(text as void*, text.Len(), 9)
    println "levels:", fast.len, best.len

    none := compress.XzCompressCheck(text as void*, text.Len(), 6, compress.XzCheckNone)
    sha := compress.XzCompressCheck(text as void*, text.Len(), 6, compress.XzCheckSha256)
    println "checks:", none.len, xz.len, sha.len
    small := compress.XzDecompressBuf(sha, ok)
    println "sha256 stream:", ok, small.len

    lz := compress.LzmaCompressStr(text)
    again := compress.LzmaDecompressBuf(lz, ok)
    println ".lzma:", lz.len, "bytes", ok, again.len
    lz9 := compress.LzmaCompressLevel(text as void*, text.Len(), 9)
    println "level 9:", lz9.len
    buf := compress.LzmaCompressBuf(back)
    println "buffer:", compress.BufEquals(buf, lz)
    println "empty:", compress.XzCompressBuf(compress.NewByteBuf()).len
end
Output.xz: 2250 to 128 bytes true 2250 magic: 253 55 122 levels: 128 128 checks: 120 128 152 sha256 stream: true 2250 .lzma: 76 bytes true 2250 level 9: 76 buffer: true empty: 32

Inside 7z archives and xz blocks the compressed data comes without a container. LzmaRawDecompress and Lzma2RawDecompress read that form when you know the properties and the unpacked size, and BcjDecode and DeltaDecode undo the filters that sit in front of it. Dictionaries above 1 GiB (LzmaMaxDictSize) are refused.

Read and write .zst files

ZstdDecompress opens the .zst files that zstd, tar --zstd and 7z a -tzstd write: one frame or several in a row, any compression level, with or without a checksum, and frames written with long-distance matching or many threads. The decoder checks as it reads. A bad table, an offset that points before the data, a size that does not add up or a wrong XXH64 checksum gives ok false and an empty result; ZstdDecompressEx also says which of these it was, through the ZstdErr... codes. The whole result is kept in memory, so on data you did not make yourself call ZstdDecompressMax with the biggest size you accept.

compress-guide-zstd.salam
import compress

const PACKED := """
28b52ffd6459008502009285111290cf0184e3820d4c2042136477707af2ed3a409c9edcb0e3e7730fed14fb943c8038
53dea7a9ea4f2fc3f0527dda9e972a958b464c916376e4546b1cd18bf07ca9de9831090100407fa04214ee20b3a1
"""

func nib(c: int): int:
    if c >= 48 and c <= 57:
        ret c - 48
    end
    if c >= 97 and c <= 102:
        ret c - 87
    end
    ret -1
end

func unhex(s: str): compress.ByteBuf:
    mut b := compress.NewByteBuf()
    bp := s as u8*
    mut hi := -1
    repeat s.Len() in i:
        v := nib(bp[i] as int)
        if v < 0:
            continue
        end
        if hi < 0:
            hi = v
        else:
            compress.BufPush(b, hi * 16 + v)
            hi = -1
        end
    end
    ret b
end

func main:
    packed := unhex(PACKED)
    mut ok := false
    text := compress.ZstdDecompressBuf(packed, ok)
    println "decoded:", ok, text.len, "bytes from", packed.len
    println compress.BufToStrUnsafe(text)

    mut damaged := compress.BufClone(packed)
    compress.BufSet(damaged, damaged.len - 20, compress.BufGet(damaged, damaged.len - 20) ^ 1)
    mut status := 0
    broken := compress.ZstdDecompressEx(damaged.data, damaged.len, 0, null, 0, status)
    println "damaged file:", status, compress.ZstdStatusText(status), broken.len

    small := compress.ZstdDecompressMax(packed.data, packed.len, 100, ok)
    println "capped at 100 bytes:", ok, small.len

    mine := compress.ZstdCompressLevel(text.data, text.len, 19)
    again := compress.ZstdDecompressBuf(mine, ok)
    println "our own frame:", mine.len, "bytes,", compress.BufEquals(again, text)
    println "size recorded in the header:", compress.ZstdContentSize(mine.data, mine.len)
end
Outputdecoded: true 345 bytes from 94 Salam is a programming language. This text is packed with zstd. Salam is a programming language. This text is packed with zstd. Salam is a programming language. This text is packed with zstd. Salam is a programming language. This text is packed with zstd. Salam is a programming language. This text is packed with zstd. Last line, not repeated. damaged file: 4 checksum mismatch 0 capped at 100 bytes: false 0 our own frame: 110 bytes, true size recorded in the header: 345

ZstdCompressLevel writes a standard frame at a level from 1 to 22 (0 stores the data as it is), with the content size and a checksum in the header. The writer is deliberately simple: it finds repeats with a greedy or lazy search over a hash chain and codes them with the predefined tables, so the result is usually a quarter to a half larger than the zstd tool's at the same level. Every reader accepts it. Dictionaries are supported for reading only (ZstdDecompressDict); the writer does not use one.

Unpack a big file without holding it in memory

Everything above works on a whole buffer. For a file that may not fit in memory, NewFileBitReader reads a part of a file in pieces of 64 KiB and RawInflateToSink writes the unpacked data into a FileSink, which counts the bytes, keeps a CRC-32 and stops at a cap you choose. The archive package uses exactly these calls to unpack zip, gzip and RAR entries to disk. FileSeek, FileSize and FileReadAt use 64 bit offsets, and CRC32Update lets you checksum data that arrives in parts.

compress-guide-stream.salam
import compress
import mem

func main:
    text := "stream me, stream me, stream me. " * 100
    packed := compress.RawDeflate(text as void*, text.Len())

    mut sink := compress.SinkOpen("packed.bin", 0 as i64)
    println compress.SinkWrite(sink, packed.data, packed.len), sink.written == (packed.len as i64), sink.failed
    println compress.SinkClose(sink), sink.fp == null

    fp := compress.FileOpenRead("packed.bin")
    size := compress.FileSize(fp)
    println size == (packed.len as i64)
    head := mem.Allocate(4 as u64)
    got := compress.FileReadAt(fp, 0 as i64, head, 4)
    println got, compress.FileSeek(fp, 0 as i64)
    mem.Free(head)

    mut reader := compress.NewFileBitReader(fp, 0 as i64, size)
    println reader.fp == fp, reader.left == size
    mut out := compress.SinkOpen("text.txt", 0 as i64)
    ok := compress.RawInflateToSink(reader, out, false)
    compress.FileBitReaderFree(reader)
    println ok, compress.SinkClose(out), out.written == (text.Len() as i64), out.crc == compress.CRC32Str(text)

    mut small := compress.SinkOpen("small.txt", 100 as i64)
    mut again := compress.NewFileBitReader(fp, 0 as i64, size)
    println compress.RawInflateToSink(again, small, false), small.over, small.cap
    compress.FileBitReaderFree(again)
    compress.SinkClose(small)

    mut part := compress.SinkOpen("part.bin", 0 as i64)
    println compress.SinkCopyFrom(part, fp, 2 as i64, 10 as i64), part.written
    compress.SinkClose(part)
    compress.FileClose(fp)

    first := "hello, "
    second := "world"
    running := compress.CRC32Update(compress.CRC32Update(0 as u32, first as void*, first.Len()), second as void*, second.Len())
    println running == compress.CRC32Str(first + second)

    wf := compress.FileOpenWrite("empty.bin")
    println compress.FileSize(wf)
    compress.FileClose(wf)
    println compress.FileOpenRead("no-such-file.bin") == null
end
Outputtrue true false true true true 4 true true true true true true true false true 100 true 10 true 0 true

A sink that reaches its cap sets over and refuses the rest, so a small hostile file cannot fill the disk. Check the result of SinkClose too: a full disk shows up there as a failed write.

Which function for which job

JobUse
Data that other programs will readGzipCompressStr, GzipCompressBuf, GzipDecompressBuf
Write a .xz or .lzma fileXzCompressStr, XzCompressLevel, XzCompressCheck, LzmaCompressStr, LzmaCompressLevel
Read a .xz or .lzma fileXzDecompressBuf, LzmaDecompressBuf; the ...Max variants for data from outside
Read or write a .zst fileZstdCompressBuf, ZstdCompressLevel, ZstdDecompressBuf, and ZstdDecompressMax for data from outside
Smaller data than gzip, or a .bz2 streamBzip2CompressStr, Bzip2CompressLevel, Bzip2DecompressBuf, and Bzip2DecompressMax for untrusted input
Standard DEFLATE with no header, as zip entries and zlib streams hold itRawDeflate, and RawInflate with the original size
Smallest result for data only you readCompressBest, then Decompress with the method it chose
Build bytes, then read them backNewByteBuf, BufPush..., ReaderFromBuf, Rd...
Fields narrower than a byteNewBitWriter, BitWriteBits, BitReadBits
Detect damaged dataCRC32Buf, or Adler32Buf when speed matters more
Binary into textBufToBase64, and encoding.Base64DecodeBytes back

Buffers are not freed for you. Call BufFree on a buffer you no longer need when a program makes many of them, for example in a loop or a long running server.

Types

FileSink

struct FileSink
A destination for unpacked data that writes to a file piece by piece, counts the bytes, keeps a running CRC-32 and can refuse more than a set number of bytes. The decoders write into it, so a big result never has to sit in memory whole.
NameTypeDescription
fpvoid*The open file; null after SinkClose.
writteni64 = 0The number of bytes written so far.
crcu32 = 0The CRC-32 of the bytes written so far.
capi64 = 0The most bytes the sink accepts; zero means no limit.
overbool = falseA write would have passed cap and was refused.
failedbool = falseThe file could not be created or a write to it failed.
emitfunc(void*, int, void*): boolThe function a sink made by SinkCallback passes each piece to; return false to stop the decoder.
has_emitbool = falseTrue when the sink calls emit.
ctxvoid*The value handed to emit as its third argument.
compress-Stream.salam
import compress
import mem

func main:
    text := "stream me, stream me, stream me. " * 100
    packed := compress.RawDeflate(text as void*, text.Len())

    mut sink := compress.SinkOpen("packed.bin", 0 as i64)
    println compress.SinkWrite(sink, packed.data, packed.len), sink.written == (packed.len as i64), sink.failed
    println compress.SinkClose(sink), sink.fp == null

    fp := compress.FileOpenRead("packed.bin")
    size := compress.FileSize(fp)
    println size == (packed.len as i64)
    head := mem.Allocate(4 as u64)
    got := compress.FileReadAt(fp, 0 as i64, head, 4)
    println got, compress.FileSeek(fp, 0 as i64)
    mem.Free(head)

    mut reader := compress.NewFileBitReader(fp, 0 as i64, size)
    println reader.fp == fp, reader.left == size
    mut out := compress.SinkOpen("text.txt", 0 as i64)
    ok := compress.RawInflateToSink(reader, out, false)
    compress.FileBitReaderFree(reader)
    println ok, compress.SinkClose(out), out.written == (text.Len() as i64), out.crc == compress.CRC32Str(text)

    mut small := compress.SinkOpen("small.txt", 100 as i64)
    mut again := compress.NewFileBitReader(fp, 0 as i64, size)
    println compress.RawInflateToSink(again, small, false), small.over, small.cap
    compress.FileBitReaderFree(again)
    compress.SinkClose(small)

    mut part := compress.SinkOpen("part.bin", 0 as i64)
    println compress.SinkCopyFrom(part, fp, 2 as i64, 10 as i64), part.written
    compress.SinkClose(part)
    compress.FileClose(fp)

    first := "hello, "
    second := "world"
    running := compress.CRC32Update(compress.CRC32Update(0 as u32, first as void*, first.Len()), second as void*, second.Len())
    println running == compress.CRC32Str(first + second)

    gen := (off: i64, dst: void*, n: int, _c: void*):
        p := dst as u8*
        repeat n in k:
            p[k] = ((off + (k as i64)) & (255 as i64)) as u8
        end
        ret n
    end
    marker := mem.Allocate(8 as u64)
    cfp := compress.CallbackOpenRead(1000 as i64, gen, marker)
    println compress.FileSize(cfp), compress.CallbackContext(cfp) == marker
    probe := mem.Allocate(4 as u64)
    println compress.FileReadAt(cfp, 258 as i64, probe, 4), (probe as u8*)[0], (probe as u8*)[3]
    compress.FileClose(cfp)
    mem.Free(probe)
    mem.Free(marker)

    call := mem.AllocateZeroed(16 as u64) as u8*
    call[0] = 0xE8 as u8
    call[2] = 1 as u8
    mut mask := 0
    done := compress.BcjStep(compress.BcjX86, call as void*, 16, 0 as i64, mask)
    println done, call[1] as int, call[2] as int, call[3] as int, call[4] as int, mask
    mem.Free(call as void*)

    wf := compress.FileOpenWrite("empty.bin")
    println compress.FileSize(wf)
    compress.FileClose(wf)
    println compress.FileOpenRead("no-such-file.bin") == null
end
Outputtrue true false true true true 4 true true true true true true true false true 100 true 10 true 1000 true 4 2 5 12 251 0 0 0 0 0 true

Method

enum Method
The compression method, as an enum.
NameTypeDescription
NoneNo compression.
RLERun length encoding.
HuffmanHuffman.
LZ77LZ77.
LZWLZW.
DeflateLZ77 and Huffman.
BWTBurrows Wheeler.
compress-Methods.salam
import compress

func main:
    text := "ababababab hello hello hello hello hello hello hello hello"
    b := compress.BufFromStr(text)
    each method in [compress.MethodNone, compress.MethodRLE, compress.MethodHuffman, compress.MethodLZ77, compress.MethodLZW, compress.MethodDeflate, compress.MethodBWT]:
        mut c := compress.CompressBuf(method, b)
        mut back := compress.DecompressBuf(method, c)
        println method, compress.BufLen(c), compress.BufEquals(b, back)
        compress.BufFree(c)
        compress.BufFree(back)
    end
    mut best := 0
    c := compress.CompressBest(b.data, b.len, best)
    println best, compress.Ratio(c.len, b.len) < 1.0
    s := compress.CompressStr(compress.MethodDeflate, "aaaaaaaaaa")
    b3 := compress.Decompress(compress.MethodDeflate, s.data, s.len)
    println compress.BufToStrUnsafe(b3)
    b4 := compress.Compress(compress.MethodRLE, b.data, b.len)
    println compress.BufLen(b4) > 0, compress.Method.Huffman
    println compress.BufLen(compress.RLEDecompress(compress.RLECompress(b.data, b.len).data, compress.RLECompress(b.data, b.len).len)) == b.len
    h := compress.HuffmanCompress(b.data, b.len)
    println compress.BufEquals(compress.HuffmanDecompress(h.data, h.len), b)
    l := compress.LZ77Compress(b.data, b.len)
    println compress.BufEquals(compress.LZ77Decompress(l.data, l.len), b)
    w := compress.LZWCompress(b.data, b.len)
    println compress.BufEquals(compress.LZWDecompress(w.data, w.len), b)
    bw := compress.BWTCompress(b.data, b.len)
    println compress.BufEquals(compress.BWTDecompress(bw.data, bw.len), b)
    mut primary := 0
    t := compress.BWTTransform(b.data, 3, primary)
    println primary, compress.BufToStrUnsafe(compress.BWTInverse(t.data, t.len, primary)) == "aba"
    m := compress.MTFEncode(b)
    println compress.BufEquals(compress.MTFDecode(m), b)
    mut good := false
    println compress.BufEquals(compress.RLEDecompressMax(b4.data, b4.len, 0, good), b), good
    println compress.BufEquals(compress.HuffmanDecompressMax(h.data, h.len, 0, good), b), good
    println compress.BufEquals(compress.LZ77DecompressMax(l.data, l.len, 0, good), b), good
    println compress.BufEquals(compress.LZWDecompressMax(w.data, w.len, 0, good), b), good
    println compress.BufEquals(compress.BWTDecompressMax(bw.data, bw.len, 0, good), b), good
    df := compress.DeflateCompress(b.data, b.len)
    println compress.BufEquals(compress.DeflateDecompressMax(df.data, df.len, 0, good), b), good
    println compress.BufLen(compress.DecompressMax(compress.MethodLZ77, l.data, l.len, 1, good)), good
    println compress.BufLen(compress.DecompressMax(compress.MethodLZ77, "not compressed data" as void*, 19, 0, good)), good
end
Output0 62 true 1 63 true 2 281 true 3 22 true 4 35 true 5 545 true 6 282 true 3 true aaaaaaaaaa true 2 true true true true true 1 true true true true true true true true true true true true true true 0 false 0 false

ByteBuf

struct ByteBuf
A byte buffer: an array of bytes that grows by itself. Most packages in this group take and give binary data in one.
NameTypeDescription
datavoid*The memory address of the bytes.
lenint = 0The number of bytes.
capint = 0The capacity.
compress-Buffer.salam
import compress

func main:
    mut empty := compress.NewByteBuf()
    println compress.BufLen(empty)
    compress.BufFree(empty)
    mut b := compress.NewByteBufCap(16)
    compress.BufPush(b, 72)
    compress.BufPush(b, 105)
    mut other := compress.BufFromStr("!!")
    compress.BufPushBuf(b, other)
    compress.BufPushBytes(b, other.data, 1)
    compress.BufPushU32(b, 10)
    println compress.BufLen(b), compress.BufGet(b, 0), compress.BufToBase64(b)
    compress.BufSet(b, 0, 104)
    mut c := compress.BufClone(b)
    println compress.BufEquals(b, c), compress.BufToStrUnsafe(compress.BufFromStr("hello"))
    mut r := compress.ReaderFromBuf(b)
    println compress.RdLeft(r), compress.RdByte(r), compress.RdByte(r)
    three := compress.RdBytes(r, 3)
    println compress.BufLen(three), compress.RdU32(r), compress.RdLeft(r), r.bad
    _ := compress.RdByte(r)
    println r.bad
    mut r2 := compress.NewByteReader(c.data, 2)
    println compress.RdByte(r2)
    compress.BufReset(b)
    println compress.BufLen(b)
    compress.BufFree(b)
    compress.BufFree(c)
    compress.BufFree(other)
end
Output0 9 72 SGkhISEAAAAK true hello 9 104 105 3 10 0 false true 104 0

ByteReader

struct ByteReader
A sequential reader of bytes over one piece of memory.
NameTypeDescription
datavoid*The address of the data.
lenint = 0The length of the data.
posint = 0The current position.
badbool = falseA read went past the end of the data.

The example for this item is shown with the ones above: compress-Buffer.salam

BitWriter

struct BitWriter
A writer of bits; it fills each byte from the least significant bit.
NameTypeDescription
bufByteBufThe bytes written.
accint = 0The bits waiting.
nbitsint = 0The number of bits waiting.
compress-Bits.salam
import compress

func main:
    mut w := compress.NewBitWriter()
    compress.BitWriteBit(w, 1)
    compress.BitWriteBits(w, 5, 3)
    compress.BitWriteBits(w, 255, 8)
    compress.BitFlush(w)
    println compress.BitWriterLen(w)
    mut r := compress.BitReaderFromBuf(w.buf)
    println compress.BitReadBit(r), compress.BitReadBits(r, 3), compress.BitReadBits(r, 8)
    mut r2 := compress.NewBitReader(w.buf.data, 1)
    println compress.BitReadBits(r2, 4)
    compress.BitWriterFree(w)
end
Output2 1 5 255 11

BitReader

struct BitReader
A reader of bits in the same order.
NameTypeDescription
datavoid*The data.
lenint = 0The length of the data.
posint = 0The current byte position.
accint = 0The bits read.
nbitsint = 0The number of bits ready.
badbool = falseA read went past the end of the data.
fpvoid*The file a bit reader made by NewFileBitReader reads from; null for a reader over memory.
lefti64 = 0The number of bytes of the file that are not read yet.
foffi64 = 0Where in the file the next piece is read from, when the reader was made by NewFileBitReader.

The example for this item is shown with the ones above: compress-Bits.salam

Constants and variables

DecodeOk

const DecodeOk: i32 = 0
The why value 0: the data was decoded and every check passed.
compress-Why.salam
import compress

func name(why: int): str:
    if why == compress.DecodeOk: ret "ok" end
    if why == compress.DecodeTooBig: ret "too big" end
    ret "corrupt"
end

func main:
    raw := compress.BufFromStr("abcabcabcabcabcabcabcabcabcabcabcabc")
    gz := compress.GzipCompress(raw.data, raw.len)
    xz := compress.XzCompress(raw.data, raw.len)
    bz := compress.Bzip2Compress(raw.data, raw.len)
    lz := compress.LzmaCompress(raw.data, raw.len)
    mut why := 0
    _ := compress.GzipDecompressWhy(gz.data, gz.len, 10, why)
    println "gzip", name(why)
    _ := compress.XzDecompressWhy(xz.data, xz.len, 10, why)
    println "xz", name(why)
    _ := compress.Bzip2DecompressWhy(bz.data, bz.len, 10, why)
    println "bzip2", name(why)
    _ := compress.LzmaDecompressWhy(lz.data, lz.len, 10, why)
    println "lzma", name(why)
    back := compress.GzipDecompressWhy(gz.data, gz.len, 0, why)
    println "no limit", name(why), back.len
    _ := compress.GzipDecompressWhy(gz.data, gz.len - 6, 0, why)
    println "cut", name(why)
end
Outputgzip too big xz too big bzip2 too big lzma too big no limit ok 36 cut corrupt

DecodeCorrupt

const DecodeCorrupt: i32 = 1
The why value 1: the data is damaged, cut short, not the expected format, or its checksum does not match.

The example for this item is shown with the ones above: compress-Why.salam

DecodeTooBig

const DecodeTooBig: i32 = 2
The why value 2: the data is valid so far but would unpack to more than the limit you gave.

The example for this item is shown with the ones above: compress-Why.salam

DecodeIo

const DecodeIo: i32 = 3
The result value 3: the data was valid so far but the output side failed, for example the disk was full.
compress-Bzip2File.salam
import archive
import compress
import mem
import os
import str

func collect(p: void*, n: int, c: void*): bool:
    compress.BufPushBytes((c as compress.ByteBuf*)[0], p, n)
    ret true
end

func main:
    mut text := compress.NewByteBuf()
    repeat 5000 in i:
        line := "line " + str.FromInt((i * 7) as i64) + "\n"
        compress.BufPushBytes(text, line as void*, line.Len())
    end
    mut packed := compress.Bzip2Compress(text.data, text.len)
    archive.WriteBinaryFileBuf("bz.tmp", packed)

    fp := compress.FileOpenRead("bz.tmp")
    got := mem.AllocateZeroed(sizeof(compress.ByteBuf)) as compress.ByteBuf*
    got[0] = compress.NewByteBuf()
    mut sink := compress.SinkCallback(0 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
    st := compress.Bzip2FileToSink(fp, 0 as i64, compress.FileSize(fp), sink)
    println st == compress.DecodeOk, sink.written == (text.len as i64), compress.BufEquals(got[0], text)
    compress.FileClose(fp)

    fp2 := compress.FileOpenRead("bz.tmp")
    mut small := compress.SinkCallback(100 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
    println compress.Bzip2FileToSink(fp2, 0 as i64, compress.FileSize(fp2), small) == compress.DecodeTooBig
    compress.FileClose(fp2)
    println compress.DecodeIo
    compress.BufFree(got[0])
    mem.Free(got as void*)
    compress.BufFree(text)
    compress.BufFree(packed)
    os.Remove("bz.tmp")
end
Outputtrue true true true 3

XzCheckNone

const XzCheckNone: i32 = 0
No integrity check in the xz file (the smallest header and footer).
compress-XzWrite.salam
/*
 * Salam Programming Language (2024-2026)
 *
 *   +-------------------+
 *   |     S A L A M     |
 *   +-------------------+
 *
 * Designed by Seyyed Ali Mohammadiyeh and the Salam Team
 * Born from a decade of language design experience (since 2018)
 *
 * Repository: https://github.com/SalamLang/Salam
 *
 */

package main

import compress

func main:
    text := "the quick brown fox jumps over the lazy dog. " * 50
    mut ok := false

    xz := compress.XzCompressStr(text)
    back := compress.XzDecompressBuf(xz, ok)
    println ".xz:", text.Len(), "to", xz.len, "bytes", ok, back.len
    println "magic:", compress.BufGet(xz, 0), compress.BufGet(xz, 1), compress.BufGet(xz, 2)

    fast := compress.XzCompressLevel(text as void*, text.Len(), 0)
    best := compress.XzCompressLevel(text as void*, text.Len(), 9)
    println "levels:", fast.len, best.len

    none := compress.XzCompressCheck(text as void*, text.Len(), 6, compress.XzCheckNone)
    sha := compress.XzCompressCheck(text as void*, text.Len(), 6, compress.XzCheckSha256)
    println "checks:", none.len, xz.len, sha.len
    small := compress.XzDecompressBuf(sha, ok)
    println "sha256 stream:", ok, small.len

    lz := compress.LzmaCompressStr(text)
    again := compress.LzmaDecompressBuf(lz, ok)
    println ".lzma:", lz.len, "bytes", ok, again.len
    lz9 := compress.LzmaCompressLevel(text as void*, text.Len(), 9)
    println "level 9:", lz9.len
    buf := compress.LzmaCompressBuf(back)
    println "buffer:", compress.BufEquals(buf, lz)
    println "empty:", compress.XzCompressBuf(compress.NewByteBuf()).len
end
Output.xz: 2250 to 128 bytes true 2250 magic: 253 55 122 levels: 128 128 checks: 120 128 152 sha256 stream: true 2250 .lzma: 76 bytes true 2250 level 9: 76 buffer: true empty: 32

XzCheckCrc32

const XzCheckCrc32: i32 = 1
A CRC-32 of the unpacked data.

The example for this item is shown with the ones above: compress-XzWrite.salam

XzCheckCrc64

const XzCheckCrc64: i32 = 4
A CRC-64 of the unpacked data, the default of xz.

The example for this item is shown with the ones above: compress-XzWrite.salam

XzCheckSha256

const XzCheckSha256: i32 = 10
A SHA-256 of the unpacked data.

The example for this item is shown with the ones above: compress-XzWrite.salam

BcjX86

const BcjX86: i32 = 4
The branch filter for x86 code (CALL and JMP).
compress-XzRaw.salam
import compress
import str

const RAW_LZMA := """
0029984986240afb8a34e481db780958f3d0144436dd3bb4ca45234051fa042e5be576402f66b72801fb0f3b421e32e3eb
a071037cdd425338444ede4cb8c8c742828bc92c1b53df8ea763015d902c3d49f9c602ad8d68e0bcfed44dc0
"""

const RAW_LZMA2 := """
e0014e00595d0029984986240afb8a34e481db780958f3d0144436994e538c60ac746c02dc278329cadecbb68a986c0818
f3d60fca80edf59196fa1d512909ac7b63cbe3650a7fe9860c6f9bfc85ae148860e1e0b15ff7ea8080b5235108050000
"""

func nib(c: int): int:
    if c >= 48 and c <= 57:
        ret c - 48
    end
    if c >= 97 and c <= 102:
        ret c - 87
    end
    ret -1
end

func unhex(s: str): compress.ByteBuf:
    mut b := compress.NewByteBuf()
    bp := s as u8*
    mut hi := -1
    repeat s.Len() in i:
        v := nib(bp[i] as int)
        if v < 0:
            continue
        end
        if hi < 0:
            hi = v
        else:
            compress.BufPush(b, hi * 16 + v)
            hi = -1
        end
    end
    ret b
end

func show(label: str, b: compress.ByteBuf):
    mut line := label
    repeat b.len in i:
        v := compress.BufGet(b, i)
        line += " "
        line += str.FromInt(v as i64)
    end
    println line
end

func main:
    println "dictionary limit:", compress.LzmaMaxDictSize
    props := unhex("5d00000100")
    raw1 := unhex(RAW_LZMA)
    mut ok := false
    a := compress.LzmaRawDecompress(props.data, props.len, raw1.data, raw1.len, 335, 0, ok)
    println "lzma raw:", ok, a.len
    raw2 := unhex(RAW_LZMA2)
    b := compress.Lzma2RawDecompress(16, raw2.data, raw2.len, 335, 0, ok)
    println "lzma2 raw:", ok, compress.BufEquals(a, b)
    c := compress.Lzma2RawDecompress(16, raw2.data, raw2.len, 100, 0, ok)
    println "lzma2 raw, wrong size:", ok, c.len

    call := unhex("e810000000909090")
    show("before", call)
    println compress.BcjDecode(compress.BcjX86, call.data, call.len, 0)
    show("x86   ", call)

    steps := unhex("01010101")
    println compress.DeltaDecode(steps.data, steps.len, 1)
    show("delta ", steps)
    println compress.BcjDecode(99, steps.data, steps.len, 0), compress.DeltaDecode(steps.data, steps.len, 0)
end
Outputdictionary limit: 1073741824 lzma raw: true 335 lzma2 raw: true true lzma2 raw, wrong size: false 0 before 232 16 0 0 0 144 144 144 true x86 232 11 0 0 0 144 144 144 true delta 1 2 3 4 false false

BcjPowerPC

const BcjPowerPC: i32 = 5
The branch filter for big-endian PowerPC code.

The example for this item is shown with the ones above: compress-XzRaw.salam

BcjIA64

const BcjIA64: i32 = 6
The branch filter for Itanium (IA-64) code.

The example for this item is shown with the ones above: compress-XzRaw.salam

BcjARM

const BcjARM: i32 = 7
The branch filter for 32 bit ARM code.

The example for this item is shown with the ones above: compress-XzRaw.salam

BcjARMThumb

const BcjARMThumb: i32 = 8
The branch filter for ARM Thumb code.

The example for this item is shown with the ones above: compress-XzRaw.salam

BcjSPARC

const BcjSPARC: i32 = 9
The branch filter for SPARC code.

The example for this item is shown with the ones above: compress-XzRaw.salam

BcjARM64

const BcjARM64: i32 = 10
The branch filter for 64 bit ARM code.

The example for this item is shown with the ones above: compress-XzRaw.salam

BcjRISCV

const BcjRISCV: i32 = 11
The branch filter for RISC-V code.

The example for this item is shown with the ones above: compress-XzRaw.salam

Method numbers Method…

The same Method enum cases as plain numbers, for Compress and Decompress.
NameValue
MethodNone0
MethodRLE1
MethodHuffman2
MethodLZ773
MethodLZW4
MethodDeflate5
MethodBWT6

The example for this item is shown with the ones above: compress-Methods.salam

ZstdOk

const ZstdOk: i32 = 0
Status code 0: the data was decoded and every check passed.
compress-Zstd.salam
import compress
import str

func main:
    mut text := compress.NewByteBuf()
    repeat 60 in i:
        line := "entry " + str.FromInt(i * 37 % 101) + ": Salam writes zstd frames\n"
        compress.BufPushBytes(text, line as void*, line.Len())
    end
    packed := compress.ZstdCompress(text.data, text.len)
    println "default level:", packed.len, "bytes from", text.len
    println "is zstd:", compress.IsZstd(packed.data, packed.len), compress.IsZstdBuf(packed), compress.IsZstd(text.data, text.len)
    println "size in the header:", compress.ZstdContentSize(packed.data, packed.len)

    mut ok := false
    back := compress.ZstdDecompress(packed.data, packed.len, ok)
    println "decoded:", ok, compress.BufEquals(back, text)
    back2 := compress.ZstdDecompressBuf(packed, ok)
    println "buffer form:", ok, back2.len

    fast := compress.ZstdCompressLevel(text.data, text.len, 1)
    best := compress.ZstdCompressLevel(text.data, text.len, 19)
    stored := compress.ZstdCompressLevel(text.data, text.len, 0)
    println "levels 1 and 19 smaller than stored:", fast.len < stored.len, best.len < stored.len
    println "stored adds", stored.len - text.len, "bytes of framing"
    plain := compress.ZstdCompressOpts(text.data, text.len, 3, false)
    println "no checksum saves", packed.len - plain.len, "bytes"
    fromstr := compress.ZstdCompressStr("hello hello hello hello")
    fromstr_buf := compress.ZstdCompressBuf(text)
    println "string and buffer forms:", fromstr.len, fromstr_buf.len

    mut status := 0
    capped := compress.ZstdDecompressEx(packed.data, packed.len, 100, null, 0, status)
    println "capped at 100:", status, compress.ZstdStatusText(status), capped.len
    limit := compress.ZstdDecompressMax(packed.data, packed.len, text.len, ok)
    println "capped at the exact size:", ok, limit.len
    bad := compress.BufFromStr("not a zstd stream")
    nope := compress.ZstdDecompressEx(bad.data, bad.len, 0, null, 0, status)
    println "garbage:", status, compress.ZstdStatusText(status), nope.len
    println "codes:", compress.ZstdOk, compress.ZstdErrFormat, compress.ZstdErrTruncated, compress.ZstdErrCorrupt, compress.ZstdErrChecksum, compress.ZstdErrTooBig, compress.ZstdErrDict
end
Outputdefault level: 236 bytes from 2095 is zstd: true true false size in the header: 2095 decoded: true true buffer form: true 2095 levels 1 and 19 smaller than stored: true true stored adds 14 bytes of framing no checksum saves 4 bytes string and buffer forms: 25 236 capped at 100: 5 output exceeds the limit 0 capped at the exact size: true 2095 garbage: 1 not a zstd stream 0 codes: 0 1 2 3 4 5 6

ZstdErrFormat

const ZstdErrFormat: i32 = 1
Status code 1: the data does not start with a Zstandard frame, or a header sets something this reader does not accept (a reserved bit, or a window above 2 GiB).

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdErrTruncated

const ZstdErrTruncated: i32 = 2
Status code 2: the data ends in the middle of a header, a block or a frame.

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdErrCorrupt

const ZstdErrCorrupt: i32 = 3
Status code 3: a block does not decode. The cause can be a bad table, an offset that reaches before the start of the data, a size that does not add up or a reserved block type.

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdErrChecksum

const ZstdErrChecksum: i32 = 4
Status code 4: the frame decoded but its XXH64 content checksum does not match.

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdErrTooBig

const ZstdErrTooBig: i32 = 5
Status code 5: the unpacked data would pass the limit you gave, or the default limit of just under 1 GiB.

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdErrDict

const ZstdErrDict: i32 = 6
Status code 6: the frame names a dictionary that was not given, or the dictionary given is damaged or has another ID.

The example for this item is shown with the ones above: compress-Zstd.salam

LzmaMaxDictSize

const LzmaMaxDictSize: i32 = 1073741824
The largest dictionary size the decoders accept, 1 GiB (1073741824). A file that asks for more is refused with ok false.

The example for this item is shown with the ones above: compress-XzRaw.salam

Functions

MTFEncode

func MTFEncode(inbuf: ByteBuf): ByteBuf
Move to front encoding: every byte is replaced by its place in a list of the recent bytes.

The example for this item is shown with the ones above: compress-Methods.salam

MTFDecode

func MTFDecode(inbuf: ByteBuf): ByteBuf
Undoes move to front.

The example for this item is shown with the ones above: compress-Methods.salam

BWTTransform

func BWTTransform(p: void*, n: int, primary_out &: int): ByteBuf
The Burrows Wheeler transform: moves the bytes so that similar bytes end up next to each other; the index of the original row comes back in primary_out.

The example for this item is shown with the ones above: compress-Methods.salam

BWTInverse

func BWTInverse(p: void*, n: int, primary: int): ByteBuf
Undoes the Burrows Wheeler transform, given the index of the original row.

The example for this item is shown with the ones above: compress-Methods.salam

BWTCompress

func BWTCompress(p: void*, n: int): ByteBuf
Burrows Wheeler, then move to front, then compression.

The example for this item is shown with the ones above: compress-Methods.salam

BWTDecompressMax

func BWTDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBuf
Like BWTDecompress but with a cap on the output size (0 means 1 GiB) and an ok flag. Damaged, truncated or oversized input gives ok false and an empty buffer; it never reads or writes out of bounds.

The example for this item is shown with the ones above: compress-Methods.salam

BWTDecompress

func BWTDecompress(p: void*, n: int): ByteBuf
Decompresses data compressed with Burrows Wheeler.

The example for this item is shown with the ones above: compress-Methods.salam

LZ77Compress

func LZ77Compress(p: void*, n: int): ByteBuf
LZ77 compression: repeated pieces are replaced by a reference to where they were before.

The example for this item is shown with the ones above: compress-Methods.salam

LZ77DecompressMax

func LZ77DecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBuf
Like LZ77Decompress but with a cap on the output size (0 means 1 GiB) and an ok flag. Damaged, truncated or oversized input gives ok false and an empty buffer; it never reads or writes out of bounds.

The example for this item is shown with the ones above: compress-Methods.salam

LZ77Decompress

func LZ77Decompress(p: void*, n: int): ByteBuf
Decompresses LZ77 data.

The example for this item is shown with the ones above: compress-Methods.salam

ZstdCompressFileToSink

func ZstdCompressFileToSink(fp: void*, off: i64, len: i64, level: int, sink &: FileSink): bool
Compresses len bytes of the open file fp from off into one zstd frame with a checksum and writes it to a sink block by block. The input is held in a buffer of about two windows (a window is 1 MiB at the default level), so the size of the file does not matter. The bytes are the same as ZstdCompressLevel would give. level below 1 stores the blocks raw.
compress-CompressSink.salam
import archive
import compress
import mem
import os
import str

func collect(p: void*, n: int, c: void*): bool:
    compress.BufPushBytes((c as compress.ByteBuf*)[0], p, n)
    ret true
end

func main:
    mut text := compress.NewByteBuf()
    repeat 20000 in i:
        line := "record " + str.FromInt((i * 13) as i64) + " ok\n"
        compress.BufPushBytes(text, line as void*, line.Len())
    end
    archive.WriteBinaryFileBuf("sink.tmp", text)
    fp := compress.FileOpenRead("sink.tmp")
    size := compress.FileSize(fp)
    out := mem.AllocateZeroed(sizeof(compress.ByteBuf)) as compress.ByteBuf*
    emit := (p: void*, n: int, c: void*): ret collect(p, n, c) end

    out[0] = compress.NewByteBuf()
    mut bz := compress.SinkCallback(0 as i64, emit, out as void*)
    println compress.Bzip2CompressFileToSink(fp, 0 as i64, size, 9, bz)
    mut one := compress.Bzip2CompressLevel(text.data, text.len, 9)
    println "bzip2 same bytes", compress.BufEquals(out[0], one)
    mut ok := false
    mut back := compress.Bzip2Decompress(out[0].data, out[0].len, ok)
    println "bzip2 round trip", ok, compress.BufEquals(back, text)
    compress.BufFree(back)
    compress.BufFree(one)
    compress.BufFree(out[0])

    out[0] = compress.NewByteBuf()
    mut zs := compress.SinkCallback(0 as i64, emit, out as void*)
    println compress.ZstdCompressFileToSink(fp, 0 as i64, size, 3, zs)
    one = compress.ZstdCompressLevel(text.data, text.len, 3)
    println "zstd same bytes", compress.BufEquals(out[0], one)
    back = compress.ZstdDecompress(out[0].data, out[0].len, ok)
    println "zstd round trip", ok, compress.BufEquals(back, text)
    compress.BufFree(back)
    compress.BufFree(one)
    compress.BufFree(out[0])

    out[0] = compress.NewByteBuf()
    mut lz := compress.SinkCallback(0 as i64, emit, out as void*)
    println compress.LzmaCompressFileToSink(fp, 0 as i64, size, 6, true, lz)
    back = compress.LzmaRawDecompress(out[0].data, 5, ((out[0].data as i64) + (5 as i64)) as void*, out[0].len - 5, text.len, 0, ok)
    println "lzma round trip", ok, compress.BufEquals(back, text)
    compress.BufFree(back)
    compress.BufFree(out[0])

    compress.FileClose(fp)
    mem.Free(out as void*)
    compress.BufFree(text)
    os.Remove("sink.tmp")
end
Outputtrue bzip2 same bytes true bzip2 round trip true true true zstd same bytes true zstd round trip true true true lzma round trip true true

FileOpenRead

func FileOpenRead(path: str): void*
Opens a file for reading bytes. Returns null when the file cannot be opened.

The example for this item is shown with the ones above: compress-Stream.salam

FileOpenWrite

func FileOpenWrite(path: str): void*
Creates a file, or empties an existing one, for writing bytes. Returns null when it cannot be opened.

The example for this item is shown with the ones above: compress-Stream.salam

FileClose

func FileClose(fp: void*)
Closes a file opened with FileOpenRead or FileOpenWrite. A null file is ignored.

The example for this item is shown with the ones above: compress-Stream.salam

FileSeek

func FileSeek(fp: void*, off: i64): bool
Moves to byte off of the file with a 64 bit offset. True when it worked.

The example for this item is shown with the ones above: compress-Stream.salam

FileSize

func FileSize(fp: void*): i64
The size of the file in bytes, or -1 when it cannot be found. The position goes back to the start.

The example for this item is shown with the ones above: compress-Stream.salam

FileReadAt

func FileReadAt(fp: void*, off: i64, dst: void*, n: int): int
Reads up to n bytes starting at byte off of the file into dst and returns how many were read; 0 when the file cannot be positioned. The offset is 64 bit, so files over 2 GB work.

The example for this item is shown with the ones above: compress-Stream.salam

CallbackOpenRead

func CallbackOpenRead(total: i64, read: func(i64, void*, int, void*): int, ctx: void*): void*
Makes a file handle that is read by calling read(off, dst, n, ctx) instead of a file; total is what FileSize reports. FileReadAt on it calls your function, which returns the number of bytes it put in dst (0 at the end). The decoders that read a file (LzmaFileToSink, bzip2, inflate and the rest) accept it, which is how a decrypting stage sits in front of a decompressor. Close it with FileClose; the ctx stays yours.

The example for this item is shown with the ones above: compress-Stream.salam

CallbackContext

func CallbackContext(fp: void*): void*
The ctx given to CallbackOpenRead, or null for any other handle.

The example for this item is shown with the ones above: compress-Stream.salam

MultiOpenRead

func MultiOpenRead(paths: Vector<str>): void*
Opens several files as one for reading: the pieces of a split archive, in order. Pass the result to FileReadAt, FileSize and FileClose like a single file; a read that crosses the end of one piece continues in the next. Returns null when any piece cannot be opened.
compress-LzmaFile.salam
import archive
import compress
import mem
import str

func collect(p: void*, n: int, c: void*): bool:
    compress.BufPushBytes((c as compress.ByteBuf*)[0], p, n)
    ret true
end

func main:
    mut text := compress.NewByteBuf()
    repeat 2000 in i:
        line := "row " + str.FromInt((i * 3) as i64) + "\n"
        compress.BufPushBytes(text, line as void*, line.Len())
    end
    packed := compress.LzmaCompress(text.data, text.len)
    archive.WriteBinaryFileBuf("lz.tmp", packed)

    fp := compress.FileOpenRead("lz.tmp")
    got := mem.AllocateZeroed(sizeof(compress.ByteBuf)) as compress.ByteBuf*
    got[0] = compress.NewByteBuf()
    mut sink := compress.SinkCallback(0 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
    st := compress.LzmaFileToSink(packed.data, 5, fp, 13 as i64, (packed.len - 13) as i64, text.len as i64, sink, 0 as i64)
    compress.FileClose(fp)
    println st, sink.written, compress.BufEquals(got[0], text)

    fp2 := compress.FileOpenRead("lz.tmp")
    mut small := compress.SinkCallback(0 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
    println compress.LzmaFileToSink(packed.data, 5, fp2, 13 as i64, (packed.len - 13) as i64, text.len as i64, small, 100 as i64)
    compress.FileClose(fp2)
    compress.BufFree(got[0])
    mem.Free(got as void*)
end
Output0 17628 true 2

SinkOpen

func SinkOpen(path: str, cap: i64): FileSink
Creates a file and returns a sink for it with a cap of cap bytes (0 for no limit). failed is true when the file cannot be created.

The example for this item is shown with the ones above: compress-Stream.salam

SinkCallback

func SinkCallback(cap: i64, emit: func(void*, int, void*): bool, ctx: void*): FileSink
Makes a sink that passes every piece it is given to emit(p, n, ctx) instead of a file. The piece is counted and checksummed like any other; cap still stops it with over. emit returns false to stop the decoder.

The example for this item is shown with the ones above: compress-LzmaFile.salam

SinkWrite

func SinkWrite(s &: FileSink, p: void*, n: int): bool
Writes n bytes to the sink and adds them to the count and the CRC-32. False once the sink has failed or when the bytes would pass the cap.

The example for this item is shown with the ones above: compress-Stream.salam

SinkClose

func SinkClose(s &: FileSink): bool
Closes the file of a sink. True when no write failed.

The example for this item is shown with the ones above: compress-Stream.salam

SinkCopyFrom

func SinkCopyFrom(s &: FileSink, fp: void*, off: i64, len: i64): bool
Copies len bytes from byte off of an open file into the sink, in pieces. False when the file is too short or the sink refuses the data.

The example for this item is shown with the ones above: compress-Stream.salam

GzipCompress

func GzipCompress(p: void*, n: int): ByteBuf
Compresses the data into the gzip format, the one in .gz files.
compress-Gzip.salam
import compress

func main:
    text := "hello gzip! hello gzip! hello gzip!"
    g := compress.GzipCompressStr(text)
    println compress.BufGet(g, 0), compress.BufGet(g, 1)
    mut ok := false
    b := compress.GzipDecompressBuf(g, ok)
    println ok, compress.BufToStrUnsafe(b) == text
    b2 := compress.BufFromStr(text)
    g2 := compress.GzipCompressBuf(b2)
    g3 := compress.GzipCompress(b2.data, b2.len)
    println compress.BufEquals(g2, g3)
    _ := compress.GzipDecompress(b2.data, b2.len, ok)
    println ok
    d := compress.DeflateCompress(b2.data, b2.len)
    println compress.BufEquals(compress.DeflateDecompress(d.data, d.len), b2)
    raw := compress.RawDeflate(b2.data, b2.len)
    back := compress.RawInflate(raw.data, raw.len, b2.len, ok)
    println ok, compress.BufEquals(back, b2)
    _ := compress.RawInflateMax(raw.data, raw.len, b2.len, 10, ok)
    println ok
    full := compress.RawInflateMax(raw.data, raw.len, b2.len, 1000, ok)
    println ok, compress.BufEquals(full, b2)
    _ := compress.GzipDecompressMax(g.data, g.len, 10, ok)
    println ok
    whole := compress.GzipDecompressMax(g.data, g.len, 1000, ok)
    println ok, compress.BufToStrUnsafe(whole) == text
end
Output31 139 true true true false true true true false true true false true true

GzipDecompress

func GzipDecompress(p: void*, n: int, ok &: bool): ByteBuf
Decompresses gzip data; ok becomes false when the data is not gzip. The checksum inside the file is not checked.

The example for this item is shown with the ones above: compress-Gzip.salam

GzipDecompressMax

func GzipDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBuf
Like GzipDecompress, but stops with ok false when the unpacked data would pass max bytes (0 means no limit); use it on data from outside so a small file cannot fill memory.

The example for this item is shown with the ones above: compress-Gzip.salam

GzipDecompressWhy

func GzipDecompressWhy(p: void*, n: int, max: int, why &: int): ByteBuf
Like GzipDecompressMax, but says why it failed: why is set to DecodeOk, DecodeCorrupt (bad or cut data, wrong checksum) or DecodeTooBig (the unpacked data would pass max). On failure the result is empty.

The example for this item is shown with the ones above: compress-Why.salam

GzipCompressBuf

func GzipCompressBuf(b: ByteBuf): ByteBuf
Gzips a buffer.

The example for this item is shown with the ones above: compress-Gzip.salam

GzipCompressStr

func GzipCompressStr(s: str): ByteBuf
Gzips a string.

The example for this item is shown with the ones above: compress-Gzip.salam

GzipDecompressBuf

func GzipDecompressBuf(b: ByteBuf, ok &: bool): ByteBuf
Decompresses a gzip buffer.

The example for this item is shown with the ones above: compress-Gzip.salam

LZWCompress

func LZWCompress(p: void*, n: int): ByteBuf
LZW compression, with a dictionary built as it goes.

The example for this item is shown with the ones above: compress-Methods.salam

LZWDecompressMax

func LZWDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBuf
Like LZWDecompress but with a cap on the output size (0 means 1 GiB) and an ok flag. Damaged, truncated or oversized input gives ok false and an empty buffer; it never reads or writes out of bounds.

The example for this item is shown with the ones above: compress-Methods.salam

LZWDecompress

func LZWDecompress(p: void*, n: int): ByteBuf
Decompresses LZW data.

The example for this item is shown with the ones above: compress-Methods.salam

NewByteBuf

func NewByteBuf(): ByteBuf
An empty buffer.

The example for this item is shown with the ones above: compress-Buffer.salam

NewByteBufCap

func NewByteBufCap(capacity: int): ByteBuf
An empty buffer with an initial capacity.

The example for this item is shown with the ones above: compress-Buffer.salam

BufPush

func BufPush(b &: ByteBuf, v: int)
Adds one byte.

The example for this item is shown with the ones above: compress-Buffer.salam

BufPushBytes

func BufPushBytes(b &: ByteBuf, p: void*, n: int)
Adds n bytes from memory.

The example for this item is shown with the ones above: compress-Buffer.salam

BufPushBuf

func BufPushBuf(b &: ByteBuf, src: ByteBuf)
Adds the bytes of another buffer.

The example for this item is shown with the ones above: compress-Buffer.salam

BufPushU32

func BufPushU32(b &: ByteBuf, v: int)
Adds a 32 bit integer as 4 little-endian bytes.

The example for this item is shown with the ones above: compress-Buffer.salam

BufGet

func BufGet(b: ByteBuf, i: int): int
Byte number i.

The example for this item is shown with the ones above: compress-Buffer.salam

BufSet

func BufSet(b &: ByteBuf, i: int, v: int)
Sets byte number i.

The example for this item is shown with the ones above: compress-Buffer.salam

BufLen

func BufLen(b: ByteBuf): int
The number of bytes.

The example for this item is shown with the ones above: compress-Buffer.salam

BufFree

func BufFree(b &: ByteBuf)
Frees the memory of the buffer.

The example for this item is shown with the ones above: compress-Buffer.salam

BufReset

func BufReset(b &: ByteBuf)
Empties the buffer and keeps the memory.

The example for this item is shown with the ones above: compress-Buffer.salam

BufClone

func BufClone(b: ByteBuf): ByteBuf
An independent copy of the buffer.

The example for this item is shown with the ones above: compress-Buffer.salam

BufEquals

func BufEquals(a: ByteBuf, b: ByteBuf): bool
True when two buffers hold the same bytes.

The example for this item is shown with the ones above: compress-Buffer.salam

BufFromStr

func BufFromStr(s: str): ByteBuf
Puts the bytes of a string into a new buffer.

The example for this item is shown with the ones above: compress-Buffer.salam

BufToStrUnsafe

func BufToStrUnsafe(b: ByteBuf): str
Takes the buffer to be a string; when the buffer holds a zero byte, the string ends right there.

The example for this item is shown with the ones above: compress-Buffer.salam

BufToBase64

func BufToBase64(b: ByteBuf): str
Turns the buffer into base64, for showing binary data.

The example for this item is shown with the ones above: compress-Buffer.salam

NewByteReader

func NewByteReader(p: void*, n: int): ByteReader
A reader over n bytes of memory.

The example for this item is shown with the ones above: compress-Buffer.salam

ReaderFromBuf

func ReaderFromBuf(b: ByteBuf): ByteReader
A reader over the bytes of a buffer.

The example for this item is shown with the ones above: compress-Buffer.salam

RdLeft

func RdLeft(r: ByteReader): int
The number of bytes not yet read.

The example for this item is shown with the ones above: compress-Buffer.salam

RdByte

func RdByte(r &: ByteReader): int
Reads one byte; past the end bad becomes true.

The example for this item is shown with the ones above: compress-Buffer.salam

RdU32

func RdU32(r &: ByteReader): int
Reads a 4 byte little-endian integer.

The example for this item is shown with the ones above: compress-Buffer.salam

RdBytes

func RdBytes(r &: ByteReader, n: int): ByteBuf
Reads n bytes into a new buffer.

The example for this item is shown with the ones above: compress-Buffer.salam

LzmaCompressLevel

func LzmaCompressLevel(p: void*, n: int, level: int): ByteBuf
Like LzmaCompress with a level from 0 to 9, as for XzCompressLevel.

The example for this item is shown with the ones above: compress-XzWrite.salam

LzmaCompress

func LzmaCompress(p: void*, n: int): ByteBuf
Compresses the data into the old .lzma format at level 6: a 13 byte header with the properties, the dictionary size and the exact unpacked size, then the LZMA stream. The format has no checksum.

The example for this item is shown with the ones above: compress-XzWrite.salam

LzmaCompressBuf

func LzmaCompressBuf(b: ByteBuf): ByteBuf
Compresses a buffer into the .lzma format.

The example for this item is shown with the ones above: compress-XzWrite.salam

LzmaCompressStr

func LzmaCompressStr(s: str): ByteBuf
Compresses a string into the .lzma format.

The example for this item is shown with the ones above: compress-XzWrite.salam

XzCompressCheck

func XzCompressCheck(p: void*, n: int, level: int, check: int): ByteBuf
Like XzCompressLevel, and check is the integrity check stored with the block: XzCheckNone, XzCheckCrc32, XzCheckCrc64 or XzCheckSha256. Any other number gives CRC-64.

The example for this item is shown with the ones above: compress-XzWrite.salam

XzCompressLevel

func XzCompressLevel(p: void*, n: int, level: int): ByteBuf
Like XzCompress with a level from 0 (fastest) to 9 (smallest); a number outside that range is moved to the nearest end. The level sets the window (256 KiB at 0, 8 MiB at 6, 64 MiB at 9, never bigger than the data) and how hard the match finder searches.

The example for this item is shown with the ones above: compress-XzWrite.salam

XzCompress

func XzCompress(p: void*, n: int): ByteBuf
Compresses the data into a standard .xz file at level 6 with a CRC-64 check, the way xz does by default. Empty input gives a valid stream with no block.

The example for this item is shown with the ones above: compress-XzWrite.salam

XzCompressBuf

func XzCompressBuf(b: ByteBuf): ByteBuf
Compresses a buffer into an xz file.

The example for this item is shown with the ones above: compress-XzWrite.salam

XzCompressStr

func XzCompressStr(s: str): ByteBuf
Compresses a string into an xz file.

The example for this item is shown with the ones above: compress-XzWrite.salam

LzmaCompressFileToSink

func LzmaCompressFileToSink(fp: void*, off: i64, len: i64, level: int, with_props: bool, sink &: FileSink): bool
Compresses len bytes of the open file fp from off into one bare LZMA stream (no size field, no end marker) and writes it to a sink as it goes. With with_props the stream starts with the 5 property bytes that LzmaFileToSink wants. Only a window of about twice the dictionary is held in memory. The output can differ from LzmaCompressLevel by a fraction of a percent because the matcher sees the data in pieces.

The example for this item is shown with the ones above: compress-CompressSink.salam

RLECompress

func RLECompress(p: void*, n: int): ByteBuf
Run length encoding: shortens runs of the same byte; for data that repeats a lot.

The example for this item is shown with the ones above: compress-Methods.salam

RLEDecompressMax

func RLEDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBuf
Like RLEDecompress but with a cap on the output size (0 means 1 GiB) and an ok flag. Damaged, truncated or oversized input gives ok false and an empty buffer; it never reads or writes out of bounds.

The example for this item is shown with the ones above: compress-Methods.salam

RLEDecompress

func RLEDecompress(p: void*, n: int): ByteBuf
Decompresses RLE data.

The example for this item is shown with the ones above: compress-Methods.salam

RawInflateToSink

func RawInflateToSink(r &: BitReader, sink &: FileSink, d64: bool): bool
Decompresses raw DEFLATE from a bit reader straight into a sink (Deflate64 when d64 is true). False when the data is broken or the sink refuses it because of its cap or a failed write.

The example for this item is shown with the ones above: compress-Stream.salam

RawInflate

func RawInflate(p: void*, n: int, expected_size: int, ok &: bool): ByteBuf
Decompresses raw DEFLATE; expected_size is the rough size of the output.

The example for this item is shown with the ones above: compress-Gzip.salam

RawInflateMax

func RawInflateMax(p: void*, n: int, expected_size: int, max: int, ok &: bool): ByteBuf
Same as RawInflate, but stops with ok false once the output would pass max bytes, so a small hostile stream cannot expand without bound. max of 0 means no limit.

The example for this item is shown with the ones above: compress-Gzip.salam

RawInflate64Max

func RawInflate64Max(p: void*, n: int, expected_size: int, max: int, ok &: bool): ByteBuf
Same as RawInflateMax for Deflate64, the DEFLATE variant of 7-Zip and Zip with a 64 KiB window, 16 bit long matches and two more distance codes.

The example for this item is shown with the ones above: compress-Gzip.salam

ZstdStatusText

func ZstdStatusText(code: int): str
A short English description of a status code, such as "checksum mismatch".

The example for this item is shown with the ones above: compress-Zstd.salam

IsZstd

func IsZstd(p: void*, n: int): bool
True when the data starts with the Zstandard magic number (28 B5 2F FD) or the magic number of a skippable frame. It reads four bytes and checks nothing else.

The example for this item is shown with the ones above: compress-Zstd.salam

IsZstdBuf

func IsZstdBuf(b: ByteBuf): bool
IsZstd on a byte buffer.

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdDecompressEx

func ZstdDecompressEx(p: void*, n: int, max: int, dict: void*, dn: int, status &: int): ByteBuf
The full decoder. It decodes every frame in the data one after the other, passes over skippable frames, and verifies the XXH64 checksum of each frame that has one. max is the largest output accepted (0 means just under 1 GiB). dict and dn give an optional dictionary (null and 0 for none), either a trained one or plain content. status receives one of the ZstdOk and ZstdErr... codes, and the result is an empty buffer on any failure.

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdDecompressMax

func ZstdDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBuf
Like ZstdDecompress, but stops with ok false when the unpacked data would pass max bytes (0 means no limit). Use it on data from outside: a few bytes of Zstandard can describe gigabytes.

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdDecompress

func ZstdDecompress(p: void*, n: int, ok &: bool): ByteBuf
Decompresses Zstandard data, the content of a .zst file: one frame or several in a row, with skippable frames passed over. It handles every block type, including Huffman and FSE compressed ones, repeated offsets and the XXH64 checksum when the frame carries one. ok is false for damaged, truncated or unsupported data, and the result is then empty. A frame that needs a dictionary fails here; use ZstdDecompressDict.

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdDecompressBuf

func ZstdDecompressBuf(b: ByteBuf, ok &: bool): ByteBuf
Decompresses a Zstandard buffer.

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdDecompressDict

func ZstdDecompressDict(p: void*, n: int, dict: void*, dn: int, ok &: bool): ByteBuf
Decompresses data that was made with a dictionary. The dictionary can be a trained one, the file zstd --train writes, or any bytes, which then serve as raw content. A frame that names a dictionary ID needs the dictionary with that ID.
compress-ZstdDict.salam
import compress

const FRAME := """
28b52ffd645900ed010092430c11a0ed7425bfff01a825dda80e11a94a1c014020bd51a7cf6fcc41d92e1f9a01043a9c
4edf68a4ea1867b8bed41b6e2a240200f7874039c1d515ee20b3a1
"""

func nib(c: int): int:
    if c >= 48 and c <= 57:
        ret c - 48
    end
    if c >= 97 and c <= 102:
        ret c - 87
    end
    ret -1
end

func unhex(s: str): compress.ByteBuf:
    mut b := compress.NewByteBuf()
    bp := s as u8*
    mut hi := -1
    repeat s.Len() in i:
        v := nib(bp[i] as int)
        if v < 0:
            continue
        end
        if hi < 0:
            hi = v
        else:
            compress.BufPush(b, hi * 16 + v)
            hi = -1
        end
    end
    ret b
end

func main:
    frame := unhex(FRAME)
    dict := compress.BufFromStr("Salam is a programming language. ")
    mut status := 0
    without := compress.ZstdDecompressEx(frame.data, frame.len, 0, null, 0, status)
    println "without the dictionary:", status, compress.ZstdStatusText(status), without.len
    mut ok := false
    text := compress.ZstdDecompressDict(frame.data, frame.len, dict.data, dict.len, ok)
    println "with the dictionary:", ok, text.len, "bytes from", frame.len
    println compress.BufToStrUnsafe(text)
    other := compress.BufFromStr("some other bytes")
    wrong := compress.ZstdDecompressEx(frame.data, frame.len, 0, other.data, other.len, status)
    println "with a different dictionary:", status, wrong.len
end
Outputwithout the dictionary: 3 corrupt data 0 with the dictionary: true 345 bytes from 75 Salam is a programming language. This text is packed with zstd. Salam is a programming language. This text is packed with zstd. Salam is a programming language. This text is packed with zstd. Salam is a programming language. This text is packed with zstd. Salam is a programming language. This text is packed with zstd. Last line, not repeated. with a different dictionary: 3 0

ZstdCompressOpts

func ZstdCompressOpts(p: void*, n: int, level: int, checksum: bool): ByteBuf
Compresses to one Zstandard frame at level, with or without the XXH64 content checksum. The frame records the unpacked size.

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdCompressLevel

func ZstdCompressLevel(p: void*, n: int, level: int): ByteBuf
Compresses at a level, with the checksum. Level 0 or less stores the data without compressing it; 1 to 22 search harder for repeats as the number grows, and from 10 up the window is larger. The result is a standard frame that the zstd tool and every other reader accepts. The writer is simple (a greedy or lazy search and the predefined entropy tables), so its output is larger than the reference tool's at the same level, but it is valid everywhere.

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdCompress

func ZstdCompress(p: void*, n: int): ByteBuf
Compresses at level 3, with the checksum.

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdCompressBuf

func ZstdCompressBuf(b: ByteBuf): ByteBuf
Compresses a buffer at level 3.

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdCompressStr

func ZstdCompressStr(s: str): ByteBuf
Compresses a string at level 3.

The example for this item is shown with the ones above: compress-Zstd.salam

ZstdContentSize

func ZstdContentSize(p: void*, n: int): i64
The unpacked size written in the header of the first frame, passing over skippable frames before it. It returns -1 when the frame does not record a size (as when it was written from a pipe) and -2 when the data is not a Zstandard frame or the header is cut short. It is only what the header claims, so do not use it alone to size a buffer for data from outside.

The example for this item is shown with the ones above: compress-Zstd.salam

NewBitWriter

func NewBitWriter(): BitWriter
A new bit writer.

The example for this item is shown with the ones above: compress-Bits.salam

BitWriteBit

func BitWriteBit(w &: BitWriter, bit: int)
Writes one bit.

The example for this item is shown with the ones above: compress-Bits.salam

BitWriteBits

func BitWriteBits(w &: BitWriter, value: int, count: int)
Writes the low count bits of value.

The example for this item is shown with the ones above: compress-Bits.salam

BitFlush

func BitFlush(w &: BitWriter)
Sends the remaining bits to the last byte, padded with zeros.

The example for this item is shown with the ones above: compress-Bits.salam

BitWriterLen

func BitWriterLen(w &: BitWriter): int
The number of bytes written.

The example for this item is shown with the ones above: compress-Bits.salam

BitWriterFree

func BitWriterFree(w &: BitWriter)
Frees the memory.

The example for this item is shown with the ones above: compress-Bits.salam

NewBitReader

func NewBitReader(p: void*, n: int): BitReader
A bit reader over memory.

The example for this item is shown with the ones above: compress-Bits.salam

NewFileBitReader

func NewFileBitReader(fp: void*, off: i64, len: i64): BitReader
A bit reader over len bytes of a file, starting at byte off, that reads the file in pieces of 64 KiB. bad is true when the file cannot be positioned. Free it with FileBitReaderFree.

The example for this item is shown with the ones above: compress-Stream.salam

FileBitReaderFree

func FileBitReaderFree(r &: BitReader)
Frees the read buffer of a bit reader made by NewFileBitReader. It does not close the file.

The example for this item is shown with the ones above: compress-Stream.salam

BitReaderFromBuf

func BitReaderFromBuf(b: ByteBuf): BitReader
A bit reader over a buffer.
compress-Fields.salam
import compress

func main:
    mut b := compress.NewByteBuf()
    compress.BufPushBytes(b, "abc" as void*, 3)
    println b.len, b.cap >= b.len, b.data != null
    mut r := compress.ReaderFromBuf(b)
    println r.len, r.pos, r.bad, r.data == b.data
    _ := compress.RdByte(r)
    println r.pos
    _ := compress.RdBytes(r, 5)
    println r.bad

    mut w := compress.NewBitWriter()
    compress.BitWriteBits(w, 5, 3)
    println w.nbits, w.acc, w.buf.len
    compress.BitFlush(w)
    println w.nbits, w.buf.len
    mut br := compress.BitReaderFromBuf(w.buf)
    println br.len, br.pos, br.nbits, br.acc, br.bad, br.data == w.buf.data
    println compress.BitReadBits(br, 3), br.nbits, br.pos
    _ := compress.BitReadBits(br, 16)
    println br.bad
    compress.BitWriterFree(w)
    compress.BufFree(b)

    methods := [compress.MethodNone, compress.MethodRLE, compress.MethodHuffman, compress.MethodLZ77, compress.MethodLZW, compress.MethodDeflate, compress.MethodBWT]
    each m in methods:
        out := compress.CompressStr(m, "aaaaabbbbbaaaaabbbbb")
        println m, compress.BufToStrUnsafe(compress.Decompress(m, out.data, out.len))
    end
end
Output3 true true 3 0 false true 1 true 3 5 0 0 1 1 0 0 0 false true 5 5 1 true 0 aaaaabbbbbaaaaabbbbb 1 aaaaabbbbbaaaaabbbbb 2 aaaaabbbbbaaaaabbbbb 3 aaaaabbbbbaaaaabbbbb 4 aaaaabbbbbaaaaabbbbb 5 aaaaabbbbbaaaaabbbbb 6 aaaaabbbbbaaaaabbbbb

BitReadBit

func BitReadBit(r &: BitReader): int
Reads one bit.

The example for this item is shown with the ones above: compress-Bits.salam

BitReadBits

func BitReadBits(r &: BitReader, count: int): int
Reads count bits.

The example for this item is shown with the ones above: compress-Bits.salam

ZstdFileToSink

func ZstdFileToSink(fp: void*, off: i64, len: i64, sink &: FileSink, memcap: i64): int
Decodes a zstd file of len bytes read from fp at off into a sink block by block, with a sliding window: memory is the frame's window size (from its header) plus a few hundred kilobytes, however big the output is. A frame whose window is over memcap (0 means 1 GiB) is refused with ZstdErrTooBig, and so is one whose declared size or output passes the sink's own cap. The content checksum is verified, several frames and skippable frames are accepted, and a frame that needs a dictionary gives ZstdErrDict. When the sink itself fails, sink.failed is set.
compress-ZstdFile.salam
import archive
import compress
import mem
import os
import str

func collect(p: void*, n: int, c: void*): bool:
    compress.BufPushBytes((c as compress.ByteBuf*)[0], p, n)
    ret true
end

func main:
    mut text := compress.NewByteBuf()
    repeat 5000 in i:
        line := "line " + str.FromInt((i * 7) as i64) + "\n"
        compress.BufPushBytes(text, line as void*, line.Len())
    end
    mut packed := compress.ZstdCompress(text.data, text.len)
    archive.WriteBinaryFileBuf("zst.tmp", packed)

    fp := compress.FileOpenRead("zst.tmp")
    got := mem.AllocateZeroed(sizeof(compress.ByteBuf)) as compress.ByteBuf*
    got[0] = compress.NewByteBuf()
    mut sink := compress.SinkCallback(0 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
    st := compress.ZstdFileToSink(fp, 0 as i64, compress.FileSize(fp), sink, 0 as i64)
    println st == compress.ZstdOk, sink.written == (text.len as i64), compress.BufEquals(got[0], text)
    compress.FileClose(fp)

    fp2 := compress.FileOpenRead("zst.tmp")
    mut small := compress.SinkCallback(100 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
    println compress.ZstdFileToSink(fp2, 0 as i64, compress.FileSize(fp2), small, 0 as i64) == compress.ZstdErrTooBig
    compress.FileClose(fp2)
    compress.BufFree(got[0])
    mem.Free(got as void*)
    compress.BufFree(text)
    compress.BufFree(packed)
    os.Remove("zst.tmp")
end
Outputtrue true true true

BcjStep

func BcjStep(kind: int, p: void*, n: int, pos: i64, state &: int): int
Runs one step of a branch filter decoder (BcjX86, BcjPowerPC, BcjIA64, BcjARM, BcjARMThumb, BcjSPARC, BcjARM64 or BcjRISCV) over n bytes in place. pos is the position of the first byte in the whole stream and state carries the x86 history between calls. It returns how many bytes are final; hold the rest back and give them again at the front of the next piece. A step can be run on any piece size and gives the same bytes as one call over the whole data. -1 for an unknown kind.

The example for this item is shown with the ones above: compress-Stream.salam

BcjDecode

func BcjDecode(kind: int, p: void*, n: int, start: int): bool
Undoes a branch filter in place on n bytes. kind is one of the Bcj... numbers, and start the address the first byte had (0 for xz and 7z). Returns false for an unknown kind.

The example for this item is shown with the ones above: compress-XzRaw.salam

DeltaDecode

func DeltaDecode(p: void*, n: int, dist: int): bool
Undoes the delta filter in place; dist is the distance, 1 to 256. Returns false for any other distance.

The example for this item is shown with the ones above: compress-XzRaw.salam

SwapDecode

func SwapDecode(p: void*, n: int, width: int): bool
Undoes the byte swap filter in place: width 2 or 4 reverses every group of that many bytes; a trailing partial group stays as it is. Returns false for any other width.

The example for this item is shown with the ones above: compress-XzRaw.salam

Compress

func Compress(method: int, p: void*, n: int): ByteBuf
Compresses the data with the given method. The output does not carry the method number, so you have to know it to decompress.

The example for this item is shown with the ones above: compress-Methods.salam

DecompressMax

func DecompressMax(method: int, p: void*, n: int, max: int, ok &: bool): ByteBuf
Like Decompress with a cap on the output size (0 means 1 GiB) and an ok flag; bad input gives ok false and an empty buffer.

The example for this item is shown with the ones above: compress-Methods.salam

Decompress

func Decompress(method: int, p: void*, n: int): ByteBuf
Decompresses data that was compressed with the given method.

The example for this item is shown with the ones above: compress-Methods.salam

CompressStr

func CompressStr(method: int, s: str): ByteBuf
Compresses a string.

The example for this item is shown with the ones above: compress-Methods.salam

CompressBuf

func CompressBuf(method: int, b: ByteBuf): ByteBuf
Compresses a buffer.

The example for this item is shown with the ones above: compress-Methods.salam

DecompressBuf

func DecompressBuf(method: int, b: ByteBuf): ByteBuf
Decompresses a compressed buffer.

The example for this item is shown with the ones above: compress-Methods.salam

CompressBest

func CompressBest(p: void*, n: int, method_out &: int): ByteBuf
Tries every method, returns the shortest output and puts its method in method_out.

The example for this item is shown with the ones above: compress-Methods.salam

Ratio

func Ratio(compressed_len: int, original_len: int): f64
The ratio of the compressed size to the original; under 1 means it got smaller.

The example for this item is shown with the ones above: compress-Methods.salam

CRC32Update

func CRC32Update(state: u32, p: void*, n: int): u32
Continues a CRC-32 over more data. Pass the value returned for the earlier part (0 for the first part); the result equals CRC32 of all the parts joined, so a big file can be checksummed piece by piece.

The example for this item is shown with the ones above: compress-Stream.salam

CRC32

func CRC32(p: void*, n: int): u32
The CRC-32 checksum, the one in ZIP and gzip, for checking that data is intact.
compress-Checksum.salam
import compress

func main:
    println compress.CRC32Str("123456789"), compress.Adler32Str("Wikipedia")
    b := compress.BufFromStr("123456789")
    println compress.CRC32(b.data, b.len) == compress.CRC32Buf(b)
    println compress.Adler32(b.data, b.len) == compress.Adler32Buf(b)
    println compress.CRC32Str("hello")
end
Output3421780262 300286872 true true 907060870

CRC32Str

func CRC32Str(s: str): u32
The CRC-32 of a string.

The example for this item is shown with the ones above: compress-Checksum.salam

CRC32Buf

func CRC32Buf(b: ByteBuf): u32
The CRC-32 of a buffer.

The example for this item is shown with the ones above: compress-Checksum.salam

Adler32

func Adler32(p: void*, n: int): u32
The Adler-32 checksum, the one in zlib; faster than CRC-32 but a little weaker.

The example for this item is shown with the ones above: compress-Checksum.salam

Adler32Str

func Adler32Str(s: str): u32
The Adler-32 of a string.

The example for this item is shown with the ones above: compress-Checksum.salam

Adler32Buf

func Adler32Buf(b: ByteBuf): u32
The Adler-32 of a buffer.

The example for this item is shown with the ones above: compress-Checksum.salam

XXH64

func XXH64(p: void*, n: int, seed: u64): u64
The 64-bit xxHash of n bytes with a seed. A Zstandard frame uses the low 32 bits of the seed 0 value as its content checksum.
compress-XXH64.salam
import compress
import conv

func main:
    println conv.FormatHex(compress.XXH64Str(""))
    println conv.FormatHex(compress.XXH64Str("abc"))
    text := compress.BufFromStr("Salam")
    println conv.FormatHex(compress.XXH64Buf(text))
    println conv.FormatHex(compress.XXH64(text.data, text.len, 0 as u64))
    println conv.FormatHex(compress.XXH64(text.data, text.len, 1 as u64))
end
Outputef46db3751d8e999 44bc2cf5ad770999 4c3f8722aa877c20 4c3f8722aa877c20 ece1ed8ee82f809f

XXH64Str

func XXH64Str(s: str): u64
The xxHash64 of a string, with seed 0.

The example for this item is shown with the ones above: compress-XXH64.salam

XXH64Buf

func XXH64Buf(b: ByteBuf): u64
The xxHash64 of a buffer, with seed 0.

The example for this item is shown with the ones above: compress-XXH64.salam

DeflateCompress

func DeflateCompress(p: void*, n: int): ByteBuf
LZ77 and Huffman in a format of Salam's own. This is not standard zlib DEFLATE; for the standard format use RawDeflate or gzip.

The example for this item is shown with the ones above: compress-Gzip.salam

DeflateDecompressMax

func DeflateDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBuf
Like DeflateDecompress but with a cap on the output size (0 means 1 GiB) and an ok flag. Damaged, truncated or oversized input gives ok false and an empty buffer; it never reads or writes out of bounds.

The example for this item is shown with the ones above: compress-Methods.salam

DeflateDecompress

func DeflateDecompress(p: void*, n: int): ByteBuf
Decompresses the output of DeflateCompress.

The example for this item is shown with the ones above: compress-Gzip.salam

RawDeflateTo

func RawDeflateTo(w &: BitWriter, p: void*, n: int, last: bool)
Like RawDeflate, but writes into a BitWriter you keep, so a long input can be given in pieces: every call but the last passes last false, the last passes true, and call BitFlush at the end. Each piece is compressed on its own, without matches into earlier pieces.

The example for this item is shown with the ones above: compress-Gzip.salam

RawDeflate

func RawDeflate(p: void*, n: int): ByteBuf
Standard DEFLATE (RFC 1951) with no header; what sits inside ZIP and gzip.

The example for this item is shown with the ones above: compress-Gzip.salam

HuffmanCompress

func HuffmanCompress(p: void*, n: int): ByteBuf
Huffman coding: the more frequent bytes get shorter codes. The code table is stored with the data, so for small data the output gets bigger.

The example for this item is shown with the ones above: compress-Methods.salam

HuffmanDecompressMax

func HuffmanDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBuf
Like HuffmanDecompress but with a cap on the output size (0 means 1 GiB) and an ok flag. Damaged, truncated or oversized input gives ok false and an empty buffer; it never reads or writes out of bounds.

The example for this item is shown with the ones above: compress-Methods.salam

HuffmanDecompress

func HuffmanDecompress(p: void*, n: int): ByteBuf
Decompresses Huffman data.

The example for this item is shown with the ones above: compress-Methods.salam

XzDecompress

func XzDecompress(p: void*, n: int, ok &: bool): ByteBuf
Decompresses the bytes of a .xz file: the stream header and footer, every block with its filters (LZMA2, the branch filters for x86, ARM, ARM64, PowerPC, SPARC and IA-64, and delta), the index and the check (CRC-32, CRC-64 or SHA-256, whichever the file declares). Streams written one after the other are joined. ok is false for damaged, truncated or unsupported input, and the result is then empty. The result is capped at just under 1 GiB; use XzDecompressMax on data from outside to set a smaller cap.
compress-Xz.salam
import compress

const XZ := """
fd377a585a000004e6d6b44604c05fcf022101160000000000000000bb0ccf6ce0014e00575d0029984986240afb8a34e4
81db780958f3d0144436dd3bb4ca45234051fa042e5be576402f66b72801fb0f3b421e32e3eba071037cdd425338444ede
4cb8c8c742828bc92c1b53df8ea763015d902c3d49f9c602aacf40a000005b35fff6fd65c4e300017bcf020000005e9efc
63b1c467fb020000000004595a
"""

const LZMA := """
5d00008000ffffffffffffffff0029984986240afb8a34e481db780958f3d0144436dd3bb4ca45234051fa042e5be57640
2f66b72801fb0f3b421e32e3eba071037cdd425338444ede4cb8c8c742828bc92c1b53df8ea763015d902c3d49f9c602ad
8d68e0bcfed44dc0
"""

func nib(c: int): int:
    if c >= 48 and c <= 57:
        ret c - 48
    end
    if c >= 97 and c <= 102:
        ret c - 87
    end
    ret -1
end

func unhex(s: str): compress.ByteBuf:
    mut b := compress.NewByteBuf()
    bp := s as u8*
    mut hi := -1
    repeat s.Len() in i:
        v := nib(bp[i] as int)
        if v < 0:
            continue
        end
        if hi < 0:
            hi = v
        else:
            compress.BufPush(b, hi * 16 + v)
            hi = -1
        end
    end
    ret b
end

func main:
    xz := unhex(XZ)
    lz := unhex(LZMA)
    mut ok := false
    a := compress.XzDecompress(xz.data, xz.len, ok)
    println ".xz:", ok, a.len
    b := compress.XzDecompressBuf(xz, ok)
    println ".xz buffer:", ok, compress.BufEquals(a, b)
    c := compress.XzDecompressMax(xz.data, xz.len, a.len, ok)
    println ".xz with max:", ok, c.len
    d := compress.XzDecompressMax(xz.data, xz.len, a.len - 1, ok)
    println ".xz one byte over max:", ok, d.len
    e := compress.LzmaDecompress(lz.data, lz.len, ok)
    println ".lzma:", ok, compress.BufEquals(a, e)
    f := compress.LzmaDecompressBuf(lz, ok)
    println ".lzma buffer:", ok, f.len
    g := compress.LzmaDecompressMax(lz.data, lz.len, 50, ok)
    println ".lzma with max 50:", ok, g.len
end
Output.xz: true 335 .xz buffer: true true .xz with max: true 335 .xz one byte over max: false 0 .lzma: true true .lzma buffer: true 335 .lzma with max 50: false 0

XzDecompressMax

func XzDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBuf
Like XzDecompress, but stops with ok false when the unpacked data would pass max bytes (0 means no limit), so a tiny file cannot expand without bound.

The example for this item is shown with the ones above: compress-Xz.salam

XzDecompressWhy

func XzDecompressWhy(p: void*, n: int, max: int, why &: int): ByteBuf
Like XzDecompressMax, but sets why to DecodeOk, DecodeCorrupt or DecodeTooBig.

The example for this item is shown with the ones above: compress-Why.salam

XzDecompressBuf

func XzDecompressBuf(b: ByteBuf, ok &: bool): ByteBuf
Decompresses an xz buffer.

The example for this item is shown with the ones above: compress-Xz.salam

Bzip2CompressLevel

func Bzip2CompressLevel(p: void*, n: int, level: int): ByteBuf
Like Bzip2Compress with a level from 1 to 9, the block size in units of 100000 bytes; a number outside that range is moved to the nearest end. A lower level uses less memory.
compress-Bzip2.salam
import compress

func main:
    text := "hello bzip2! hello bzip2! hello bzip2! hello bzip2!"
    z := compress.Bzip2CompressStr(text)
    println compress.BufGet(z, 0), compress.BufGet(z, 1), compress.BufGet(z, 2), compress.BufGet(z, 3)
    mut ok := false
    b := compress.Bzip2DecompressBuf(z, ok)
    println ok, compress.BufToStrUnsafe(b) == text
    src := compress.BufFromStr(text)
    z2 := compress.Bzip2CompressBuf(src)
    z3 := compress.Bzip2Compress(src.data, src.len)
    println compress.BufEquals(z2, z3)
    z4 := compress.Bzip2CompressLevel(src.data, src.len, 1)
    println compress.BufGet(z4, 3)
    back := compress.Bzip2Decompress(z4.data, z4.len, ok)
    println ok, compress.BufEquals(back, src)
    _ := compress.Bzip2Decompress(src.data, src.len, ok)
    println ok
    _ := compress.Bzip2DecompressMax(z.data, z.len, 10, ok)
    println ok
    whole := compress.Bzip2DecompressMax(z.data, z.len, 1000, ok)
    println ok, compress.BufToStrUnsafe(whole) == text
end
Output66 90 104 57 true true true 49 true true false false true true

Bzip2Compress

func Bzip2Compress(p: void*, n: int): ByteBuf
Compresses the data into a standard bzip2 stream (.bz2) with the biggest block size, level 9.

The example for this item is shown with the ones above: compress-Bzip2.salam

Bzip2CompressBuf

func Bzip2CompressBuf(b: ByteBuf): ByteBuf
Compresses a buffer with bzip2.

The example for this item is shown with the ones above: compress-Bzip2.salam

Bzip2CompressStr

func Bzip2CompressStr(s: str): ByteBuf
Compresses a string with bzip2.

The example for this item is shown with the ones above: compress-Bzip2.salam

Bzip2Decompress

func Bzip2Decompress(p: void*, n: int, ok &: bool): ByteBuf
Decompresses bzip2 data made of one or more streams; ok becomes false when the data is not valid bzip2, when a block or stream CRC does not match or when anything follows the last stream. A failure returns an empty buffer.

The example for this item is shown with the ones above: compress-Bzip2.salam

Bzip2DecompressMax

func Bzip2DecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBuf
Like Bzip2Decompress, but stops with ok false when the unpacked data would pass max bytes (0 means no limit); use it on data from outside so a small file cannot fill memory.

The example for this item is shown with the ones above: compress-Bzip2.salam

Bzip2DecompressWhy

func Bzip2DecompressWhy(p: void*, n: int, max: int, why &: int): ByteBuf
Like Bzip2DecompressMax, but sets why to DecodeOk, DecodeCorrupt or DecodeTooBig, so a stream over the limit is told apart from a damaged one.

The example for this item is shown with the ones above: compress-Why.salam

Bzip2DecompressBuf

func Bzip2DecompressBuf(b: ByteBuf, ok &: bool): ByteBuf
Decompresses a bzip2 buffer.

The example for this item is shown with the ones above: compress-Bzip2.salam

Bzip2FileToSink

func Bzip2FileToSink(fp: void*, off: i64, len: i64, sink &: FileSink): int
Decodes a bzip2 stream of len bytes read from fp at off into a sink, one block at a time, so memory stays at the block (at most 900 KB of table data plus 1 MiB of output) whatever the size of the file. Several streams in a row are accepted. Every block's CRC and the stream CRC are checked. The result is DecodeOk, DecodeCorrupt, DecodeTooBig (the sink's own cap was passed) or DecodeIo (the sink could not take the data).

The example for this item is shown with the ones above: compress-Bzip2File.salam

Bzip2CompressFileToSink

func Bzip2CompressFileToSink(fp: void*, off: i64, len: i64, level: int, sink &: FileSink): bool
Compresses len bytes of the open file fp from off into one bzip2 stream and writes it to a sink block by block, so memory is one block (about 18 MB of working tables at level 9) plus 256 KiB of input, whatever the size of the file. level is 1 to 9. The bytes are the same as Bzip2CompressLevel would give. Returns false when the file cannot be read or the sink refuses.

The example for this item is shown with the ones above: compress-CompressSink.salam

XzFileToSink

func XzFileToSink(fp: void*, off: i64, len: i64, sink &: FileSink, memcap: i64): int
Decodes an xz file of len bytes read from fp at off into a sink without holding it in memory. The index at the end of each stream is read first, so the unpacked size is known (and checked against the sink's cap) before anything is written; then every block is unpacked through a window the size of its dictionary, and its CRC-32, CRC-64 or SHA-256 is checked. Several streams in a row and stream padding are accepted. A file that uses a Delta or BCJ filter is read whole and decoded in memory instead (up to 2 GB). memcap limits the window (0 for none). The result is DecodeOk, DecodeCorrupt, DecodeTooBig or DecodeIo.
compress-XzFile.salam
import archive
import compress
import mem
import os
import str

func collect(p: void*, n: int, c: void*): bool:
    compress.BufPushBytes((c as compress.ByteBuf*)[0], p, n)
    ret true
end

func main:
    mut text := compress.NewByteBuf()
    repeat 5000 in i:
        line := "line " + str.FromInt((i * 7) as i64) + "\n"
        compress.BufPushBytes(text, line as void*, line.Len())
    end
    mut packed := compress.XzCompress(text.data, text.len)
    archive.WriteBinaryFileBuf("xz.tmp", packed)

    fp := compress.FileOpenRead("xz.tmp")
    got := mem.AllocateZeroed(sizeof(compress.ByteBuf)) as compress.ByteBuf*
    got[0] = compress.NewByteBuf()
    mut sink := compress.SinkCallback(0 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
    st := compress.XzFileToSink(fp, 0 as i64, compress.FileSize(fp), sink, 0 as i64)
    println st == compress.DecodeOk, sink.written == (text.len as i64), compress.BufEquals(got[0], text)
    compress.FileClose(fp)

    fp2 := compress.FileOpenRead("xz.tmp")
    mut small := compress.SinkCallback(100 as i64, (p: void*, n: int, c: void*): ret collect(p, n, c) end, got as void*)
    println compress.XzFileToSink(fp2, 0 as i64, compress.FileSize(fp2), small, 0 as i64) == compress.DecodeTooBig
    compress.FileClose(fp2)
    compress.BufFree(got[0])
    mem.Free(got as void*)
    compress.BufFree(text)
    compress.BufFree(packed)
    os.Remove("xz.tmp")
end
Outputtrue true true true

Lzma2RawDecompress

func Lzma2RawDecompress(dict_prop: int, p: void*, n: int, out_size: int, max: int, ok &: bool): ByteBuf
Decompresses a bare LZMA2 stream, as xz blocks and 7z hold it. dict_prop is the one byte dictionary property (0 to 40). out_size is the exact unpacked size, or -1 when it is not known; max of 0 means no limit.

The example for this item is shown with the ones above: compress-XzRaw.salam

LzmaRawDecompress

func LzmaRawDecompress(props: void*, plen: int, p: void*, n: int, out_size: int, max: int, ok &: bool): ByteBuf
Decompresses a bare LZMA stream with no header, the way 7z stores it. props is the 5 byte coder property (one byte for lc, lp and pb, then the dictionary size, little-endian), and out_size the exact unpacked size, which the stream must produce. max of 0 means no limit.

The example for this item is shown with the ones above: compress-XzRaw.salam

LzmaDecompress

func LzmaDecompress(p: void*, n: int, ok &: bool): ByteBuf
Decompresses the old .lzma format (what xz --format=lzma writes), whose 13 byte header holds the properties, the dictionary size and the unpacked size, or "unknown" followed by an end marker. The format has no checksum, so only the structure is checked. Dictionary sizes above LzmaMaxDictSize are refused.

The example for this item is shown with the ones above: compress-Xz.salam

LzmaDecompressMax

func LzmaDecompressMax(p: void*, n: int, max: int, ok &: bool): ByteBuf
Like LzmaDecompress, but stops with ok false when the unpacked data would pass max bytes (0 means no limit).

The example for this item is shown with the ones above: compress-Xz.salam

LzmaDecompressWhy

func LzmaDecompressWhy(p: void*, n: int, max: int, why &: int): ByteBuf
Like LzmaDecompressMax, but sets why to DecodeOk, DecodeCorrupt or DecodeTooBig.

The example for this item is shown with the ones above: compress-Why.salam

LzmaDecompressBuf

func LzmaDecompressBuf(b: ByteBuf, ok &: bool): ByteBuf
Decompresses an lzma buffer.

The example for this item is shown with the ones above: compress-Xz.salam

LzmaFileToSink

func LzmaFileToSink(props: void*, plen: int, fp: void*, off: i64, len: i64, out_size: i64, sink &: FileSink, memcap: i64): int
Decodes a bare LZMA stream (the 5 byte props, then the data from off for len bytes of the open file fp) straight into a sink. Only a sliding window the size of the dictionary is kept, never the whole result, so the unpacked size can pass 2 GiB. out_size is the exact unpacked size. memcap is the most memory the window may take (0 for no limit). Returns 0 on success, 1 for bad data, 2 when the window would be over memcap or over 2 GiB, and 3 when the sink refused data.

The example for this item is shown with the ones above: compress-LzmaFile.salam

Lzma2FileToSink

func Lzma2FileToSink(dict_prop: int, fp: void*, off: i64, len: i64, out_size: i64, sink &: FileSink, memcap: i64): int
Like LzmaFileToSink for a bare LZMA2 stream; dict_prop is the one byte dictionary property (0 to 40). The window is the dictionary size, or the unpacked size when that is smaller.

The example for this item is shown with the ones above: compress-LzmaFile.salam