fastports

simdutf 8.2.0

converters/simdutf

Unicode validation and transcoding with SIMD

Category
converters
Maintainer
fuz@FreeBSD.org
WWW
https://simdutf.github.io/simdutf/
License
APACHE20 MIT
USES
cmake:testing compiler:c++17-lang python:build,test iconv

Description

This library provide fast Unicode functions such as

 - ASCII, UTF-8, UTF-16LE/BE and UTF-32 validation, with and without
   error identification,
 - transcoding between each of Latin1, UTF-8, UTF-16LE/BE, and UTF-32,
   with and without validation, with and without error identification
 - From an UTF-8 string, compute the size of the Latin1/UTF-16/UTF-32
   equivalent string,
 - From an UTF-16LE/BE string, compute the size of the
   Latin1/UTF-8/UTF-32 equivalent string,
 - From an UTF-32 string, compute the size of the UTF-8 or UTF-16LE
   equivalent string,
 - UTF-8 and UTF-16LE/BE character counting.
 - UTF-16 endianness change (UTF16-LE/BE to UTF-16-BE/LE)
 - Base64 encoding and decoding

The functions are accelerated using SIMD instructions (e.g., ARM NEON,
SSE, AVX, AVX-512, etc.). When your strings contain hundreds of
characters, we can often transcode them at speeds exceeding a billion
characters per second. You should expect high speeds not only with
English strings (ASCII) but also Chinese, Japanese, Arabic, and so
forth. We handle the full character range (including, for example,
emojis).

The library compiles down to a small library of a few hundred kilobytes.
Our functions are exception-free and non allocating. We have extensive
tests and extensive benchmarks.

Dependencies

Commit History

may be incomplete — full history at freebsd-ports on GitHub

Loading commit history — this may take up to a minute on first view. Reload the page in a moment.