Please note: this code is still alpha / pre-production. Everything here should be considered preliminary.
If you like zsv+lib, please give it a star!
zsv+lib is a fast CSV parser library and extensible command-line utility. It achieves high performance using SIMD operations, efficient memory use and other optimization techniques.
Preliminary performance results compare favorably vs other CSV utilities (
mlr (miller) etc).
Below were results on a pre-M1 OSX MBA; on most platforms zsvlib was 2x faster, though in some cases the advantage was as small as
20% (see also M1 update note below):
** See 12/19 update re M1 processor at https://github.com/liquidaty/zsv/blob/main/app/benchmark/README.md
zsv is an extensible CSV utility, which uses zsvlib,
for tasks such as slicing and dicing, querying with SQL,
combining, converting, serializing, flattening and more.
zsv is streamlined for easy development of custom dynamic extensions, one of which is
available here and offers added features such as statification and validation reporting,
automated column mapping and transformation, and github-like capabilities for sharing
zsv are written in C, but since zsvlib is a library, and
extensions are just shared libraries, you can extend
your own code in any programming language, so long as it has been compiled
into a shared library that implements the expected interface.
zsvwith a few lines of code via modular plug-in framework. Just write a few custom functions and compile into a distributable DLL that any existing zsv installation can use
zsvare permissive licensed
Pre-built binaries for OSX, Windows, Linux and BSD are available at https://zsvhub.com/download
zsv runs best (and by far, fastest) as a desktop CLI. But, you can also try out an extended
version in the browser (though it runs much slower), at
https://zsvhub.com/playground/. A tutorial that demonstrates a small subset of the
capabilities of ZSV and the ZSVHub extension is available at
Our objectives, which we were unable to find in a pre-existing project, are:
There are several excellent tools that achieve high performance. Among those we considered were xsv and tsv-utils. While they met our performance objective, both were designed primarily as a utility and not a library, and were not easy enough, for our needs, to customize. This was because they were not designed for modular customizations that could be maintained (or licensed) independently of the related project (in addition to the fact that they were written in Rust and D, respectively, which happen to be languages with which we lacked deep experience). Others we considered were Miller (mlr), csvkit and Go (csv module), which did not meet our performance objective. We also considered various libraries using SIMD, but none seemed to (yet) meet the "real-world CSV" objective.
Hence zsv was created as a library and a versatile application, both optimized for speed and ease of development for extending and/or customizing to your needs
zsv comes with several built-in commands:
echo: read CSV from stdin and write it back out to stdout. This is mostly useful for demonstrating how to use the API and also how to create a plug-in, and has some limited utility beyond that e.g. for adding/removing the UTF8 BOM, or cleaning up bad UTF8
select: re-shape CSV by skipping leading garbage, combining header rows into a single header, selecting or excluding specified columns, removing duplicate columns, sampling, searching and more
sql: run ad-hoc SQL query on a CSV file
desc: provide a quick description of your table data
pretty: format for console (fixed-width) display, or convert to markdown format
2tsv: convert CSV to JSON or TSV
serialize(inverse of flatten): convert an NxM table to a single 3x (Nx(M-1)) table with columns: Row, Column Name, Column Value
flatten(inverse of serialize): flatten a table by combining rows that share a common value in a specified identifier column
stack: merge CSV files vertically
Each of these can also be built as an independent executable.
./configure && sudo make install
See INSTALL.md for more details.
After installing, run
zsv help to see usage details. The typical syntax is
zsv <command> <parameters>
zsv sql my_population_data.csv "select * from data where population > 100000"
In addition to the above extensions, at least one third-party extensions will be made available. If you would like to add your extensions to this list, please contact the project maintainers.
You can extend
zsv by providing a pre-compiled shared or static library that
defines the functions specified in
extension_template.h and which
zsv loads in
one of three ways:
zsvexecutable and loaded at runtime if/as/when the custom mode is invoked
You can build and run a sample extension by running
make test from app/ext_example.
The easiest way to implement your own extension is to copy and customize the template files in app/ext_template
This alpha release does not yet implement the full range of core features that are planned for implementation prior to beta release. If you are interested in helping, please post an issue.