Software

Pnyx is an open source project. The source code and unit tests are hosted on GitHub, where you should report issues and submit pull requests. As a licensed product, we respond to issues quickly, especially during business hours. The Pnyx project is growing and we look forward to your feedback and feature requests.

The goal of the Pnyx project is to create a high-level programming library for performing ETL tasks. Linux provides a lot of these tools today, for free. However, they are disjoint: they were designed as stand-alone utilities.

For simple ETL tasks, most skilled Linux users can throw together a string of commands (sed, grep, awk, tee, tail, etc) to get the job done. But as the ETL tasks become more complicated, higher level coordination is needed. Most developers turn to scripting languages, like Bash, Python, Pearl, etc. These scripting languages are productive too (we use them daily), and for medium complexity tasks, they are great. Of course, there are drawbacks. For most development teams, these tools are not used on a daily basis, so extra knowledge is required of the team members.

Testing also becomes an issue. Any software worth spending time and money to build, even ETL software, is worth testing. Of course, every developer tests as they build the software. But what about automated testing? Does your current automated testing framework support testing scripts that access the file system? Most of them do, but there is an impedance to configuring filesystems across platforms, managing OS versions and managing dependencies. These issues are productivity killers, which is why containers have become so popular of late (like Docker).

Finally, as ETL tasks get even more complicated, it becomes unwieldy to manage without resulting to high level programming languages. Most sufficiently complicated projects already use high level programming languages and employ many talented developers with these skills. Wouldn't it be better to solve your ETL tasks with the same tools used by your product development team? Maybe. The problem with most high level programming languages, it that they have surprising low-level libraries for access the file-system. C# and Java are two great examples of languages with wonderful support for accessing Streams. But a Stream is a low level construct, and C#/Java libraries offer little beyond that.

This is the niche that Pnyx fills in your project's toolset. It offers a platform for building very complicated transformations, with the power of a high level programming language and the ease-of-use like the well known Linux utilities. The details of streams, resource management, temporary files, file encodings, and newline processing are all handled by the Pnyx library. Code written for Pnyx is trivial to test, which can be seen by the myriad of test cases for the Pnyx project. The library is simple to learn for developers, and is well documented.

For Pnyx, the future is bright. We have plans to port the library to other languages like Java, Javascript, and Python. And, with each release, Pnyx adds more features and bug fixes to support our valued customers.