Contributing to pyomnisci¶
As an open-source company, OmniSci welcomes contributions to all of its open-source repositories, including pyomnisci. All discussion and development takes place via the pyomnisci GitHub repository.
It is suggested, but not required, that you create a GitHub issue before contributing a feature or bug fix. This is so that other developers 1) know that you are working on the feature/issue and 2) that internal OmniSci experts can help you navigate any database-specific logic that may not be obvious within pyomnisci. All patches should be submitted as pull requests, and upon passing the test suite and review by OmniSci, will be merged to master for release as part of the next package release cycle.
Development Environment Setup¶
pyomnisci is written in plain Python 3 (i.e. no Cython), and as such, doesn’t require any specialized development environment outside of installing the dependencies. However, we do suggest creating a new conda development enviornment with the provided conda environment.yml file to ensure that your changes work without relying on unspecified system-level Python packages.
Two development environment files are provided: one to provide the packages needed to develop on CPU only, and the other to provide GPU development packages. Only one is required, but you may decide to use both in order to run pytest against a CPU or GPU environment.
A pyomnisci development environment can be setup with the following:
# clone pyomnisci repo git clone https://github.com/omnisci/pyomnisci.git && cd pyomnisci conda env create -f ./environment.yml # ensure you have activated the environment conda activate omnisci-dev # install pre-commit hooks make develop
# from the pyomnisci project root conda env create -f environment_gpu.yml # ensure you have activated the environment conda activate omnisci-gpu-dev # install pre-commit hooks make develop
At this point, you have everything you need to develop pyomnisci. However, to run the test suite, you need to be running an instance of OmniSci on the same machine you are devloping on. OmniSci provides Docker images that work great for this purpose.
Docker Environment Setup¶
OmniSci Core CPU-only¶
Unless you are planning on developing GPU-specific functionality in pyomnisci, using the CPU image is enough to run the test suite:
docker run \ -d \ --name omnisci \ -p 6274:6274 \ -p 6278:6278 \ --ipc=host \ -v /home/<username>/omnisci-storage:/omnisci-storage \ omnisci/core-os-cpu
- With the above code, we:
create/run an instance of OmniSci Core CPU as a daemon (i.e. running in the background until stopped)
6274(binary connection) and
ipc=hostfor testing shared memory/IPC functionality
point to a local directory to store data loaded to OmniSci. This allows our container to be ephemeral.
To run the test suite, call
pytest from the top-level pyomnisci folder:
(pyomnisci_dev) laptop:~/github_work/pyomnisci$ pytest
pytest will run through the test suite, running the tests against the Docker container. Because we are using CPU-only, the
test suite skips the GPU tests, and you can expect to see the following messages at the end of the test suite run:
=============================================== short test summary info ================================================ SKIPPED  tests/test_data_no_nulls_gpu.py:15: No GPU available SKIPPED  tests/test_deallocate.py:34: No GPU available SKIPPED  tests/test_deallocate.py:54: deallocate non-functional in recent distros SKIPPED  tests/test_deallocate.py:67: No GPU available SKIPPED  tests/test_deallocate.py:80: deallocate non-functional in recent distros SKIPPED  tests/test_deallocate.py:92: No GPU available SKIPPED  tests/test_deallocate.py:105: deallocate non-functional in recent distros SKIPPED  tests/test_integration.py:207: No GPU available SKIPPED  tests/test_integration.py:238: No GPU available ================================== 69 passed, 13 skipped, 1 warnings in 19.40 seconds ==================================
OmniSci Core GPU-enabled¶
To run the pyomnisci test suite with the GPU tests, the workflow is pretty much the same as CPU-only, except with the OmniSci Core GPU-enabled container:
docker run \ --runtime=nvidia \ -d \ --name omnisci \ -p 6274:6274 \ -p 6278:6278 \ --ipc=host \ -v /home/<username>/omnisci-storage:/omnisci-storage \ omnisci/core-os-cuda
You also need to install cudf in your development environment. Because cudf is in active development, and requires attention to the specific version of CUDA installed, we recommend checking the cudf documentation to get the most up-to-date installation instructions.
Updating Apache Thrift Bindings¶
When the upstream mapd-core project updates its Apache Thrift definition file, the bindings shipped with
pyomnisci need to be regenerated. Note that the omniscidb repository must be cloned locally.
# Clone the omnisci repository git clone https://github.com/omnisci/omniscidb # Ensure you are at the root of the omnisci directory. cd ./omniscidb # Use Thrift to generate the Python bindings thrift -gen py -r omnisci.thrift # Copy the generated bindings to the pyomnisci root cp -r ./gen-py/omnisci/* ../pyomnisci/omnisci/
Updating the Documentation¶
The documentation for pyomnisci is generated by ReadTheDocs on each commit. Some pages (such as this one) are manually created, others such as the API Reference is generated by the docstrings from each method.
If you are planning on making non-trival changes to the documentation and want to preview the result before making a commit, you need to install sphinx and sphinx-rtd-theme into your development environment:
pip install sphinx sphinx-rtd-theme
Once you have sphinx installed, to build the documentation switch to the
pyomnisci/docs directory and run
make html. This will update the documentation
pyomnisci/docs/build/html directory. From that directory, running
python -m http.server will allow you to preview the site on
in the browser. Run
make html each time you save a file to see the file changes in the documentation.
Publishing a new package version¶
pyomnisci doesn’t currently follow a rigid release schedule; rather, when enough functionality is deemed to be “enough” for a new version to be released, or a sufficiently serious bug/issue is fixed, we will release a new version. pyomnisci is distributed via PyPI and conda-forge.
Prior to submitting to PyPI and/or conda-forge, create a new release tag on GitHub (with notes), then run
git pull to bring this tag to your
local pyomnisci repository folder.
To publish to PyPI, we use the twine package via the CLI. twine only allows for submitting to PyPI by registered users (currently, internal OmniSci employees):
conda install twine python setup.py sdist twine upload dist/*
Publishing a package to PyPI is near instantaneous after runnning
twine upload dist/*. Before running
twine upload, be sure
dist directory only has the current version of the package you are intending to upload.
The release process for conda-forge is triggered via creating a new version number on the pyomnisci GitHub repository. Given the volume of packages released on conda-forge, it can take several hours for the bot to open a PR on pyomnisci-feedstock. There is nothing that needs to be done to speed this up, just be patient.
When the conda-forge bot opens a PR on the pyomnisci-feedstock repo, one of the feedstock maintainers needs to validate the correctness of the PR, check the accuracy of the package versions on the meta.yaml recipe file, and then merge once the CI tests pass.