This tutorial will walk through the steps to make a distributable python package, host the package on github, distribute the package on PyPi, the Python Package Index, write documentation for your package, and host the docs on readthedocs. The instructions are for Ubuntu/Linux 64-bit operating systems. This is a bare bones introduction which should provide you with the essential info for getting started distributing software. The online literature for each aspect of distributing python software can be pretty daunting. I've included links to the clearest and most concise treatments I've found on the separate topics.
To complete this tutorial you will need to set up (free) accounts for Github, PyPi, and ReadTheDocs linked to a common email address if you don't already have these accounts set up. You can find instructions at the links below:
reStructuredText and Markdown are lightweight markup languages used for documentation among other things. This tutorial is written in Markdown. The docs for the toy python package used for demonstration are written in rSt.
To find out more about these common markup languages see the links below:
The world of python users is currently split between python2 and python3, which have some compatibility issues. If you want your code accessible to the greatest number of people you should try to write python code that is backwards/forwards compatible. Here is a link to some tools to help with the compatibility issue.
http://python-future.org/automatic_conversion.html
Pip is the tool that is used by PyPi to distribute python software. Sometimes it is good to install new python software in a virtual environment to make sure that it doesn't conflict with the core python distribution of your OS. To install pip and virtualenv:
$ sudo apt-get install python-pip python-virtualenv
You are going to need git installed on your machine. git is version control software used by github which acts as a central repository for your software. You can learn more about git here https://blog.udemy.com/git-tutorial-a-comprehensive-guide/. To install git:
$ sudo apt-get update
$ sudo apt-get install git
A python module is simply a .py file with reusable python functions and classes.
A python package is folder that contains a collection of python modules and packages that has a special __init__.py file at the top level. This file can simply be blank which is considered good practice, but can also be used to share code between the package's subpackages and modules.
For an in depth explanation of python packages and modules see here http://www.learnpython.org/en/Modules_and_Packages
I made a toy software package which is just complicated enough to explain the basics of package distribution, but simple enough to still be able to understand what is going on. You should clone this package from github. From the command line, in the directory where you want the package:
$ git clone https://github.com/aarontuor/helloMyName.git
I made this package following the minimal structure tutorial found here http://python-packaging.readthedocs.io/en/latest/minimal.html
I've already set up helloMyName on github, pypi, and readthedocs. Let's set up a virtual environment to install helloMyName and run it's limited utility in a python terminal.
$ mkdir ~/pythonDist
$ virtualenv --system-site-packages ~/pythonDist
$ source ~/pythonDist/bin/activate
$ pip install helloMyName
$ python
> from helloMyName import welcome
> welcome.hi(8, mood='depressed', verbose=True)
hello world! I am depressed to be a program.
I am veeeeery depressed to meet you
8
It doesn't do much but it is freely distributed software!
Now let's use the code you have cloned to make your own distribution! First let's get rid of everything that was specific to my distribution:
First let's change the helloMyName folders by replacing 'MyName' with your name or whatever you like. From now on in this document I'll refer to whatever you replaced 'MyName' with as 'YourName'. So where ever you see the string of characters 'YourName' you should replace with whatever you replaced the characters 'MyName' with.
Now in the top level helloYourName folder:
$ rm -rf docs
$ rm -rf dist
$ rm -rf .git
helloYourName has the following structure:
helloYourName # top level of distribution
setup.py # for installing the package
README.rst # for package's github homepage
helloYourName # package
__init__.py # so python knows this is a package
welcome.py # the module of reusable python functionsSo, our distributed code contains one package and one module.
Now let's look at the contents of setup.py, the code that is used to install your program.
from setuptools import setup, find_packages
setup(name='helloMyname',
version=0.02,
description='Python package example',
url='http://mywebsite',
author='myname',
author_email='tuora@students.wwu.edu',
license='none',
packages=find_packages(), # or list of package paths from this directory
zip_safe=False,
install_requires=[],
classifiers=['Programming Language :: Python'],
keywords=['Package Distribution'])
You'll wan't to change the name, url, author, and author_email arguments for your package. Also you probably want an open source license named LICENSE, in the top level directory of helloYourName. A guide to open source licenses can be found here http://docs.python-guide.org/en/latest/writing/license/
You can install this package manually by running the setup.py script with either a 'develop' or 'install' argument. Assuming you are in the virtual environment we set up and in the top level directory of helloYourName if you run:
$ python setup.py develop
Then a symlink to the source code will be copied to the directory ~/pythonDist/lib/python2.7/site-packages/helloYourName. If you use 'install' instead of 'develop' all the source in helloYourName will be copied to this location instead.
For a more detailed introduction to basic info about setup.py files look here: http://the-hitchhikers-guide-to-packaging.readthedocs.io/en/latest/quickstart.html.
We will automatically generate most of our documentation from python doc-strings using Sphinx, the python documentation generator. So we need to install Sphinx. I recommend a virtual environment for installing Sphinx but it isn't strictly necessary. In the top level directory of helloYourName:
$ pip install --upgrade sphinx sphinx-autobuild # installs Sphinx
$ mkdir docs
$ cd docs
$ sphinx-quickstart # populates docs directory with basic sphinx docs build
At this point you will be asked a series of questions to customize your initial docs build. Many of these you can just hit enter and choose the default. Some information you must provide, such as the project name (helloYourName), the author name (YourName), the project version (0.1), and the project release (0.1.1). Version numbers typically have two numbers separated by a decimal and release numbers have the version number and some release qualifier after another decimal. There are many lengthy discussions online about version and release numbers. The most straightforward can be found here http://the-hitchhikers-guide-to-packaging.readthedocs.io/en/latest/specification.html. Basically each new version and release should have a larger number. So, start small and work your way up.
Here are some questions which you should answer yes to so that important extensions are enabled:
autodoc: automatically insert docstrings from modules (y/n) [n]: y
doctest: automatically test code snippets in doctest blocks (y/n) [n]: y
mathjax: include math, rendered in the browser by MathJax (y/n) [n]: y
viewcode: include links to the source code of documented Python objects (y/n) [n]: yNow you can make your first docs:
$ make html # makes html files in docs/_build/html/
Now the framework of your documentation is set up! You can view these docs at helloYourName/docs/_build/html/index.html.
The sphinx documentation is pretty dense and bewildering but you can find it here http://www.sphinx-doc.org/en/stable/.
conf.py¶sphinx-quickstart populated your docs directory with a file conf.py. This is the file that tells
sphinx where you specify all the configurable options.
Find a small image (your_pic.png) to use as your logo, and place it in helloYourName/docs/_static.
Add these lines in the appropriate spots in conf.py.
import os
import sys
sys.path.append(os.path.abspath('sphinxext')) # lets python know where to find your sphinx extensions
sys.path.append(os.path.abspath('../helloYourName/')) # This is so sphinx knows where to find your module
html_theme = "sphinx_rtd_theme" # Nice clean theme.
html_logo = '_static/your_pic.png' # adds logo to documents pages.
The index.rst file that sphinx-quickstart populated in your docs folder is analogous to an index.html file for a website. It is the base page of your documentation and may be all you really need for a small package. We'll go into adding pages a little later. Your index.rst file should look something like below:
.. helloYourName documentation master file, created by
sphinx-quickstart on Wed Jan 4 16:40:47 2017.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to helloYourName's documentation!
=======================================
Contents:
.. toctree::
:maxdepth: 2
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`Add these lines above 'Indices and tables' in index.rst so that your docstrings in the welcome module will be used to auto-document the module.
.. automodule:: welcome
:members:
:undoc-members:Let's look at how the docstring is formatted to achieve this magic!
def hi(some_number, mood='excited', verbose=False):
"""
Fancy hello world function. :math:`\sum_{i=1}^{10} badDocs*BadlyWrittenCode = frustration`
:param some_number: (int) Something to return.
:param mood: (str) A string describing a mood.
:param verbose: (boolean) Where or not to be a gushing hello world program.
:return: (int) some_number
"""
These are rSt style docstrings. Other popular styles of python docstrings are numpy, and google style docstrings which each follow their own syntax. A concise description of these different formats of docstrings can be found here: http://daouzli.com/blog/docstring.html. In order to parse numpy and google docstrings with sphinx autodoc you need to add the sphinx extension Napolean http://www.sphinx-doc.org/en/1.5.1/ext/napoleon.html. The pycharm IDE will auto-complete your docstrings for you in any of these styles if you type three quotation marks and return.
It's nice to give examples in your documentation, but you should make sure that your examples give the correct output. Doctest was designed for this:
In your docstring for the hi function include the following:
:examples:
>>> from helloAaron import welcome
>>> welcome.hi(1, 'thrilled', verbose=True)
hello world! I am thrilled to be a program.
I am veeeeery thrilled to meet you
1
Now from docs run:
$ make html
$ make doctest
From the helloYourName directory:
$ python setup.py register
$ python setup.py sdist upload
You should be prompted for your PyPi user name and password. If for some reason the software gets confused you
can add a .pypirc file with the following contents:
[server-login]
repository=https://pypi.python.org/pypi
username:yourPyPiUserName
password:yourPyPiPasswordNow your software is hosted on PyPi the Python Package Index!
Now anyone can install your package by:
$ pip install helloMyName --user
Try installing your neighbor's package and trying out the code:
from helloYourName import welcome
welcome.hi(1, 'sad', True)
helloYourName top level directory:$ git init
$ git add --all
$ git commit -m 'first commit'
$ git remote add origin https://github.com/<YourGitHubUserName>/helloYourName.git
$ git push origin master
First let's set it up so that your github will sync with your readthedocs account.
If your project is hosted on GitHub, you can easily add a hook that will rebuild your docs whenever you push updates. From your repo on github:
Note
The GitHub URL in your Read the Docs project must match the URL on GitHub. The URL is case-sensitive.
If you ever need to manually set the webhook on GitHub, you can point it at https://readthedocs.org/github.
Import a ProjectImport ManuallyNext.Build. Wait a bit.Congratulations your docs are hosted on readthedocs!
You may include some command line python scripts in your package. Let's make the following script at helloMyName/bin/helloWorld.py:
#!/usr/bin/env python
import argparse
def return_parser():
parser = argparse.ArgumentParser("Command line tool to print Hello World and demonstrate documenting scripts")
parser.add_argument("arg0", type=int, help="A needless integer argument for demonstration")
parser.add_argument("-needless_arg_1", type=int, default=1, help="A needless integer argument for demonstration")
parser.add_argument("-needless_arg_2", type=str, default='oops', help="A needless string argument for demonstration")
return parser
if __name__ == '__main__':
args = return_parser().parse_args()
print(args.arg0)
print(args.needless_arg_1)
print(args.needless_arg_2)
print('Hello world!')
Now in helloMyName/setup.py add the following argument to the call to setup:
scripts=['bin/helloWorld']
We can test out this command line script now. From the top level helloMyName directory:
$ python setup.py develop
$ helloWorld 1
1
1
oops
Hello world!
To document the script, in helloMyName/docs/index.rst include the following line:
helloWorld.py
==================
.. argparse::
:ref: helloWorld.return_parser
:prog: helloWorldNow in conf.py include 'sphinxarg.ext' in the extensions list.
Now from the command line:
$ pip install sphinx-argparse
Now from the helloMyName/docs/ directory:
make html
You can also link additional pages to your documentation. In helloMyName/docs/index.rst in the toctree section include these lines:
.. toctree::
:maxdepth: 2
:caption: Contents:
welcome.rst
goodbye.rstNow make your welcome.rst, and goodbye.rst files in the docs directory.
welcome
=======
This is a tutorial of how to use the welcome.rst module. It's really great.
### goodbye.rstWe plan to extend this package to include a goodbye module. ```
Now remake the docs with the make html command.
For a few final touches let's customize your README.rst file. Right now the README.rst has all my info in it so you should swap out the urls and names for your docs and project name. Don't forget to add, commit and push your changes to github!
If you don't like using rSt for your README github will also parse a README.md file.
Now that we've made some changes let's update the project on Pypi and ReadTheDocs.