We use python in a very complicated and critical scenario – automatic deposit and withdrawal of crypto-currencies.

It is hard to say whether choosing Python in this case is a good decision. It is a highly dynamic and untyped (by default) language which performs poorly. However, we did observe some of its merits in our engineering experience, including flexibility, ease of debugging, speed of development, pleasant syntax, and plenty of well-designed libraries.

In this article, I’ll focus more on the technical aspects, like code snippets or workflows that might be helpful to you. We will also discuss some high-level ideas.

Writing Code

Repo organization

│   ├── control
│   └── postinst
├── Makefile
├── README.md
├── base*
│   ├── __init__.py
│   ├── ...
│   └── util.py
├── bin
│   ├── load_addresses
│   └── ...
├── btc
│   ├── __init__.py
│   ├── ...
│   └── worker
│       ├── __init__.py
│       ├── ...
│       └── broadcast.py
├── build.sh
├── db
│   ├── 1001-initial-tables.sql
│   └── ...
├── broadcast.service
├── requirements.txt
├── setup.py
└── tests
    └── main

NOTE: the base module is annotated with a * because it is in fact maintained as a git submodule.

Basically we follow what the most Python packages will do, putting the library source code under a specific folder (btc in this case). Each level of folder needs to have a __init__.py if you want to import this folder path as a module as well (e.g. things defined in btc/worker/__init__.py can be found under namespace btc.worker after importing.

The setup.py is responsible for registering and installing the package, such that you can import it with python anywhere.

from setuptools import setup, find_packages

    packages=find_packages(exclude=['contrib', 'docs', 'tests*']),

The tests/main is a python script for integration test. In our project, all integration test’s relevant files are put inside a local and exclusive folder, e.g. tests/test01. So we can support a maximum concurrency rate of up to N. However, we should use virtualization in future to solve the external resource race problem.

We put all MySQL schemas in db. They are numbered according to the date being checked in, and all modification of schema must be appeneded incrementally as new SQL file, rather than directly modifying the previous .sql files.

We also rely on virtualenv for our development.

This is the script run by build bot each time for CI:

set -x
set -e

## run in build bot

virtualenv -p python3 .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -e .
make check


  • All pure utilities in bin are chmod a+x‘d, renamed as extension-less, and added shebang #!/usr/bin/env python3 head.
  • All command line tools should use argparse to parse arguments
  • Respect PEP-8 for basic code style
  • Respect PEP-257 and PEP-287 for writing inline documentation
  • Write type hints with best efforts
  • Use pymysql through the standard API in PEP-248 and PEP-249
  • Do NOT use float for precise calculation
  • For a complex constructor with a lot of string arguments, name each argument on the call-site
  • Extend generic functionalities through default arguments and try not to break legacy code
  • Don’t catch generic Exception, catch what you specifically need
  • Don’t over-use the exception mechanism – it might be meaningful to abort in some cases

Static Check

Lint code with PyLint

PyLint can detect most of the static errors. There are several tips on using it:

  1. Install it in the project’s virtualenv (along with other in requirements.txt), or else it might fail to find some depended packages
  2. Using vanilla pylint is too strict for CI purpose. We will let build-bot run pylint -d W -d C -d R -d U instead. You might need to use annotation # pylint: disable=<error-name> to disable lint of particular type of errors about particular line, file, imported module, or the entire project.

Check type hints statically with mypy

mypy can check the code against type hints. We usually run it as mypy --ignore-missing-imports.

Type hints can also be a natural and important documentation.

Dynamic Check

Check type hints dynamically with enforce

enforce makes further use of type hints to check type dynamically (since Python supports powerful reflection). This can make your code solid-safe.

However, as you may expect, performance will be affected – but in our case, perf is the last thing to consider.

Defensive Programming

We use a lot of assert in our code. This makes our code more robust against problems, and also serve as a nice documentation about the pre-condition of the code.

When using a conditional without else, as yourself if this is an exception that should be reported.

Integration Test and Code Coverage

Our integration test can achieve more than 80% code coverage. The uncovered part are almost all exception handling code.

The major external states of our service are database and blockchain. All of them will be version-controlled and initialized from scratch in each run of CI. This caught most bugs in a reproducible way before them creeping into production. Also, the integration test is written from the end user’s perspective for more flexibility against code change, and everything is made as close to the production environment as possible.

Continuous Integration

Our CI process includes two parts:

  • static check
    • pylint
    • mypy
  • integration test
    • during which coverage will be collected and checked as well

By going through all 357 build logs in past two months for one typical project, we found that distribution of failure reasons is:

  • integration test error: 39
  • lint error: 9
  • type error: 5
  • coverage error: 1

Packaging and Release

Our software consists of two types: daemons and utilities.

We pack the Python runtime, libraries and our scripts together as a standalone Linux executable with the pyinstaller tool. The advantage is that we don’t have to install any Python dependencies on the target machine. However, it will become harder to debug since only bytecode is available in released binary.

The daemons are managed as systemd services. We release the software as a .deb package.

All the packaging work are done in the Makefile.

Operation and Monitoring

The stdout log of our daemons will be collected by syslog. We manage the daemons using standard interface provided by systemd, such as systemctl status/start/stop/restart. Finally, our monitoring infrastructure is based on collecting simple metrics: latency, traffic, errors, saturation (see prometheus for more). We also health-check our services through active heartbeat.