I will walk through a sample web application built using python, flask and docker. The application invokes a specialized math function in a library co-developed during application development.
The aim of this blog is to highlight some project structure best practices, that I learnt and used in my own application.
For those not familiar with flask, flask is a small web framework that implements the python WSGI interface.
WSGI is an application level contract to develop web applications in python. The other side of the contract is implemented by WSGI webservers like UWSGI.
By no means, these practices are limited to a flask application, and could be applied everywhere.
Summary
I will highlight the following areas using a sample project which can be found here
- Project layout.
- setup.py
- requirements.txt
- __init__.py
- Local Deployment
- Intra-package and inter-package references.
- Class factories
- Test cases and code coverage
- DockerFiles
Project Layout
The project layout looks like this.
.
├── docker-compose.yml
├── Dockerfile
├── Dockerfile.unittests
├── requirements.txt
├── run_coverage.sh
├── uwsgi.ini
└──-bettermath-lib
├── setup.py
└── bettermathlib
├── better_random.py
├── __init__.py
└── bettermathlib_tests
├── tests_bettermathlib.py
├── __init__.py
└── randomweb-app
├── setup.py
└── randomwebapp_tests
├── tests_basic.py
├── tests_client.py
├── __init__.py
└── randomweb_app
├── config.py
├── create_app.py
├── flask_app.py
├── random_creator.py
├── __init__.py
└── main
├── views.py
├── __init__.py
└── static
├── about.txt
└── templates
├── base.html
└── betterrandom.html
A distribution is a folder that can be installed and moved around anywhere. bettermath-lib and randomweb-app are the self-contained distributions.
A package is a directory contains other packages or modules and is also called "Import Package" because its importable.
The file __init__.py is a special file that is executed when a package is imported in the runtime.
Here bettermathlib, bettermathlib_tests are packages inside the distribution bettermath-lib. randomweb_app and randomwebapp_tests are packages inside the distribution randomweb-app, and finally main is a package inside the package randomweb_app.
setup.py
setup.py is part of setuptools and defines properties of a distribution.
from setuptools import setup, find_packages
setup(
name='randomweb-app',
version='1.0',
description='Random Webapp',
author='Fishy Baker',
author_email='fishybaker@hotmail.com',
packages=find_packages(),
install_requires = [
'Flask >= 1.1.1',
'Flask-Bootstrap >= 3.3.7.1',
'Flask-WTF >= 0.14.2',
'jsonschema >= 3.0.1',
'bettermath-lib >= 1.0'
],
include_package_data=True,
package_data={'randomweb_app': ['templates/*', 'static/*']}
)
There are three main parts of setup.py
packages=find_packages(),
find_packages is a neat helper to automatically include all packages inside randomweb-app as part of the distributable. The other option would be to create a static python list with a list of packages.
include_package_data=True,
package_data={'randomweb_app': ['templates/*', 'static/*']}
The third importand section is install_requires. It is typically a list of minimum versions of other distributions that your package has a dependency on.
The canonical way to obtain this list is via pip freeze command.
The catch is that we will not use this list in our application environment.
The consumers of your distribution should ideally be free to choose the latest and greatest stable versions, and ensure that their environment is tested.
requirements.txt
requirements.txt is the list of dependencies that the application is tested with. In the production environment, you will want to install specific versions of those dependencies.
requirements.txt is not part of the distribution. That role is fullfilled by setup.py's install_requires.
Hence requirements.txt lives at a folder level higher than the distributions.
pip freeze > requirements.txt
The typical production deployment should execute a variant of pip install with -r requirements.txt.
pip install -r requirements.txt
__init__.py
__init__.py plays a crucial role in namespacing the package's contents appropriately.
The class BetterRandom is in module better_random.py. So a consumer of my distribution would have to import bettermathlib.better_random to access BetterRandom.
By directly importing BetterRandom in __init__.py we bring BetterRandom to bettermathlib namespace, and then "import bettermathlib" is a much better import style than "import bettermathlib.better_random"
from .better_random import BetterRandom
Local Deployment
Pip has an option -e, called the editable option, using which pip will reference the local source paths in the deployment index.
pip install --no-deps -e bettermath-lib/
pip install --no-deps -e randomweb-app/
After this, both the packages are self-contained and installed in the dev environment. We can continue to make code changes without re-installing the packages.
We also want to exclude these distributions in the applications requirements.txt in case we need to regenerate requirements.txt.
pip freeze --exclude-editable > requirements.txt
For co-developed distributions we will call pip seperately for these distributions.
Intra-package and inter package references.
References is now where we will see the advantage of keeping packages distributable come in.
For referencing modules within a package, we will use the intra-package references using the . notation.
We should be ok taking dependecies on where the relative paths of modules within a package are.
from ..random_creator import random_int
For inter-package references, we can now use absolute references without worrying about where the actual code for the package resides.
from bettermathlib import BetterRandom
Class Factories
This section is flask-app sepcific, but the principle applies everywhere.
This application is a flask micro-service, and does not need a "main" anywhere. Services like uwsgi and flask commandline expect to find a callable object in a module.
I created a module called flask_app.py inside randomwebapp which creates a global app object using a class factory which lives inside another module create_app.py.
Hence the module we want to define in uwsgi.ini becomes randomwebapp.flask_app.
module = randomweb_app.flask_app
callable = app
The class factory function create_app.create_app does flask initializaiton based on the passed configuration. This is now the only user object we want to expose in my package's namespace, as it will be useful in testing.
Test cases and code coverage
With distributed packages, there are a few options for testing. We cannot rely on test-discovery, as we want my tests to be deployable.
We can exploit the __init__.py again to bring all test classes directly into the namespace of the test package.
from .tests_bettermathlib import BetterMathTestCases
Now unittests can be run via the unittest module. We can choose to run both the test packages together or seperately.
python -m unittest randomwebapp_tests bettermathlib_tests
For Flask we can use a flask command-line-interface property to explicitly define all the test packages that verify this application.
app = create_app(os.getenv('FLASK_CONFIG') or 'default')
@app.cli.command()
def test():
"""Run the unit tests."""
import unittest
testmodules = [
'bettermathlib_tests',
'randomwebapp_tests',
]
suite = unittest.TestSuite()
for t in testmodules:
suite.addTest(unittest.defaultTestLoader.loadTestsFromName(t))
unittest.TextTestRunner(verbosity=2).run(suite)
Now flask tests starts working for me -
(venv) ...>flask test
test_better_random (bettermathlib_tests.tests_bettermathlib.BetterMathTestCases) ... ok
test_app_exists (randomwebapp_tests.tests_basic.BasicsTestCase) ... ok
test_app_is_testing (randomwebapp_tests.tests_basic.BasicsTestCase) ... ok
test_home_page (randomwebapp_tests.tests_client.FlaskClientTestCase) ... ok
----------------------------------------------------------------------
Ran 4 tests in 0.036s
OK
and code coverage
coverage run -m flask test
coverage report
DockerFiles
There are two DockerFiles one for production and one for unit-testing. The production docker files installs the packages non-editable.
RUN python3 -m pip install -r requirements.txt
RUN python3 -m pip install --no-deps bettermath-lib/
RUN python3 -m pip install --no-deps randomweb-app/
The unittest takes the production docker image and uninstalls the two packages and re-installs them editable. This allows us to retrieve code-coverage results from a container execution.
RUN python3 -m pip uninstall -y bettermath-lib
RUN python3 -m pip uninstall -y randomweb-app
RUN python3 -m pip install coverage
RUN python3 -m pip install --no-deps -e bettermath-lib/
RUN python3 -m pip install --no-deps -e randomweb-app/
Thank you. The complete sample project is here.
Go Top