How to get external DLLs into a python package

So, I recently stumbled upon the problem of having to include an external DLL into a python package, which should also work when turned into a windows executable by pyinstaller. This post, is meant to be a help if you run into the same problems I was having (and of course as my personal prosthetic knowledge).

The structure of this post will be:

• Finding dependencies
• Testing the DLL inside python
• Make the loaded DLL work inside a python package
• Bundling the DLL with pyinstaller

Finding dependencies

First of all, we need to make sure, that the DLL we want to load has no missing dependencies. There are many free tools out there to do that, I use dependencywalker for windows, which you can get here. For my scenario, all the functions I need are in API_C.dll. A scan with dependencywalker has the following output:

From this screenshot, you can see, that the dependencies of API_C.dll are API.dll which in turn depends on XAPI.dll. So in python, I need to make sure that either all dependencies are found in the same source or that the dependent dlls are loaded first. One solution would be to put the folder with all dlls in it on the path, but as I am aiming for a simple distribution of a package I don’t want to depend on the end-user’s PATH variable somehow.

Testing the DLL inside python

My next step is testing of the correct loading of the dll inside python. For testing, I put all the needed dll files in a folder with a simple dll_test.py python script:

import ctypes
import os

this_dir = os.path.abspath(os.path.dirname(__file__))

dep1 = ctypes.cdll.LoadLibrary(os.path.join(this_dir, 'XAPI.dll'))
dep2 = ctypes.cdll.LoadLibrary(os.path.join(this_dir,'API.dll'))
my_dll = ctypes.cdll.LoadLibrary(os.path.join(this_dir, 'API_C.dll'))

print(my_dll.GetVersion())

There is a function inside API_C.dll that is called GetVersion and that I can use to verify, my dll has loaded correctly. The os.path constructor helps to get the correct path at runtime. Note, that this joining with __file__ is outdated and one should use importlib’s resources handling as in this SO answer. I nevertheless use the __file__ syntax here because it works both in my IDE without the need to install the package as well as in the console with my installed package. It will most probably not work in an .egg distribution.

Make the loaded DLL work in a python package

To get everything into a package, a little bit more structure around one simple python code is needed. This is a minimal example of one package with one subpackage. The following code is posted bottom up:

C:\path\to\my\package_root
|   setup.py
|
\---my_package
    |   __init__.py
    |
    \---my_subpackage
            API.dll
            API_C.dll
            dll_test.py
            XAPI.dll
            __init__.py

In my_subpackage I have all needed DLL files. The dll_test.py has been modified a little to work better with my own package structure:

# -*- coding: utf-8 -*-

__all__ = ['dll_load_test']

import ctypes
import os

this_dir = os.path.abspath(os.path.dirname(__file__))


def dll_load_test():
    dep1 = ctypes.cdll.LoadLibrary(os.path.join(this_dir, 'XAPI.dll'))
    dep2 = ctypes.cdll.LoadLibrary(os.path.join(this_dir,'API.dll')
    my_dll = ctypes.cdll.LoadLibrary(os.path.join(this_dir, 'API_C.dll'))

    print(my_dll.GetVersion())
	
if __name__ == '__main__':
    dll_load_test()

The __init__.py file inside the subpackage simply loads this function:

# -*- coding: utf-8 -*-

__all__ = ['dll_load_test']

from .dll_test import *

In the my_package folder, the __init__.py simply loads the subpackage and defines the package version:

# -*- coding: utf-8 -*-

__version__ = '1.0.0'

from . import my_subpackage

And finally, there’s the setup.py in the package root, where I use setuptools to define package metadata as well as the data to be packed:

# -*- coding: utf-8 -*-

from setuptools import setup, find_packages

setup(name='DLL_test',
      version='1.0.0',
      description='DLL test package',
      url='https://www.dschoni.de',
      author='Dschoni',
      author_email='scripting@dschoni.de',
      license='MIT',
      packages=find_packages(),
      package_data={'':['*.dll']}, # This is the most important line.
      zip_safe=False)

Note that in this case, I am including all *.dll files in all subpackages. This could be done manually for each subpackage or even each file. There’s a lot more documentation over here. Now, you should be able to install the package e.g. with pip install . inside the package root. Once install, verify, that all DLL files are copied to the correct place in your install directory. You should be able to import the package in python now and use the DLL.

Bundling the DLL with pyinstaller

The last step is bundling the dll files with pyinstaller. To be able to do that make a simple testscript in the package root (or anywhere else) that loads your package. Run pyinstaller your_testscript.py and find the resulting your_testscript.spec file. Make sure to add the absolute or relative path to the DLL files in the datas statement such as:

datas=[('\path\to\API_C.dll', '.'),
       ('\path\to\API.dll', '.')]

The second entry specifies the path, where the DLL should be copied to in the resulting distribution folder. Now run pyinstaller your_testscript.py and you should find the DLLs together with an exe file in the distribution directory. Run the exe and make sure, the DLL is correctly loaded.