Building a plugin architecture with Python

It’s no secret that one of the greatest software design principles when extending functionality is to consider using plugins (where possible)*

I will keep this post short and avoid repetition, if you don’t know what a plugin system is then now would be a good time to find out from here. Now that is out of the way, let us create a small use-case that we need a solution for.

Problem statement

Given a broad category of devices that share similar functionality and a communication interface with indecent protocols, in this case let’s call them inverters (for a solar system). These inverters can provide us with power readings for input (solar and or battery), output (to mains/load) and device information/status e.t.c that we can access though some text based. Say for example to change from hybrid mode (solar and grid) to just offline mode (solar and battery only).

At this point we know that a monolithic application would be either too large to accommodate all the possible inverter types and protocols, so our best bet would be to create a core application that only knows about the high level features (read device info/status, read power data and send commands to change some settings). We can achieve this by creating small units to handle the complexities of the communication protocols (taking in the core application commands and transforming them into commands the inverter can understand). This approach would allow us to ship a bare application and apply/download a plugin for a specific inverter/s which we have.

Fundamental plugin concepts

A common plugin system typically needs to define some form of contract/s that the plugins can extend and the core application can use as an abstraction layer for communication, this could be in the form of a package with some abstract classes. Typically the application would using the following steps/phases to interact with the plugins:

Discovery

This is the mechanism by which a running application can find out which plugins it has at its disposal. To “discover” a plugin, one has to look in certain places, and also know what to look for.

Registration

This is the mechanism by which a plugin tells an application — “I’m here, ready to do work”. Admittedly, registration usually has a large overlap with discovery.

Application Hooks

Hooks are also called “mount points” or “extension points”. These are the places where the plugin can “attach” itself to the application, signalling that it wants to know about certain events and participate in the flow. The exact nature of hooks is very much dependent on the application.

Exposing application API to plugins

To make plugins truly powerful and versatile, the application needs to give them access to itself, by means of exposing an API the plugins can use.

Code examples

I will keep this section short so for the sake of time saving and simplicity, please see the full sample-project on github

wax911/plugin-architecture

Let’s start off by defining some sort of configuration file, which will tell our application which plugins to load and where to get them: (settings/configuration.yaml)

Discovery

Let us create a few contracts that all plugins will implement/extend, one contract will be used for registering a plugin when it is loaded/registered (will go into more detail later) and the other defines the behaviour of the plugin (what it can do and the information we can expect to get):

The references models can be found here, these are just data classes for demonstration purposes

N.B. IPluginRegistry extends type because we need some sort of mechanism to register plugins which we will not instantiating immediately after the discovery phase, think of this as just keeping information on what class type we’re dealing with (we would eventually call plugin_registries[index]() to instantiate the class)

If we look at the init(constructor) block inside our IPluginRegistry we can see that it contains something interesting, it checks if the loaded class (this can be considered as part of the hook phase) is not the plugin contract (PluginCore) not to be confused with the instanceof operator/function.

Registration

In the sample project all our plugins will be placed in a plugins directory, this is to simulate a case where a plugins are downloaded and kept.

Sample plugin directory structure which can be found here

Each of the plugins will have a configuration file called plugin.yaml that will contain information regarding which “main” file to load, dependencies which we will install automatically (yes, I know this would be a big security risk in production if plugins are not audited against malicious intent)

plugin.yaml configuration file for the sample plugin

sample.py the main sample plugin file which can be found here

Our sample plugins extends PluginCore and expects a logger as a dependency from the application (This is to simulate the main application providing dependencies to its plugins, this could be in the form of a configured network client, a database wrapper to provide read only functionality or anything really) and has an invoke method which is called by the core application to run commands on the destination device.

PluginUseCase discover and load/hook plugins

The above code snippet is from the PluginUseCase class where the application will search the plugin directory for a configuration file (plugin.yaml) which is handled in PluginUtility (for all the installation and dependency resolution details), and finally loaded/registered after all checks are done.

You can read more about what import_module is from here

Application hooks and exposing the API

Plugins are loaded through __search_for_plugins at this point

After the plugins are loaded we clean up the global registry (IPluginRegistry) since all loaded plugins will be stored in the PluginUseCase scope, and we’re ready to use our plugins through the following functions which our core application will call on.

Plugin instance creation and invocation

Arguably one could omit the register_plugin and hook_plugin functions and directly do this inline in the the calling core application module, but for the sake of demonstrating the phases separately we’re doing it this way

Application core module which can be found here

In this example we’re just calling calling all the loaded plugins, depending on our use-case we may just want one plugin and that’s easily done by simply specifying a single plugin in our configuration.yaml file.

Running the project and we’d get the following output:

Thank you for reading and let me know if you have any questions :)

Credits for inspiration: https://eli.thegreenplace.net/2012/08/07/fundamental-concepts-of-plugin-infrastructures