September 5th, 2012, 06:33 AM
Making a python plugin system
I was hoping that somebody could help me with either changing the way I am approaching this problem or with some suggestions on how to implement it.
I am creating a script that will retrieve information from a number of devices. There are several types of devices, Currently I am able to one using telnet to connect and with telnetlib and run the commands to retrive the data. Another device produces a handy xml page that I can parse. These are the devices that I know about in my current location. However I need to deploy this in another location and I have no idea what devices they are using or how those devices present the data once it has been converted. And I will have to leave it to others to write the modules required for these different devices. I am trying to create a standard interface so that the modules only have to return their info in a specific way. So....
What I am trying to do is work out how to manage this in a plugin type fashion. My initial investigation have gone down the road of loading modules at runtime as in the dive into python book --
My problem here though is that I end up with a list, that contains each of the modules but I have no way to match up each module in the list to the type of device that it will need to use to retrieve the date.
I have a database set up that will have the ip address of each device and the name of the module that it will need to use e.g.
192.168.1.1 | egx-100
What I am struggling with is how to bridge the gap between all the modules that have registered at run time, which I will not know the name of them ahead of time and the module that gets added to the list imported at run time... maybe if I get each module to declare it's name and the script can query each module in the list to find the name ... this just occured to me as I was typing this.
I am using a database as the results will all be stored in here so it is convenient to create a lookup table that has this information in it.
This code looks in the drivers directory and imports and modules in there. However all I have at this point is a list of each of the module objects.
current_path = os.getcwd() + "/drivers"
directory = os.getcwd() + "/drivers"
extension = ".py"
list_of_files = [file for file in os.listdir(directory) if file.lower().endswith(extension)]
module_names = 
for file_name in list_of_files:
file_name = file_name.rstrip(".py")
modules = map(__import__,module_names)
modules.print_string('this is a test')
I would appreciate any assistance or suggestions when it comes to alternative ways to approach or resolve this. As I said this will need to be something that I can deploy and then once I have properly documented the interface I can leave it to others to write their own modules for their own devices.
Thank you all and I understand if this is a big ask.
October 1st, 2012, 07:20 PM
just thought I'd chip in as answers have been pretty thin on the ground.
I've just done something similar to this as a plugin system for a CMS. I wanted users to be able to just drop a python module into a directory and click 'update' within the admin area. The python module might do anything but it would always produce some kind of html output or perform a function in the background when the module was called. Unfortunately my code is a horrendous mess at the moment because I've just got it to that 'ugly but working stage', but I can give kind of an explanation of how I got it to work.
I have one 'master' module which provides a plugin base class and a module import function which also handles reloading changed modules so I don't have to start the application each time a module is updated, deleted or added. The master module makes the main application aware of the modules it is importing by storing the module names in a list as strings.
Each submodule contains a class object whose methods process whatever (like dealing with your devices, mine produce html from dynamicly changing information) and then the collective output from the submodule is stored in standard named attributes inherited from a plugin class in the master module. This way every module can have it's own custom processing but a standard interface when it needs communication with the rest of the app.
I have then used setattr() and getattr() from the main application to interact with the submodules via the master module and it's universal plugin class. This way you don't have to know the module names before the app is run. The master module can read the imported module names, keep a list of them as strings and then using getattr() for example, you can action whatever is in a certain named module using it's name as a string.
Sorry if I haven't used much tech language to make it clearer, I'm new to python myself and never had all this object-orientated headache in C, which is like my comfort-blanket
October 2nd, 2012, 08:43 AM
Thanks for the reply, glad to hear I'm not the only one trying to do this. There seems to be little information about such mechanisms about so I am wondering how all these applications get written that have such extensive plugin systems. Especially ones that seem to work with python as the extension language. I will have a good read through your reply and absorb what you are saying and see if I have more answers than questions
Thanks for the pointers and sharing your experience.
Originally Posted by zilog6502
October 2nd, 2012, 09:20 AM
Hi again Alan,
this page, though a bit outdated is a good link for plugin frameworks....
Alchin's approach was the most thorough description I could find but not the easiest for me to understand if I were to implement it. The main problem with plugins in Python is that when a plugin is changed in some way it needs to be reloaded into the main application and there seems to be a lot of argument over how this is done. My only experience with plugin architectures has been with content management systems and applications such as Django seem to basically just restart the whole application to reload or 'install' a plugin module.
This is why I created a middle-manager module with a standard interface to the main application and just provides a base plugin class for submodules to subclass as a structure for a plugin/extension.
If your main app is written in something like C get your developers to have a quick look at the Gedit source code and how it's plugins work as Python can be used for Gedit plugins.
Last edited by zilog6502; October 2nd, 2012 at 09:22 AM.
Reason: Reworded Gedit explanation
October 3rd, 2012, 02:26 PM
thanks again, i'll have a look at that. I am the developer in question so there is only myself. The overall project is not that difficult to achieve and i have done a lot of proof of concept scripts. The issue arises as one of the 'deliverables' for the project is that it can be rolled onto other sites with hardware that i have not seen and I won't be available to adjust the script to take that into consideration.
I am lucky in the respect that it will be running on a scheduled basis so any changes to a plugin will take affect the next time it is run and changes would only be rolled out once they had been tested. Once it is loaded and running there is no dynamic changes to the plugin needed, however people will need to be able to add their code to talk to their devices without knowing anything about the gubbins in the script.
Thanks again for your input.
Originally Posted by zilog6502
October 5th, 2012, 08:16 AM
Python is a programming language that lets you work more quickly and integrate systems more effectively. You can learn to use Python and see almost immediate gains in productivity and lower maintenance costs.
October 15th, 2012, 03:15 AM