Hey Guys,
My name is Tim Patrick, and through Google Summer of Code I will be working on implementing a plugin repository with the goals of having easy one click installation of all plugins, modeled off the package manager Synaptic for Linux.
Since I am not only developing this system for the users but also for the plugin developers, I would like your input in what you would like to see in the whole plugin management, as I want to make something that you guys want to use, and not something that annoys and frustrates you so much you will want to kill me
I will briefly go over my idea: (Actually I just copied most of it from my proposal )
The repositories exist of two types - public and private.
Public repositories, allow anyone to upload plugins to the repository. An option can be configured, which prevents the plugin from being displayed until a moderator verifies its integrity.
Private repositories, on the other hand, only allow specific users to upload plugins to the repository.
These would be optionally approved by a moderator, depending on a configuration value in the conf_values table. As well, private repositories can also limit who is able to connect to and download from their repository. This would be accomplished by a unique key hash that would be downloaded to an approved site, that would log the site in and verify it. These key hashes would be stored in the repository database.
As mentioned in the mailing list, there was some question as to why an administrator would require this functionality. In my opinion it is always better to allow for new functionality, instead of limiting the potential.
An administrator may want this functionality because they only want to develop a plugin for a friend or for a group of people. However this does go against the concept of open source, so it may be better not to implement it at all.
Verification:
To verify that the plugins are safe, the main Geeklog repository will digitally sign the plugin to guarantee that the plugin is genuine. When a plugin is selected to be downloaded, if no verification is present, a warning will be displayed that alerts the administrator that the plugin may be malicious, and to click OK to consent to install it. For digitally signed plugins, the installer will determine if the signing is valid and from Geeklog, and then install the plugin. If however the signing is not valid, it will act as if the plugin is not signed at all and display a warning like above. For an example of this model, the Mozilla Add-ons site is a good example.
The reason digital signing is better than a checksum, as digital signing verifies that the plugin is not malicious, and is authenticated by the main Geeklog site. Checksumming, on the other hand, simply verifies that the integrity of the data has not been modified, which doesn't care if the data is malicious or not, but simply if it has been modified.
Blacklisting:
Repositories that contain malicious plugins can be black listed by the main Geeklog site. This would work much like Google's blacklisting for malicious sites. Users would be able to alert Geeklog of malicious repositories, and if Geeklog verified these as malicious, would be added to the Blacklist. This list would be downloaded every time the plugin manager is started, and if a user attempts to install from one of these malicious repositories, they would be alerted of the danger, and the reason the site was black listed. However they would not be prevented from installing a plugin from that repository if they wanted to.
As well, if they had a blacklisted repository in their repository list, a warning would be displayed every time the user started the plugin manager, with a request to delete the offending repository from the list.
How the repository works:
The repository consists of two separate sections. One is a database table that contains the locations of the plugins that are available, as well as other revelant data (name, documentation, etc). The other section actually holds the plugin data. Addition of a plugin is simple, using the addition tool, that will allow a repository administrator to either upload a new plugin, or verify a submitted plugin, which then will copy the plugin data to the plugin data section, while adding an entry to the plugin repository database.
Deletion from the repository works the other way, where the plugin is removed from the plugin section, and the entry is removed from the repository database.
Updates to any plugins are stored in the updates section, and listed in the updates database. The updates database state the files that have to be updated in the plugin, as well as the location of that update.
Plugin Manager:
The plugin manager, modelled off Synaptic (GNOME Package Manager that uses apt-get for Linux), would be where the administrator would be able to install plugins from the repositories. It also allows you to check for updates, and install them, as well as remove any plugins installed by the manager.
Adding and Removing Plugins:
The plugin manager has a list of repository locations, which can have repositories added or removed from the list. When the plugin manager is loaded, the repositories are checked for any new plugins, and the list of the plugins is then displayed. These plugins have a description as well with them. (Much like how synaptic works).
The administrator can then check the check box next to each plugin he/she wants added, or un-check checked boxes to remove them.
The administrator then would click to submit changes, and if removing any, a warning would be shown for each one to be removed.
When the plugin manager loads, the user is given a choice to either scroll through all listed plugins, or search through the entire plugin database. This search is not only for a name, but for keywords about the plugins functionality.
As well, plugins can be grouped according to plugin type, for example: Multimedia, Time Tracking, etc.
Update Manager:
The update manager is called from the plugin manager, and checks the repository update list for any new updates. If some are found, the user is notified of the update, as well as the severity of the update. (Eg. High Security Update). The user can then uncheck updates they do not want installed, and then can hit the update buttons to have the update performed. The updates are performed by reading the update file, and then downloading the appropriate files and overwriting the existing ones based on the instructions in the update file.
To make sure that once an update has been installed, the update is no longer shown as needing to be installed, once the update list is downloaded from the repository, each updates' unique id (AUTO_INCREMENT field) is checked against the updated table, that lists every update that has already been applied by their unique id. If an updates's unique id is in the updated table, that update is not displayed.
Perhaps this method of updating plugins can be modified to allow for Geeklog updates to the various sites, much like how the Update Manager in Linux will update to a new distribution.
-- End Proposal
My idea was that every plugin would have a configuration file that has the steps for installation, removal, etc which the plugin manager will then simply read and execute.
eg.
file:: myplugin.config.inst
<install>
getxtract myplugin/plugin.tar.gz
mov pass.txt $ROOT/pass.txt
rename srclib
db.create_table myplugintbl.sql
db.add_rows mypluginrows.sql
db.add_rows conf_rows.sql
</install>
file:: myplugin.config.rem
rm $ROOT/pass.txt
db.droptable myplugintbl
etc.
This config file would have simple commands (eg. mov, rm, db.create, db.add_rows, db.remove, etc) which allows you the developer to not worry about how the installation process is done, but just lists what commands to perform.
This not only cuts down on development time, but also helps standardize the entire plugin creation process.
Other than that one file the plugin developer would have total liberty to have whatever other structure he/she wants in the plugin directory. As long as the configuration file is there.
Plugins are stored in the repository server as a folder with the config files in it (install, remove, ..) and the tar ball that contains the plugin data as well as any other data.
As in the installation process, the gextract command would get the tar ball, extract it to the plugin directory, and then run commands from the install file it has loaded from the repository.
This allows the plugin manager to simply get the config file for the action (eg. install or remove) and then following its directions only download what is needed (and in the removal process, only the config file).
This cuts down on network bandwidth and also speed of installation and removal. Plus it saves space in the database where before removal, installation, etc instructions would have to be stored for each plugin.
Anyways I would like to know your thoughts and suggestions with this idea?
Ideas, improvements? Criticism?
Also how do you like the current GL plugin system. If you had the time what would you do to improve it? What have you always wanted or wished was never there?
- Tim
My name is Tim Patrick, and through Google Summer of Code I will be working on implementing a plugin repository with the goals of having easy one click installation of all plugins, modeled off the package manager Synaptic for Linux.
Since I am not only developing this system for the users but also for the plugin developers, I would like your input in what you would like to see in the whole plugin management, as I want to make something that you guys want to use, and not something that annoys and frustrates you so much you will want to kill me
I will briefly go over my idea: (Actually I just copied most of it from my proposal
)The repositories exist of two types - public and private.
Public repositories, allow anyone to upload plugins to the repository. An option can be configured, which prevents the plugin from being displayed until a moderator verifies its integrity.
Private repositories, on the other hand, only allow specific users to upload plugins to the repository.
These would be optionally approved by a moderator, depending on a configuration value in the conf_values table. As well, private repositories can also limit who is able to connect to and download from their repository. This would be accomplished by a unique key hash that would be downloaded to an approved site, that would log the site in and verify it. These key hashes would be stored in the repository database.
As mentioned in the mailing list, there was some question as to why an administrator would require this functionality. In my opinion it is always better to allow for new functionality, instead of limiting the potential.
An administrator may want this functionality because they only want to develop a plugin for a friend or for a group of people. However this does go against the concept of open source, so it may be better not to implement it at all.
Verification:
To verify that the plugins are safe, the main Geeklog repository will digitally sign the plugin to guarantee that the plugin is genuine. When a plugin is selected to be downloaded, if no verification is present, a warning will be displayed that alerts the administrator that the plugin may be malicious, and to click OK to consent to install it. For digitally signed plugins, the installer will determine if the signing is valid and from Geeklog, and then install the plugin. If however the signing is not valid, it will act as if the plugin is not signed at all and display a warning like above. For an example of this model, the Mozilla Add-ons site is a good example.
The reason digital signing is better than a checksum, as digital signing verifies that the plugin is not malicious, and is authenticated by the main Geeklog site. Checksumming, on the other hand, simply verifies that the integrity of the data has not been modified, which doesn't care if the data is malicious or not, but simply if it has been modified.
Blacklisting:
Repositories that contain malicious plugins can be black listed by the main Geeklog site. This would work much like Google's blacklisting for malicious sites. Users would be able to alert Geeklog of malicious repositories, and if Geeklog verified these as malicious, would be added to the Blacklist. This list would be downloaded every time the plugin manager is started, and if a user attempts to install from one of these malicious repositories, they would be alerted of the danger, and the reason the site was black listed. However they would not be prevented from installing a plugin from that repository if they wanted to.
As well, if they had a blacklisted repository in their repository list, a warning would be displayed every time the user started the plugin manager, with a request to delete the offending repository from the list.
How the repository works:
The repository consists of two separate sections. One is a database table that contains the locations of the plugins that are available, as well as other revelant data (name, documentation, etc). The other section actually holds the plugin data. Addition of a plugin is simple, using the addition tool, that will allow a repository administrator to either upload a new plugin, or verify a submitted plugin, which then will copy the plugin data to the plugin data section, while adding an entry to the plugin repository database.
Deletion from the repository works the other way, where the plugin is removed from the plugin section, and the entry is removed from the repository database.
Updates to any plugins are stored in the updates section, and listed in the updates database. The updates database state the files that have to be updated in the plugin, as well as the location of that update.
Plugin Manager:
The plugin manager, modelled off Synaptic (GNOME Package Manager that uses apt-get for Linux), would be where the administrator would be able to install plugins from the repositories. It also allows you to check for updates, and install them, as well as remove any plugins installed by the manager.
Adding and Removing Plugins:
The plugin manager has a list of repository locations, which can have repositories added or removed from the list. When the plugin manager is loaded, the repositories are checked for any new plugins, and the list of the plugins is then displayed. These plugins have a description as well with them. (Much like how synaptic works).
The administrator can then check the check box next to each plugin he/she wants added, or un-check checked boxes to remove them.
The administrator then would click to submit changes, and if removing any, a warning would be shown for each one to be removed.
When the plugin manager loads, the user is given a choice to either scroll through all listed plugins, or search through the entire plugin database. This search is not only for a name, but for keywords about the plugins functionality.
As well, plugins can be grouped according to plugin type, for example: Multimedia, Time Tracking, etc.
Update Manager:
The update manager is called from the plugin manager, and checks the repository update list for any new updates. If some are found, the user is notified of the update, as well as the severity of the update. (Eg. High Security Update). The user can then uncheck updates they do not want installed, and then can hit the update buttons to have the update performed. The updates are performed by reading the update file, and then downloading the appropriate files and overwriting the existing ones based on the instructions in the update file.
To make sure that once an update has been installed, the update is no longer shown as needing to be installed, once the update list is downloaded from the repository, each updates' unique id (AUTO_INCREMENT field) is checked against the updated table, that lists every update that has already been applied by their unique id. If an updates's unique id is in the updated table, that update is not displayed.
Perhaps this method of updating plugins can be modified to allow for Geeklog updates to the various sites, much like how the Update Manager in Linux will update to a new distribution.
-- End Proposal
My idea was that every plugin would have a configuration file that has the steps for installation, removal, etc which the plugin manager will then simply read and execute.
eg.
Text Formatted Code
file:: myplugin.config.inst
<install>
getxtract myplugin/plugin.tar.gz
mov pass.txt $ROOT/pass.txt
rename srclib
db.create_table myplugintbl.sql
db.add_rows mypluginrows.sql
db.add_rows conf_rows.sql
</install>
file:: myplugin.config.rem
rm $ROOT/pass.txt
db.droptable myplugintbl
etc.
This config file would have simple commands (eg. mov, rm, db.create, db.add_rows, db.remove, etc) which allows you the developer to not worry about how the installation process is done, but just lists what commands to perform.
This not only cuts down on development time, but also helps standardize the entire plugin creation process.
Other than that one file the plugin developer would have total liberty to have whatever other structure he/she wants in the plugin directory. As long as the configuration file is there.
Plugins are stored in the repository server as a folder with the config files in it (install, remove, ..) and the tar ball that contains the plugin data as well as any other data.
As in the installation process, the gextract command would get the tar ball, extract it to the plugin directory, and then run commands from the install file it has loaded from the repository.
This allows the plugin manager to simply get the config file for the action (eg. install or remove) and then following its directions only download what is needed (and in the removal process, only the config file).
This cuts down on network bandwidth and also speed of installation and removal. Plus it saves space in the database where before removal, installation, etc instructions would have to be stored for each plugin.
Anyways I would like to know your thoughts and suggestions with this idea?
Ideas, improvements? Criticism
?Also how do you like the current GL plugin system. If you had the time what would you do to improve it? What have you always wanted or wished was never there?
- Tim
