Managing Development in GroupServer and E-Democracy
GroupServer is a mature open source project which uses a full set of tools for managing development.
- 1 Buildout
- 2 Code Repositories
- 3 Issue Tracking
- 4 Checking Out Eggs
- 5 Pushing and Pulling Checked Out Code
- 6 Updating the Egg Repo
- 7 Dev Server
- 8 Production Deployment
- 9 Creating a new egg
GroupServer uses Buildout for building (both production and development), retrieving code, and managing development. If you have never used Buildout before, this 15 minute video provides an introduction. Documentation is at http://www.buildout.org/docs/index.html.
To use Buildout you will need setuptools. On Ubuntu, install setuptools via
sudo apt-get install python-setuptools
GroupServer and E-Democracy use a number of Buildout config files to manage building an instance and developing code.
The main config file for Buildout. This is what Buildout looks to for directions. In GroupServer, buildout.cfg indicates what core eggs should be installed from GroupServer and from third parties. It also indicates which configuration sections should run, and defines a few.
Refers to additional eggs developed for projects downstream from GroupServer. This which custom eggs should be installed, as well as which versions of those eggs. In general, anything written specifcally for E-Democracy will be referred to in this file.
Contains a list that tells Buildout how to retrieve the source for an egg. Every egg from GroupServer and E-Democracy should have an entry in here indicating the address to that egg's repository, and version control tool used on that repository.
Technically, this isn't a Buildout file. It is a file used by Mr. Developer, a Buildout extension.
This controls which version of a core egg Buildout will retrieve and build. Most core eggs will have an entry in this file that pins the egg to some version. Some 3rd party eggs will also have their versions pinned by this file.
Managing the Buildout Files
The above files need to be updated whenever a new egg is created or a new version of an egg is deployed. Because the above files are critical to building and developing GroupServer and E-Democracy, and they are updated pretty often, a repository has been created on Github to keep E-Democracy's developers' Buildout files up to date.
Assuming you've followed the instructions for Installing GroupServer and E-Democracy, you should already have a git repository setup for the Buildout files on your instance.
To update the Buildout files, simply run
git pull in the root folder of your E-Democracy instance.
If you make a change to any of the Buildout files, simply do a
git add .,
git commit, and
git push to publish those changes.
Building GroupServer/E-Democracy is one of the two things you will most often do with Buildout (checking out code for active development is the other, and is covered below.)
To build E-Democracy and Groupserver, simply run the following from the root folder of your instance of E-Democracy:
E-Democracy and GroupServer use a few repository locations and tools for storing code. The differences in location and tools roughly correspond to the division between E-Democracy specific code and GroupServer code.
source.iopen.net - Mercurial
Eggs that are part of GroupServer core almost always start with either 'gs' or 'Products'.
Github - Git
In general, any new eggs created by E-Democracy will have its repository stored on Github. These eggs will almost always start with 'edem'.
source.iopen.net - Mercurial
Some E-Democracy code is stored at https://source.iopen.net/ogn/edem and uses Mercurial for version control. Eggs with repositories here usually were created by OnlineGroups.Net (OGN) when E-Democracy would contract with OGN to write custom code.
Of the eggs which have repositories on source.iopen.net, the most important is gs.skin.ogn.edem (which is an E-Democracy egg despite starting with 'gs').
Registering for Repository Access
In order to commit code to either GroupServer's or E-Democracy's repositories on source.iopen.net, you will need to register an account on the site. Once you have registered, email techteam [at] e-democracy.org to confirm your registration (you will not be able to commit until confirming registration).
For E-Democracy's repositories on Github, you can either request permission to commit to the repository(ies) by emailing techteam [at] e-democracy.org, or you can fork the repository of interest and send us a pull request.
Setting up SSL for Mercurial Repositories
Because iopen.net's SSL certificate is not signed by a certificate authority, Mercurial will default to displaying an error when trying to send or receive code from the repositories on source.iopen.net. This can be prevented by adding the following to ~/.hgrc:
[hostfingerprints] source.iopen.net = CE:CC:DF:DF:E6:32:89:E3:8A:CD:0B:A1:8F:15:FC:B7:0A:AC:23:D3
Because GroupServer and forums.e-democracy.org run on code that spans 100+ modules/repositories, both projects maintain project level Redmine instances for issue tracking/ticketing. The vast majority of tickets should be submitted to one of the Redmine instances below.
For non-GroupServer projects (e.x. edemsignups) it is much more likely that we will use an issue tracker associated directly with the project's repository.
There are two instances of Redmine to be aware of:
- E-Democracy Redmine -https://redmine.iopen.net/projects/edem - Tickets related specificly to forums.e-democracy.org. Most problems that you encounter will probably become tickets in here.
- GroupServer Redmine - https://redmine.iopen.net/projects/groupserver - Tickets related to the GroupServer platform. In general, it is best to ask about your problem in the GroupServer Development group before submitting a ticket here.
Because iopen.net's SSL certificate is not signed by a certificate authority, you will probably see a warning from your browser about the site's certificate. This is entirely normal, and you should disregard this warning.
You will need to register an account on redmine.iopen.net. Once you have registered, you will also need to email techteam [at] e-democracy.org so we can confirm your registration.
Estimating Project Time Requirements
GroupServer's Redmine includes a couple of useful advice pages. https://redmine.iopen.net/projects/groupserver/wiki/TimeEstimation provides a great guide on how to estimate the amount of time a certain programming task will take (assuming design and specs are already established.)
Checking Out Eggs
This will checkout an egg into the src directory. Once checked out, changes to the files in src/<egg> will be reflected in the instance running on your machine. Generally, template changes are reflected immediately, while python changes will require a restart of uwsgi.
Checked out eggs under src/ are part of a Mercurial project. If eggs have been checkout for a while, you will need to do am hg pull on them before further development or pushing changes. Also, eggs under src/ are not updated when ./bin/buildout is called.
Step 0: Add ‘develop.cfg’ to the extends line in the [buildout] section of buildout.cfg
Step 1: ./bin/develop checkout <EGG NAME>
Step 2: ./bin/buildout -N
Step 3: Restart uwsgi
Pushing and Pulling Checked Out Code
This is pushing and pulling from Rhode Code or Github, which is the development repo, not the repo from which eggs are pulled during a buildout.
This is simply interacting with Mercurial or Git. If you’ve ever used Mercurial or Git before, you already know this.
Step 1: cd src/<EGG NAME>
Step 2: hg pull -or- git pull
Step 0: Make sure you have an account on RhodeCode
Step 1: hg add <ANY NEW FILES>
Step 2: hg commit
Step 3: hg push (you’ll be asked for your RhodeCode username/password)
Updating the Egg Repo
This causes the creation/updating of eggs on the egg repo - which is what Buildout pulls from.
Step 0: Have ./bin/fab and ./fabfile.py
Step 1: cd MYBUILDDIR
Step 2: ./bin/fab deployegg:THE.COMPONENT.NAME,custom
Only put the ,custom there if it is an egg specific to a customer ... in your case that would be if it is an egg that has 'edem' anywhere in the name. Otherwise, leave it off.
Step 3: If you’re planning to deploy this egg take note of the version string it produces, as you’ll need that for deploying new/updated eggs into installations.
Figuring out Egg Version Numbers
The format of version strings is <version>-<datetime>-<hg/git hash>
<version> is simply the version of the egg as defined in version.py
<datetime> is the UTC date and time that the egg was deployed, in YYYYMMDDHHMMSS.
<hg hash> is the hash generated for the repo version that the egg version represents. It will be the portion of the hg version string seen on the changelog page that follows the colon (:).
DB is updated regularly via cron based on a backup of the live db
can ssh in: deploy@ or root@
dev.forums.e-democracy.org needs to have your public key
Deploying eggs to this server
Just email version strings to firstname.lastname@example.org .
Send this email by Friday night (central time).
Creating a new egg
Make a directory in src for the egg
Copy the following files from an existing egg and edit as needed, or create these files while using the files in another egg as a reference
The docs directory
Edit to be blankish, with a brief description of what the egg does.
Four things need to change: 1. The name needs to be the same as your egg, 2. The description should be sane, 3. The namespace_packages need to reflect the hierarchy. For example, in the case of edem.group.messages.topics, namespace_packages should be set to "['edem', 'edem.group', 'edem.group.messages']". (The final directory is supposed to be missing; it is a normal package, not a namespace package.) 4. Set the install_requires to just 'setuptools' for now.
Next copy an __init__.py from a namespace-directory of the original egg.
In Python the ability to write "from package.package.package import SomeThing" is not built into the language. It is a hack. The hack is called a "namespace package". The hack is in two parts. First, there is a "namespace_packages" entry in the setup.py. Second there is a special __init__.py in all the namespace packages. Copy this special __init__.py from *any* other package to each of the directories in the namespace.
Next you need a __init__.py in the actual package (module) directory.
Just enter the following into the module directory __init__.py
- This space deliberately left blank
Zope stuff! Add a configure.zcml to your module directory. To start with, enter the following:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <configure xmlns="http://namespaces.zope.org/zope" xmlns:browser="http://namespaces.zope.org/browser" xmlns:five="http://namespaces.zope.org/five" i18n_domain="groupserver"> </configure>
Init a repo and commit:
1. Change to the top folder of the egg 2. Run Mercurial:
1. hg init . 2. hg addremove 3. hg ci -m"Initial import."
Finally, weave your new egg (which does nothing) into GroupServer.
Open "custom.cfg" in your top-level the GroupServer directory.
Add "PACKAGENAME" to the "custom-zope-eggs" list.
Open "develop.cfg" in your top-level the GroupServer directory.
Add "PACKAGENAME = fs PACKAGENAME" [all on one line] to the "sources" list.
Turn on the egg by running Mr Developer in your GroupServer directory: "./bin/develop a PACKAGENAME"
And you should have a new egg.