E-Democracy Pages Wiki

Search Wiki

 

Tools

 

Difference between revisions of "Managing Development in GroupServer and E-Democracy"

From E-Democracy.org

(Created page with "=Buildout= GroupServer uses Buildout for building (both production and development), retrieving code, and managing development. If you have never used Buildout before, [http://v...")
 
(Copy Files into the New Egg)
 
(16 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
GroupServer is a mature open source project which uses a full set of tools for managing development.
 +
 
=Buildout=
 
=Buildout=
  
Line 8: Line 10:
  
 
==Files==
 
==Files==
 +
 +
GroupServer and E-Democracy use a number of Buildout config files to manage building an instance and developing code.
  
 
===buildout.cfg===
 
===buildout.cfg===
Line 21: Line 25:
 
Contains a list that tells Buildout how to retrieve the source for an egg. Every egg from GroupServer and E-Democracy should have an entry in here indicating the address to that egg's repository, and version control tool used on that repository.
 
Contains a list that tells Buildout how to retrieve the source for an egg. Every egg from GroupServer and E-Democracy should have an entry in here indicating the address to that egg's repository, and version control tool used on that repository.
  
Technically, this isn't a Buildout file. It is a file used by [Mr. Developer https://pypi.python.org/pypi/mr.developer], a Buildout extension.  
+
In addition, this file tells Buildout to use the [https://pypi.python.org/pypi/mr.developer Mr. Developer] extension.
  
 
===versions.cfg===
 
===versions.cfg===
Line 27: Line 31:
 
This controls which version of a core egg Buildout will retrieve and build. Most core eggs will have an entry in this file that pins the egg to some version. Some 3rd party eggs will also have their versions pinned by this file.
 
This controls which version of a core egg Buildout will retrieve and build. Most core eggs will have an entry in this file that pins the egg to some version. Some 3rd party eggs will also have their versions pinned by this file.
  
 +
==Managing the Buildout Files==
  
=Issue Tracking - Redmine=
+
The above files need to be updated whenever a new egg is created or a new version of an egg is deployed. Because the above files are critical to building and developing GroupServer and E-Democracy, and they are updated pretty often, [https://github.com/e-democracy/buildout-config a repository has been created on Github] to keep E-Democracy's developers' Buildout files up to date.
  
Don’t worry if you get a bad certificate warning. That's normal.
+
Assuming you've followed the instructions for [[Installing GroupServer and E-Democracy]], you should already have a git repository setup for the Buildout files on your instance.  
  
Issue/bug tracking, and a place to put ideas for the future. Register for an account, then bug techteam@e-democracy.org to OK your registration.
+
To update the Buildout files, simply run <code>git pull</code> in the root folder of your E-Democracy instance.
  
E-Democracy Redmine -https://redmine.iopen.net/projects/edem
+
If you make a change to any of the Buildout files, simply do a <code>git add .</code>, <code>git commit</code>, and <code>git push</code> to publish those changes.
  
GroupServer Redmine - https://redmine.iopen.net/projects/groupserver
+
==Building==
  
=Repository - RhodeCode=
+
Building GroupServer/E-Democracy is one of the two things you will most often do with Buildout (checking out code for active development is the other, and is covered [[#Checking_out_Existing_Eggs|below]].)
  
https://source.iopen.net/groupserver - don’t worry if you get a bad certificate warning.
+
To build E-Democracy and Groupserver, simply run the following from the root folder of your instance of E-Democracy:
 +
<pre>./rebuild.sh</pre>
  
A mercurial repository where development code and documentation lives, can be browsed and downloaded from, and where Buildout pulls development code from.
+
=Code Repositories=
Setting up Mercurial
+
  
In ~/.hgrc, add the following:
+
E-Democracy and GroupServer use a few repository locations and tools for storing code. The differences in location and tools roughly correspond to the division between E-Democracy specific code and GroupServer code.
  
[ui]
+
==GroupServer Repository==
  
username = USERNAME <EMAILADDRESS>
+
===source.iopen.net - Mercurial===
  
[hostfingerprints]
+
All code for GroupServer core is stored at https://source.iopen.net/groupserver and uses [http://mercurial.selenic.com/ Mercurial] for version control. Each egg in GroupServer will have a corresponding repository.
  
source.iopen.net = CE:CC:DF:DF:E6:32:89:E3:8A:CD:0B:A1:8F:15:FC:B7:0A:AC:23:D3
+
Eggs that are part of GroupServer core almost always start with either 'gs' or 'Products'.
  
=Checking Out Eggs=
+
==E-Democracy Repositories==
  
This will checkout an egg into the src directory. Once checked out, changes to the files in src/<egg> will be reflected in the instance running on your machine. Generally, template changes are reflected immediately, while python changes will require a restart of uwsgi.
+
===Github - Git===
  
Checked out eggs under src/ are part of a Mercurial project. If eggs have been checkout for a while, you will need to do am hg pull on them before further development or pushing changes. Also, eggs under src/ are not updated when ./bin/buildout is called.
+
Most new code written by E-Democracy is stored at https://github.com/e-democracy and uses [http://git-scm.com/ Git] for version control. Like GroupServer, each E-Democracy egg will have a corresponding repository.
  
Step 0: Add ‘develop.cfg’ to the extends line in the [buildout] section of buildout.cfg
+
In general, any new eggs created by E-Democracy will have its repository stored on Github. These eggs will almost always start with 'edem'.
  
Step 1: ./bin/develop checkout <EGG NAME>
+
===source.iopen.net - Mercurial===
  
Step 2: ./bin/buildout -N
+
Some E-Democracy code is stored at https://source.iopen.net/ogn/edem and uses [http://mercurial.selenic.com/ Mercurial] for version control. Eggs with repositories here usually were created by [https://onlinegroups.net/ OnlineGroups.Net] (OGN) when E-Democracy would contract with OGN to write custom code.
  
Step 3: Restart uwsgi
+
Of the eggs which have repositories on source.iopen.net, the most important is gs.skin.ogn.edem (which is an E-Democracy egg despite starting with 'gs').
  
=Pushing and Pulling Checked Out Code=
+
==Registering for Repository Access==
  
This is pushing and pulling from Rhode Code or Github, which is the development repo, not the repo from which eggs are pulled during a buildout.
+
In order to commit code to either GroupServer's or E-Democracy's repositories on source.iopen.net, you will need to [https://source.iopen.net/_admin/register register an account on the site]. Once you have registered, email techteam [at] e-democracy.org to confirm your registration (you will not be able to commit until confirming registration).
  
This is simply interacting with Mercurial or Git. If you’ve ever used Mercurial or Git before, you already know this.
+
For E-Democracy's repositories on Github, you can either request permission to commit to the repository(ies) by emailing techteam [at] e-democracy.org, or you can fork the repository of interest and send us a [https://help.github.com/articles/using-pull-requests pull request].
  
Pull
+
==Setting up SSL for Mercurial Repositories==
  
Step 1: cd src/<EGG NAME>
+
Because iopen.net's SSL certificate is not signed by a certificate authority, Mercurial will default to displaying an error when trying to send or receive code from the repositories on source.iopen.net. This can be prevented by adding the following to ~/.hgrc:
  
Step 2: hg pull -or- git pull
+
<pre>
 +
[hostfingerprints]
 +
source.iopen.net = CE:CC:DF:DF:E6:32:89:E3:8A:CD:0B:A1:8F:15:FC:B7:0A:AC:23:D3
 +
</pre>
  
Push
+
=Issue Tracking=
  
Step 0: Make sure you have an account on RhodeCode
+
Because GroupServer and forums.e-democracy.org run on code that spans 100+ modules/repositories, both projects maintain project level Redmine instances for issue tracking/ticketing. The vast majority of tickets should be submitted to one of the Redmine instances below.
  
Step 1: hg add <ANY NEW FILES>
+
For non-GroupServer projects (e.x. [https://github.com/e-democracy/edemsignups edemsignups]) it is much more likely that we will use an issue tracker associated directly with the project's repository.
  
Step 2: hg commit
+
==Redmine==
  
Step 3: hg push (you’ll be asked for your RhodeCode username/password)
+
There are two instances of Redmine to be aware of:
  
=Updating the Egg Repo=
+
* E-Democracy Redmine -https://redmine.iopen.net/projects/edem - Tickets related specificly to forums.e-democracy.org. Most problems that you encounter will probably become tickets in here.
 +
* GroupServer Redmine - https://redmine.iopen.net/projects/groupserver - Tickets related to the GroupServer platform. In general, it is best to ask about your problem in the [http://groupserver.org/groups/development GroupServer Development group] before submitting a ticket here.
  
http://eggs.iopen.net/groupserver/base/ and http://eggs.iopen.net/groupserver/custom/
+
Because iopen.net's SSL certificate is not signed by a certificate authority, you will probably see a warning from your browser about the site's certificate. This is entirely normal, and you should disregard this warning.
  
This causes the creation/updating of eggs on the egg repo - which is what Buildout pulls from.
+
You will need to [https://redmine.iopen.net/account/register register an account on redmine.iopen.net]. Once you have registered, you will also need to email techteam [at] e-democracy.org so we can confirm your registration.
  
Step 0: Have ./bin/fab and ./fabfile.py
+
===Estimating Project Time Requirements===
  
Step 1: cd MYBUILDDIR
+
GroupServer's Redmine includes a couple of useful advice pages. https://redmine.iopen.net/projects/groupserver/wiki/TimeEstimation provides a great guide on how to estimate the amount of time a certain programming task will take (assuming design and specs are already established.)
  
Step 2: ./bin/fab deployegg:THE.COMPONENT.NAME,custom
+
=Deployed Eggs Server=
  
Only put the ,custom there if it is an egg specific to a customer ... in your case that would be if it is an egg that has 'edem' anywhere in the name. Otherwise, leave it off.
+
The source repositories for eggs are outlined above. These, however, are not where Buildout fetches eggs from during a build.
  
Step 3: If you’re planning to deploy this egg take note of the version string it produces, as you’ll need that for deploying new/updated eggs into installations.
+
Buildout instead fetches eggs from http://eggs.iopen.net/groupserver/, a server that hosts deployed versions of eggs. Deployed GroupServer eggs can be found in http://eggs.iopen.net/groupserver/base/, while E-Democracy eggs can be found in http://eggs.iopen.net/groupserver/custom/.
  
==Figuring out Egg Version Numbers==
+
=Editing Eggs=
  
The format of version strings is <version>-<datetime>-<hg/git hash>
+
The code for eggs involved in a build can not be directly edited. Instead, there is a process for managing the editing of eggs.
  
<version> is simply the version of the egg as defined in version.py
+
==Checking out Existing Eggs==
  
<datetime> is the UTC date and time that the egg was deployed, in YYYYMMDDHHMMSS.
+
In order to edit the eggs used in E-Democracy and GroupServer, you need to explicitly check out the egg for editing. This is done by running the following command from the root folder of your E-Democracy instance:
  
<hg hash> is the hash generated for the repo version that the egg version represents. It will be the portion of the hg version string seen on the changelog page that follows the colon (:).
+
<code>./bin/develop co <EGG NAME></code>
  
=Dev Server=
+
Doing this will cause Mr. Develop to fetch and place a folder in the src/ directory of your instance of E-Democracy that contains a version controlled repository for the egg's code. This will also tell Buildout to use the code in src/ for that egg during a build instead of the version of the egg specified in versions.cfg or custom.cfg. This, however, does not actually cause a rebuild to occur, so you will need to a <code>./bin/buildout -N</code>.
  
dev.forums.e-democracy.org
+
Once checked out, changes to the egg's files in src/ will be reflected in your instance of E-Democracy Generally, changes to templates will be reflected immediately, while changes to source files will require a restart of Zope.
  
DB is updated regularly via cron based on a backup of the live db
+
==Pushing and Pulling Code Changes==
  
can ssh in: deploy@ or root@
+
Since the folders fetched by Mr. Develop are version controlled (with master repository's URL defined in develop.cfg), you simply use [http://mercurial.selenic.com/ Mercurial] or [http://git-scm.com/ Git] (depending on the egg) to manage changes to the code. If you know how to pull, add, commit, and push with these, then you're set.
  
dev.forums.e-democracy.org needs to have your public key
+
==Deploying a new Version of an Egg==
  
==Deploying eggs to this server==
+
Once you have written code that is ready to be used by others, you are ready to deploy a new version of the egg you have been working on. Doing this will package up and upload a new version of the egg to the Deployed Eggs Server described above.
  
git pull
+
In order to deploy eggs to the Deployed Eggs Server, you need to have a copy of fabfile.py in the root folder of your E-Democracy instance. If you do not have this file, contact techteam [at] e-democracy.org to get a copy.
  
./bin/buildout -N
+
To actually deploy a new version of an E-Democracy egg, run the following command in the root folder of your E-Democracy instance:
  
./reload.sh
+
<code>./bin/fab deployegg:THE.COMPONENT.NAME,custom</code>
  
 +
If the egg you are deploying a new version for is a GroupServer core egg, then run this instead:
  
=Production Deployment=
+
<code>./bin/fab deployegg:THE.COMPONENT.NAME</code>
  
Just email version strings to codedeploy@onlinegroups.net .
+
When the deployment finishes, it will provide a version string for the newly deployed version in the format <EGG NAME> = <VERSION>. For existing eggs, overwrite the egg's entry in versions.cfg or custom.cfg with this version string. For new eggs, add this string to versions.cfg (if a new GroupServer core egg) or custom.cfg (if a new E-Democracy egg), and add the egg's name to the list of eggs to include during a build in either buildout.cfg (for GroupServer core) or custom (for E-Democracy). Then [[#Managing_the_Buildout_Files|push out these changes to the Buildout config files]].
  
Send this email by Friday night (central time).
+
==Un-check out Egg==
  
=Creating a new egg=
+
Once you have pushed changes, deployed a new version of an egg, and updated Buildout's config files, it is time to tell buildout to stop using the code in src/ for the egg in question. To deactivate checked out code, run the following command from the root folder of your E-Democracy instance:
  
See also: http://groupserver.org/groups/development/guidelines/eggs
+
<code>./bin/develop d <EGG Name></code>
  
Make a directory in src for the egg
+
The egg's source will still be in src/, but it will no longer be referenced during a build. If you prefer a clean src/ directory, feel free to delete an egg's folder from src/ after you have deactivated it.
  
Copy the following files from an existing egg and edit as needed, or create these files while using the files in another egg as a reference
+
Now, rebuild: <code>./bin/buildout -N</code>
  
The docs directory
+
==Update Production's Buildout Config Files==
  
MANIFEST.in
+
The [https://github.com/e-democracy/buildout-config buildout-config] repository doesn't affect the production server that runs forums.e-democracy.org. To update these files with new egg version strings, email codedeploy [at] onlinegroups.net. Be sure to include the egg version string produced by the egg deployment, and a short description of what was changed in the deployed egg.
  
README.txt
+
==Create a New Egg==
  
Edit to be blankish, with a brief description of what the egg does.
+
Creating a new egg involves creating a new repository on source.iopen.net or Github, adding its info to the Buildout config files, copying files into the new egg, and setting the basic configuration of the egg.
  
setup.cfg
+
GroupServer also provides [http://groupserver.org/groups/development/guidelines/eggs a page describing how to create a new egg]. The process described by GroupServer is a bit different than the one described below, but also includes a lot more detail.
  
setup.py
+
===Create a new Repository===
  
Four things need to change:
+
If you're creating a new E-Democracy egg, create a new repository on Github. If you're creating a new GroupServer egg, create a new repository on source.iopen.net. In both cases, copy the URI for the new repository, you'll need that for the next step.
1. The name needs to be the same as your egg,
+
2. The description should be sane,
+
3. The namespace_packages need to reflect the hierarchy. For example, in the case of edem.group.messages.topics, namespace_packages should be set to "['edem', 'edem.group', 'edem.group.messages']". (The final directory is supposed to be missing; it is a normal package, not a namespace package.)
+
4. Set the install_requires to just 'setuptools' for now.
+
  
version.py
+
===Update the Buildout Config Files===
  
.hgignore
+
Open develop.cfg and a line for the new egg under [sources]. This new line should include the name of the new egg, the tool used for version control (hg or git), and the URI from above. Save and close.
  
Next copy an __init__.py from a namespace-directory of the original egg.
+
If the new egg is an E-Democracy egg, add the egg's name to custom-zope-eggs section of custom.cfg. If the new egg is a GroupServer egg, and the egg's name to the zope-eggs section of buildout.cfg. Save and close.
  
In Python the ability to write "from package.package.package import SomeThing" is not built into the language. It is a hack. The hack is called a "namespace package". The hack is in two parts. First, there is a "namespace_packages" entry in the setup.py. Second there is a special __init__.py in all the namespace packages. Copy this special __init__.py from *any* other package to each of the directories in the namespace.
+
Now git add ., git commit, and git push.
  
Next you need a __init__.py in the actual package (module) directory.
+
===Copy Files into the New Egg===
  
Just enter the following into the module directory __init__.py
+
Once the Buildout config files have been updated, [[#Managing_the_Buildout_Files|check out the egg]]. Note that you may receive an error along the lines of "ERROR: abort: unknown revision 'default'!". This is OK, as you will soon create the default branch.
  
# coding=utf-8
+
In the src/ folder you should find two empty_egg folders. These are templates for new eggs - one for eggs stored in git and one for eggs stored in mercurial. Copy the contents of the appropriate folder and paste them into the newly checked out repository folder.
# This space deliberately left blank
+
  
Zope stuff! Add a configure.zcml to your module directory. To start with, enter the following:
+
===Customize the README===
          <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+
          <configure xmlns="http://namespaces.zope.org/zope"
+
            xmlns:browser="http://namespaces.zope.org/browser"
+
            xmlns:five="http://namespaces.zope.org/five"
+
            i18n_domain="groupserver">
+
          </configure>
+
  
Init a repo and commit:
+
The new egg templates include a template for the README file that you would fill out. Open the README file and, at minimum, fill in the egg name, egg purpose, the block of meta data near the top (Author through Copyright), and the Resources near the bottom. At the very bottom is a space for you to associate URLs with any text above that is in the format `TEXT`_.
  
1. Change to the top folder of the egg
+
You should also write an Introduction section, and any other sections you want to at this time.
2. Run Mercurial:
+
    1. hg init .
+
    2. hg addremove
+
    3. hg ci -m"Initial import."
+
  
Finally, weave your new egg (which does nothing) into GroupServer.
+
===Set Basic Configuration===
  
Open "custom.cfg" in your top-level the GroupServer directory.
+
In the folder for your new egg, edit setup.py. In the call to setup, a number of arguments need to be changed, including:
  
Add "PACKAGENAME" to the "custom-zope-eggs" list.
+
* name - This should be the same as your egg's name
 +
* description
 +
* classifiers - Pay attention to Development Status
 +
* keywords
 +
* author
 +
* author_email
 +
* namespace_packages - This need to reflect the hierarchy. For example, in the case of edem.group.messages.topics, namespace_packages should be set to "['edem', 'edem.group', 'edem.group.messages']". The final directory is supposed to be missing; it is a normal package, not a namespace package.
 +
* install_requires - Set this to just 'setuptools' for now. As time goes on, add other eggs that the new egg depends on.
 +
 
 +
Save and close.
 +
 
 +
Next, create the file structure represented by your egg's name. For example, the egg edem.group.messages.topics should have the folder structure edem/group/messages/topics. In each namespace folder (in this case, edem, group, and messages), make a copy of a special __init__.py that is included in the egg template.
 +
 
 +
Next you need a __init__.py in the actual package (module) directory. Just enter the following into the module directory __init__.py
 +
 
 +
<pre>
 +
# coding=utf-8
 +
# This space deliberately left blank
 +
</pre>
  
Open "develop.cfg" in your top-level the GroupServer directory.
+
Finally, create a basic configure.zcml in your module directory. Enter the following into configure.zcml:
  
Add "PACKAGENAME = fs PACKAGENAME" [all on one line] to the "sources" list.
+
<code>
 +
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
 +
<configure xmlns="http://namespaces.zope.org/zope"
 +
  xmlns:browser="http://namespaces.zope.org/browser"
 +
  xmlns:five="http://namespaces.zope.org/five"
 +
  i18n_domain="groupserver">
 +
</configure>
 +
</code>
  
Turn on the egg by running Mr Developer in your GroupServer directory: "./bin/develop a PACKAGENAME"
+
Basic configuration is done. Commit your changes, cd back up to the root folder of your E-Democracy instance, and do a <code>./bin/buildout -N</code>. Your instance of E-Democracy should not be running the egg you have just created (which does nothing.)
  
Rebuild.
+
=Testing Instance=
  
Start GroupServer.
+
E-Democracy maintains a publicly accessible testing instance of forums.e-democracy.org. This instance is located at http://dev.forums.e-democracy.org. The testing instance runs on a database that contains data from backups of production, making it useful for testing code on (mostly) live data. This database needs to be manually updated, so if you would like to have the database updated, please email techteam [at] e-democracy.org.
  
And you should have a new egg.
+
The testing instance also provides shell access for updating Buildout config files, running builds, and even editing code (though egg deploys are currently not possible from the testing instance.) To gain command line access to the testing instance, please email techteam [at] e-democracy.org.
  
==Estimating Project Time Requirements==
+
The testing instance also has its own ZMI, accessible at http://zmi.dev.forums.e-democracy.org/manage. However, you will again need to email techteam [at] e-democracy.org to gain access.
  
See https://redmine.iopen.net/projects/groupserver/wiki/TimeEstimation
+
[[Category:Technology:GroupServer Development Reference]]

Latest revision as of 12:48, 20 August 2013

GroupServer is a mature open source project which uses a full set of tools for managing development.

Buildout

GroupServer uses Buildout for building (both production and development), retrieving code, and managing development. If you have never used Buildout before, this 15 minute video provides an introduction. Documentation is at http://www.buildout.org/docs/index.html.

To use Buildout you will need setuptools. On Ubuntu, install setuptools via

sudo apt-get install python-setuptools

Files

GroupServer and E-Democracy use a number of Buildout config files to manage building an instance and developing code.

buildout.cfg

The main config file for Buildout. This is what Buildout looks to for directions. In GroupServer, buildout.cfg indicates what core eggs should be installed from GroupServer and from third parties. It also indicates which configuration sections should run, and defines a few.

custom.cfg

Refers to additional eggs developed for projects downstream from GroupServer. This which custom eggs should be installed, as well as which versions of those eggs. In general, anything written specifcally for E-Democracy will be referred to in this file.

develop.cfg

Contains a list that tells Buildout how to retrieve the source for an egg. Every egg from GroupServer and E-Democracy should have an entry in here indicating the address to that egg's repository, and version control tool used on that repository.

In addition, this file tells Buildout to use the Mr. Developer extension.

versions.cfg

This controls which version of a core egg Buildout will retrieve and build. Most core eggs will have an entry in this file that pins the egg to some version. Some 3rd party eggs will also have their versions pinned by this file.

Managing the Buildout Files

The above files need to be updated whenever a new egg is created or a new version of an egg is deployed. Because the above files are critical to building and developing GroupServer and E-Democracy, and they are updated pretty often, a repository has been created on Github to keep E-Democracy's developers' Buildout files up to date.

Assuming you've followed the instructions for Installing GroupServer and E-Democracy, you should already have a git repository setup for the Buildout files on your instance.

To update the Buildout files, simply run git pull in the root folder of your E-Democracy instance.

If you make a change to any of the Buildout files, simply do a git add ., git commit, and git push to publish those changes.

Building

Building GroupServer/E-Democracy is one of the two things you will most often do with Buildout (checking out code for active development is the other, and is covered below.)

To build E-Democracy and Groupserver, simply run the following from the root folder of your instance of E-Democracy:

./rebuild.sh

Code Repositories

E-Democracy and GroupServer use a few repository locations and tools for storing code. The differences in location and tools roughly correspond to the division between E-Democracy specific code and GroupServer code.

GroupServer Repository

source.iopen.net - Mercurial

All code for GroupServer core is stored at https://source.iopen.net/groupserver and uses Mercurial for version control. Each egg in GroupServer will have a corresponding repository.

Eggs that are part of GroupServer core almost always start with either 'gs' or 'Products'.

E-Democracy Repositories

Github - Git

Most new code written by E-Democracy is stored at https://github.com/e-democracy and uses Git for version control. Like GroupServer, each E-Democracy egg will have a corresponding repository.

In general, any new eggs created by E-Democracy will have its repository stored on Github. These eggs will almost always start with 'edem'.

source.iopen.net - Mercurial

Some E-Democracy code is stored at https://source.iopen.net/ogn/edem and uses Mercurial for version control. Eggs with repositories here usually were created by OnlineGroups.Net (OGN) when E-Democracy would contract with OGN to write custom code.

Of the eggs which have repositories on source.iopen.net, the most important is gs.skin.ogn.edem (which is an E-Democracy egg despite starting with 'gs').

Registering for Repository Access

In order to commit code to either GroupServer's or E-Democracy's repositories on source.iopen.net, you will need to register an account on the site. Once you have registered, email techteam [at] e-democracy.org to confirm your registration (you will not be able to commit until confirming registration).

For E-Democracy's repositories on Github, you can either request permission to commit to the repository(ies) by emailing techteam [at] e-democracy.org, or you can fork the repository of interest and send us a pull request.

Setting up SSL for Mercurial Repositories

Because iopen.net's SSL certificate is not signed by a certificate authority, Mercurial will default to displaying an error when trying to send or receive code from the repositories on source.iopen.net. This can be prevented by adding the following to ~/.hgrc:

[hostfingerprints]
source.iopen.net = CE:CC:DF:DF:E6:32:89:E3:8A:CD:0B:A1:8F:15:FC:B7:0A:AC:23:D3

Issue Tracking

Because GroupServer and forums.e-democracy.org run on code that spans 100+ modules/repositories, both projects maintain project level Redmine instances for issue tracking/ticketing. The vast majority of tickets should be submitted to one of the Redmine instances below.

For non-GroupServer projects (e.x. edemsignups) it is much more likely that we will use an issue tracker associated directly with the project's repository.

Redmine

There are two instances of Redmine to be aware of:

Because iopen.net's SSL certificate is not signed by a certificate authority, you will probably see a warning from your browser about the site's certificate. This is entirely normal, and you should disregard this warning.

You will need to register an account on redmine.iopen.net. Once you have registered, you will also need to email techteam [at] e-democracy.org so we can confirm your registration.

Estimating Project Time Requirements

GroupServer's Redmine includes a couple of useful advice pages. https://redmine.iopen.net/projects/groupserver/wiki/TimeEstimation provides a great guide on how to estimate the amount of time a certain programming task will take (assuming design and specs are already established.)

Deployed Eggs Server

The source repositories for eggs are outlined above. These, however, are not where Buildout fetches eggs from during a build.

Buildout instead fetches eggs from http://eggs.iopen.net/groupserver/, a server that hosts deployed versions of eggs. Deployed GroupServer eggs can be found in http://eggs.iopen.net/groupserver/base/, while E-Democracy eggs can be found in http://eggs.iopen.net/groupserver/custom/.

Editing Eggs

The code for eggs involved in a build can not be directly edited. Instead, there is a process for managing the editing of eggs.

Checking out Existing Eggs

In order to edit the eggs used in E-Democracy and GroupServer, you need to explicitly check out the egg for editing. This is done by running the following command from the root folder of your E-Democracy instance:

./bin/develop co <EGG NAME>

Doing this will cause Mr. Develop to fetch and place a folder in the src/ directory of your instance of E-Democracy that contains a version controlled repository for the egg's code. This will also tell Buildout to use the code in src/ for that egg during a build instead of the version of the egg specified in versions.cfg or custom.cfg. This, however, does not actually cause a rebuild to occur, so you will need to a ./bin/buildout -N.

Once checked out, changes to the egg's files in src/ will be reflected in your instance of E-Democracy Generally, changes to templates will be reflected immediately, while changes to source files will require a restart of Zope.

Pushing and Pulling Code Changes

Since the folders fetched by Mr. Develop are version controlled (with master repository's URL defined in develop.cfg), you simply use Mercurial or Git (depending on the egg) to manage changes to the code. If you know how to pull, add, commit, and push with these, then you're set.

Deploying a new Version of an Egg

Once you have written code that is ready to be used by others, you are ready to deploy a new version of the egg you have been working on. Doing this will package up and upload a new version of the egg to the Deployed Eggs Server described above.

In order to deploy eggs to the Deployed Eggs Server, you need to have a copy of fabfile.py in the root folder of your E-Democracy instance. If you do not have this file, contact techteam [at] e-democracy.org to get a copy.

To actually deploy a new version of an E-Democracy egg, run the following command in the root folder of your E-Democracy instance:

./bin/fab deployegg:THE.COMPONENT.NAME,custom

If the egg you are deploying a new version for is a GroupServer core egg, then run this instead:

./bin/fab deployegg:THE.COMPONENT.NAME

When the deployment finishes, it will provide a version string for the newly deployed version in the format <EGG NAME> = <VERSION>. For existing eggs, overwrite the egg's entry in versions.cfg or custom.cfg with this version string. For new eggs, add this string to versions.cfg (if a new GroupServer core egg) or custom.cfg (if a new E-Democracy egg), and add the egg's name to the list of eggs to include during a build in either buildout.cfg (for GroupServer core) or custom (for E-Democracy). Then push out these changes to the Buildout config files.

Un-check out Egg

Once you have pushed changes, deployed a new version of an egg, and updated Buildout's config files, it is time to tell buildout to stop using the code in src/ for the egg in question. To deactivate checked out code, run the following command from the root folder of your E-Democracy instance:

./bin/develop d <EGG Name>

The egg's source will still be in src/, but it will no longer be referenced during a build. If you prefer a clean src/ directory, feel free to delete an egg's folder from src/ after you have deactivated it.

Now, rebuild: ./bin/buildout -N

Update Production's Buildout Config Files

The buildout-config repository doesn't affect the production server that runs forums.e-democracy.org. To update these files with new egg version strings, email codedeploy [at] onlinegroups.net. Be sure to include the egg version string produced by the egg deployment, and a short description of what was changed in the deployed egg.

Create a New Egg

Creating a new egg involves creating a new repository on source.iopen.net or Github, adding its info to the Buildout config files, copying files into the new egg, and setting the basic configuration of the egg.

GroupServer also provides a page describing how to create a new egg. The process described by GroupServer is a bit different than the one described below, but also includes a lot more detail.

Create a new Repository

If you're creating a new E-Democracy egg, create a new repository on Github. If you're creating a new GroupServer egg, create a new repository on source.iopen.net. In both cases, copy the URI for the new repository, you'll need that for the next step.

Update the Buildout Config Files

Open develop.cfg and a line for the new egg under [sources]. This new line should include the name of the new egg, the tool used for version control (hg or git), and the URI from above. Save and close.

If the new egg is an E-Democracy egg, add the egg's name to custom-zope-eggs section of custom.cfg. If the new egg is a GroupServer egg, and the egg's name to the zope-eggs section of buildout.cfg. Save and close.

Now git add ., git commit, and git push.

Copy Files into the New Egg

Once the Buildout config files have been updated, check out the egg. Note that you may receive an error along the lines of "ERROR: abort: unknown revision 'default'!". This is OK, as you will soon create the default branch.

In the src/ folder you should find two empty_egg folders. These are templates for new eggs - one for eggs stored in git and one for eggs stored in mercurial. Copy the contents of the appropriate folder and paste them into the newly checked out repository folder.

Customize the README

The new egg templates include a template for the README file that you would fill out. Open the README file and, at minimum, fill in the egg name, egg purpose, the block of meta data near the top (Author through Copyright), and the Resources near the bottom. At the very bottom is a space for you to associate URLs with any text above that is in the format `TEXT`_.

You should also write an Introduction section, and any other sections you want to at this time.

Set Basic Configuration

In the folder for your new egg, edit setup.py. In the call to setup, a number of arguments need to be changed, including:

  • name - This should be the same as your egg's name
  • description
  • classifiers - Pay attention to Development Status
  • keywords
  • author
  • author_email
  • namespace_packages - This need to reflect the hierarchy. For example, in the case of edem.group.messages.topics, namespace_packages should be set to "['edem', 'edem.group', 'edem.group.messages']". The final directory is supposed to be missing; it is a normal package, not a namespace package.
  • install_requires - Set this to just 'setuptools' for now. As time goes on, add other eggs that the new egg depends on.

Save and close.

Next, create the file structure represented by your egg's name. For example, the egg edem.group.messages.topics should have the folder structure edem/group/messages/topics. In each namespace folder (in this case, edem, group, and messages), make a copy of a special __init__.py that is included in the egg template.

Next you need a __init__.py in the actual package (module) directory. Just enter the following into the module directory __init__.py

# coding=utf-8
# This space deliberately left blank

Finally, create a basic configure.zcml in your module directory. Enter the following into configure.zcml:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <configure xmlns="http://namespaces.zope.org/zope"

 xmlns:browser="http://namespaces.zope.org/browser"
 xmlns:five="http://namespaces.zope.org/five"
 i18n_domain="groupserver">

</configure>

Basic configuration is done. Commit your changes, cd back up to the root folder of your E-Democracy instance, and do a ./bin/buildout -N. Your instance of E-Democracy should not be running the egg you have just created (which does nothing.)

Testing Instance

E-Democracy maintains a publicly accessible testing instance of forums.e-democracy.org. This instance is located at http://dev.forums.e-democracy.org. The testing instance runs on a database that contains data from backups of production, making it useful for testing code on (mostly) live data. This database needs to be manually updated, so if you would like to have the database updated, please email techteam [at] e-democracy.org.

The testing instance also provides shell access for updating Buildout config files, running builds, and even editing code (though egg deploys are currently not possible from the testing instance.) To gain command line access to the testing instance, please email techteam [at] e-democracy.org.

The testing instance also has its own ZMI, accessible at http://zmi.dev.forums.e-democracy.org/manage. However, you will again need to email techteam [at] e-democracy.org to gain access.

 

Home - Mobile - Forums - Wiki - Blog - About - Help - Contact - People - Donate - Rules - Archives