An Evaluation Dashboard would be a page, or a group of pages, that E-Democracy staff would use to view statistics about the platform (forums.e-democracy.org), groups on the platform, groups of groups on the platform (e.g. all groups in neighborhood groups in Minneapolis), and users on the platform.
As an organization with ambitious goals and limited resources, we need to use data to drive decisions about what actions to take where.
Where and Who
A site wide evaluation dashboard could be provided at forums.e-democracy.org/evaluation/. Sub pages could display specific measures in greater detail.
Each group could also have a group level display of statistics at forums.e-democracy.org/groups/<groupId>/evaluation/. Again, sub pages could display specific measures in greater detail.
The primary audience for this would be E-Democracy staff. Depending on the nature of information displayed, or the server resources required to display, these pages could be made to only be accessible to a limited group of people, which would initially include only staff and forum managers/coaches. Outside researchers could be an additional audience, and could be granted access on request.
Statistics and Comparisons
A long list of desired statistics and evaluations can be found in the Google Doc Metrics, Research, and Evaluation Execution Plan - Detailed. In summary, these statistics include:
- Membership statistics: # of group/site members; # of joiners/leavers in a time period
- Demographics: education; household income, race, ethnicity, etc...
- Registration statistics: # who started registration; # who completed registration; # of unverified email addresses
- Posting statistics: # of people posting; average # of posts per user; % of authors
- Topics statistics: # of new topics; # of active existing topics; average # of posts per topic; most common keywords
- Technology statistics: % of web users; % of digest users; % of email only users
In most cases, a user should be able to view these statistics in aggregate across the site, for a single group, and in aggregate for a group of groups. A user should be able to see comparison displays; such as the registration statistics for a single group compared to the registration statistics of all groups in St. Paul, or % of authors compared to % of digest users.
In most cases, a user should also be able to view a time series of these statistics, include a time series of aggregate statistics. A user should be able to include time series into comparison displays.
Beyond statistics, a set of information about groups and the site that is useful includes:
- A list of group members that includes certain attributes (member organizations, email addresses, etc...)
- A list of topics or posts that include certain attributes (author, keywords, post body, etc...)
Ideally, every page related to evaluation will allow a user to download the displayed information in a CSV, and access the data in an interchange format (i.e. JSON or XML). Doing so will allow users to perform analysis beyond what is built into the evaluation pages.
A set of pages and scripts already exist that provide a level of evaluation support. In some cases, these existing tools can be modified and moved to a site/group evaluation section. In other cases, the tools would need to be rewritten to improve performance and usability.
A display of the number of posts made each month, for every group since its founding, can be viewed at http://forums.e-democracy.org/stats.html. This page also allows somebody to download the statistics as a CSV. The current page takes considerable time and server resources to display. In addition, it does not display information in a way that makes comparing multiple groups easy.
Every group has a page that displays the number of posts made each month, and the number of authors each month. This page can be found at forums.e-democracy.org/groups/<groupId>/stats.html. This page also takes some time and server resources to display, though not nearly as much as the site wide stats page. This page does not allow a user to download the statistics as a CSV, and does not allow a user to easily compare one group to another group.
Every group also has a more detailed set of statistics available at forums.e-democracy.org/groups/<groupId>/historicalstats.html. This page includes statistics about memberships, topics, and posts, for every month since the group was founded. This page does not allow users to download statistics as a CSV, and does not allow users to compare groups to other groups.
In the ZMI, /sites/edem/management_scripts/ provides useful scripts for administrators, including a few evaluation scripts. A number of scripts meant to output information on group membership or group posts can be consolidated and moved into an evaluation section on the website. These include
- reports/group_member* - Various scripts that generate lists of group members
- reports/post_detail* - Various scripts that produce lists of posts in a group