Chouette Core was born from a fork of Chouette in February 2016. Initially, the project served as the basis for IBOO, a next generation static data manager for Île-de-France Mobilités (which will be used to merge the public transport data of 100 operators in the Île-de-France region).
Since the fork, our team at af83/enRoute has been working on a large-scale update and refactor of the entire project. Now, after 5,000 commits, all actors supporting the Chouette fork have given their stamp of approval to publish this work under the same open-source Cecill-B license as the original.
A “core” version for development and forking
The primary goal of Chouette Core is to provide a clean code base for all other Chouette projects, by removing project-specific:
- Documentation (the core documentation is moving to project’s associated GitHub wiki)
- Deployment and dev tools (Capistrano, Docker, Guard, etc.)
- Code, i18n translations, and other resources
This cleaning work is still in progress. The af83/enRoute team manages two projects based on Chouette Core: IBOO Chouette, and our own SaaS project, EnRoute ITS Chouette. Forking and merging a 1,500-file project isn’t simple, but we’re trying to make it easier.
In addition to providing a clean code base, Chouette Core also aims to facilitate third-party code contributions. Our dev team at af83/enRoute has a CI system in place to give you quick feedback on pull requests. The same process is used in-house by the af83/enRoute team, and we work in the open with public pull requests. Check out the “How to contribute” section below for information about contributing feature ideas.
If you’re looking for an easily deployable version of Chouette, stay tuned. For the moment, you can use Chouette Core, but you’ll need to complete and configure the application to launch and deploy it. We invite you to publish your own “easy to install” Chouette version if you decide to try it out.
Two years of work
The Chouette Core project made public this week is the result of two years of intensive work, 20,000 man-hours, and 5,000 commits. During that time, a vast number of improvements were made, including a complete redesign of the user interface, and a new workflow that enables multiple teams to collaboratively operate on their data.
New user interface
The entire user interface has been completely redesigned. The af83/enRoute UX team worked for a full year to create a new visual hierarchy that allows the same kind of detailed transportation data to be manipulated more fluidly and efficiently.
The af83/enRoute UX and dev teams are continuously gathering and using user feedback to adapt, improve, and unite the user interfaces and resulting experience.’’
What Chouette v1 called a database has now been expanded into a special data group we call a Referential, which enables more fluid operations on data sets.
Referentials are used to store:
- Partial data (called a Data Set) or complete data (merged from a Workbench or aggregated from a Workgroup)
- Imported data
- Data created and edited through the interface
Referentials are used to provide:
- A merge operation to incrementally create a complete data set from partial ones
- A duplication operation to replicate past data sets, or to create backups
- Versioning (TODO) to instantly rollback to a previous state (for Workbench or Workgroup data for example)
The Merge operation is a completely new feature which gives teams a way to work on their data piece by piece.
Alice can enter the next three months of schedule data for line A from scratch. She could also duplicate a prior Data Set to a new time period, making only the modifications necessary to keep it up to date.
Bob can import a GTFS file with data for lines B and C into another Data Set.
These Data Sets are stored in Alice & Bob’s workbench. Each Data Set is independent and can be modified, duplicated, archived, and manipulated separately.
To work together, Alice & Bob can update their Workbench with finalized data by merging each of their Data Sets when it’s ready. Alice’s merged Data Set will provide data for line A for a given time period, and Bob’s will provide data for lines B and C.
Subsequently, Bob (or Alice) could retrieve the Data Set used for line A, make a change, and merge this changed Data Set. The new updates will override the previous data for line A.
Using these merge operations, the finalized Workbench data will always contain the last known best state of Alice’s & Bob’s work.
Chouette performs the merge by deeply inspecting the provided data. For example, new Routes would be created, existing ones kept, updated ones recreated, and outdated ones removed.
A new concept of “business signature” is now present for every data model in the application (for example, Route, JourneyPattern, RoutingConstraintZone, Footnote). Each one now has a signature computed by Chouette based on relevant information for each model. During a merge, two distinct Routes with the same signature are considered to be the “same” Route.
The merge operation leverages these signatures to understand what data needs to be changed to match the latest Data Set provided. Users no longer need to do this work by hand, and can prepare data independently.
Alice & Bob’s team works in a dedicated Workbench. They can additionally collaborate with other teams that share the same Workgroup.
Each team is responsible for its Workbench and the resulting finalized data. They can merge new data or rollback instantly to a prior finalized state to provide the most accurate data set to the Workgroup.
Multiple teams can provide data for the same transit lines. In this case, each team will see data sets for shared lines from other teams. With some coordination, this allows for collaborative work between teams in a Workgroup.
A Workgroup’s complete data is created by aggregating the current finalized data in each Workbench.
The Workgroup owner or administrator controls the creation of Workgroup data. If there’s ever a problem, they can rollback instantly to an earlier version of the aggregate data. A prior data set can also be chosen from a specific team’s Workbench to bypass errors until a fix is made.
The validation tool in Chouette has been refactored.
The validation engine is now completely modular. Users can build validation rule sets by selecting rules provided by Chouette (about 30 at this time). Each rule can be customized individually. The description, severity, and specialized attributes (e.g. speed, format) can be defined by the user.
Validation reports are stored in the database and provide detailed feedback results (even for millions of messages if needed).
Validations can be set to run automatically in a Workgroup’s workflow. Workbench and Workgroup administrators can define validation rule sets to use at each step in the flow:
- After an import
- Before a Workbench merge operation
- After a merge on the finalized data
- After a Workgroup aggregation operation on the entire resulting data set
- A nightly check to regularly validate unchanged workbench data
Other Chouette versions
The original AFIMB 3.4
Chouette Core est un fork de Chouette 3.2 apparu en février 2016 (référence à l’étiquette “afimb-master” dans les archive de Chouette Core). La plupart des caractéristiques d’origines de Chouette sont maintenues, améliorées et enrichies. Certaines caractéristiques sont cachées ou désactivées (voir les détails ci-dessous). Gardez votre instance open-source 3.4 si vous considérez ces caractéristiques comme primordiales. On a effectué pas mal de travail depuis neuf mois pour faire de Chouette Core un substitut fluide. Notre objectif est de réactiver la plupart des caractéristiques de la version 3.4 Par exemple, nous travaillons sur des caractéristiques d’appui de GTFS. Le REPO publique de Chouette Core est une étape importante dans ce parcour, et vos contributions sont les bienvenues.
Références : https://github.com/afimb/chouette2
This Norwegian team successfully integrated Chouette into their national static data manager. In April, the EnTur and af83/enRoute teams met and created a roadmap to integrate features needed by EnTur into Chouette Core.
Références : https://github.com/entur and https://en-tur.no
How to contribute
Chouette is a very large application that tries to manage diverse business needs. Evaluating a user story and integrating it into the context of the application as a whole often takes more time than it does to develop the corresponding feature.
Write up a description of your feature or general need, even if you have the means to develop it, and post it on the mailing list/forum. The dedicated mailing-list chouette-core is available to talk about new points in chouette-core and study the best solution (before starting any code).