BLOG – Analysis

Mobility as a Service : what indicators to measure performance ?

Over the last 6 years, the concept of Mobility as a Service has become increasingly popular over the world. However, how can one measure its performance to ensure it fulfils its environmental, economic and social objectives? The Smart City Institute opens the discussion in the context of the European week of mobility, which this year focuses on intermodality.

Author : Audrey Lebas, Researcher at the Smart City Institute
Mobility as a service Image large 
Ten years ago, the concept of MaaS remained unknown. It is only in 2014 that it was popularised by the Finish Transport and Innovation agency. It is now put forwards as one of the potential answer to many economic, environmental and social challenges related to urban mobility. While MaaS has a lot to offer, it is important to ensure that it meets its expectation. Hence, this article aims to lay the basis for discussion when it comes to the key points to consider measuring its performance. Part of the reflexion is based on an exchange with Laurent Chevereau, MaaS study department at CEREMA.

What is MaaS ?

MaaS can be descripted an intelligent, integrated, user-centred information, reservation, purchase and ticketing system for mobility services.

Maas scheme SmartCityInstitute    
Visual representation of MaaS - Figure from Smart City Institute Practical Guide #4, adapted from Karmargianni & Mathyas.
Mobility as a Service aims to offer the ideal combination of transport modes for each journey by knowing the real-time network conditions (supply) and user preferences (demand). MaaS implies the integration of all mobility services available on a territory: both public transport (e.g. bus, tram, metro, train) and shared modes from private operators (e.g. car-sharing, car-pooling, bikes, scooters). In the most advanced versions, the user benefits from a package allowing him/her to use an unlimited number of mobility services.
In the context of Smart City initiatves, MaaS is often put forward as a means to achieve a more fluid and sustainable mobility by allowing a significant modal shift*. It is believed that its use would overcome a number of cognitive barriers preventing changes in mobility behaviour such as the lack of information on the available offer or the stress related to connections.
Modal shift refers to the change in the market shares of different modes of transport between them (e.g. car to bicycle).
From Helsinki to Sydney, including Antwerp and Brussels, many cities have launched MaaS initiatives over the last years, with various management and governance models.
The concept seems very appealing. However, how can one ensure that it lives up to users’ expectations? This is exactly what is at stake with performance: it is therefore essential to define appropriate indicators to ensure the good completion of the initiative.

Performance measurement: a systemic approach

In any type of Smart City project, performance measurement is important because it allows us to know where we come from, where we are and where we want to go. The Smart City Institute advocates a systemic approach with a framework that monitors and evaluate each stage of the initiative based on an appropriate data collection method.
iconeInfo In simple terms, monitoring is a process of tracking the progress of an initiative by collecting and analysing data. It allows identifying potential deviations and making the necessary adjustments. Evaluation goes beyond monitoring, aims to provide a more in-depth, and nuanced analysis.
Indicators definition is one of the key steps in this process. Hence, it is important to define indicators in the following 5 categories:
  1. Input indicators that measure the effort devoted to the pursuit of an initiative by assessing the resources needed to implement the initiatives, measuring their quantity and quality.
  2. Process indicators that show whether planned activities have taken place.
  3. Output indicators that report on immediate progress resulting from the implementation of activities.
  4. Outcome indicators that determine the extent to which the activities undertaken have achieved the medium-term objectives for the target audiences.
  5. Impact indicators that monitor the long-term effects of the initiative on the recipients and the possible effects on those not targeted by the initiative.

What indicators to measure the performance of MaaS? 

Drawing up an exhaustive list of indicators for the monitoring of MaaS would require a number of characters well beyond what this blog post allows. Indeed, the levels of integration, the management and governance models, as well as the objectives that can be attributed to MaaS are numerous and specific to each territory. However, we would like to give food for thoughts with elements that we believe are common to each territory with a MaaS project.

1 - Legislation: an underestimated input indicator?

The most common elements put forward as input indicators are human and financial resources. However, in the context of MaaS, a valuable element is the establishment of a clear legislative framework to create a fertile ground for services to develop. Therefore, potential indicators could be:
  • The mention of MaaS in the territory’s mobility objectives (yes/no);
  • The number of regulations put in place to stimulate alternative modes to private cars (#);
  • The proactive development of a policy for active mobility (soft and pedestrian) (yes/no);
  • The implementation of a parking policy that favours intermodality and/or discourages car use in the city centre (yes/no);
  • The implementation of policies that encourage mobility service providers to share their data (yes/no);
  • The number of legislations aiming at strengthening the quality and regularity of data shared by transport operators towards the citizen (#);
  • The implementation of legislation to protect the data of users and providers of mobility services (yes/no);
  • Strengthening passengers' rights (e.g. refund policy in case of cancelled services) (yes/no) ;
  • ...

2 - The ecosystem: the chore of the process

Regardless of the management and governance model chosen, MaaS projects cannot be sustained without collaboration between the private sector, the public sector (including the public transport operator), the research sphere and citizens. It is therefore essential to monitor the establishment and strengthening of collaborations and partnerships. Potential indicators could therefore include :
  • The involvement of all stakeholders in defining the objectives of the MaaS project (yes/no);
  • The number of citizen consultations carried out before, during or after the project (#);
  • The mix in the composition of the steering committee (i.e. % citizen/civil society, % private sector, % public sector);
  • The definition of roles for the different stakeholders in the development of the project (yes/no);
  • The implementation of a communication campaign towards citizens and employers in the territory (yes/no) ;
  • ...

3 - Services integration: the obvious output?

One of the primary objectives of any MaaS initiative is the integration of mobility services available on a given territory. Thereupon, integration appears to be one of the most tangible output. Nonetheless, it has to be done both at the App level and in a physical way, at the infrastructure level (e.g. mobipunt). Hence, following potential indicators can be identified:
  • The number of mobility services that can be booked via the App (#) ;
  • The share of mobility services available on the territory that can be booked via the App (%);
  • The walking time between the different mobility services (minutes);
  • The number of services respecting the API open data standards (#) ;
  • ...

4 - Qualitative indicators to measure outcome and impact?

Finally, as Laurent Chevereau explained, "it is difficult to measure the outcome and the direct impact of MaaS because the approach to it is so specific to each territory". Indeed, if, in a given territory, the objective of MaaS is primarily economic, the outcome indicators will not be the same as if the objective is environmental. What about territories where the strategic objectives have not been specified? Our expert explains: "In France, for example, MaaS initiatives were mainly initiated by public transport operators, without any real long-term strategic objectives in the territory where they are located. Territorial authorities are only starting to take up the concept.
Then, it can be difficult to define causal links between MaaS and the outcomes and impacts that result from it. Advances and improvements may be attributable to the emergence of services or other externalities (e.g. increased cost of car ownership). As described by Laurent Chevereau:

The digitalisation of transport is a logical development, with or without MaaS. It is therefore necessary to reflect on the reference situation without the MaaS project and to see what the real added value of the platform is in terms of achieving potential environmental, economic or social objectives.

As the results and impacts seem difficult to quantify, the most credible and conclusive approach would be to focus on a more qualitative and in-depth approach: "Surveys and interviews should be carried out with the various MaaS stakeholders, starting with transport users", says the CEREMA study director. The following examples of indicators could be evaluated using a Likert scale* :

* A Likert scale is a five (or seven) point scale that is used to allow the individual to express how much they agree or disagree with a particular statement.
  • User satisfaction with the integration of information and/or the integration of fares in MaaS;
  • Better understanding of the offer by those excluded from mobility;
  • Change in perception of alternative modes to the private car as a result of the adoption of MaaS;
  • Behavioural changes as a result of the integration of information in the App;

MaaS performance: what takeaways?

Mobility as a Service has increasingly been put forward as a mean to meet the economic, social and environmental challenges linked to mobility. However, as with any Smart City project, it is important to monitor it to evaluate its performance, particularly through the definition of indicators. While defining concrete input, process and output indicators may be intuitive for MaaS project managers, it may be more complicated to define outcome impact indicators. Therefore, it is important to keep in mind the central of users in evaluating these two aspects.   
Interested to learn more about the definition of indicators and, more generally, about performance measurement in Smart Cities ? Do not hesitate to consult our practical handbook entitled "Monitoring and evaluation: Management tools for our territories in transition" (in French only).

Photo credit: Charles Deluvio - Unsplash

Share this news