The European legal framework and particularly safety directive 2004/49 introduce the necessity for railway operators to have a Safety Management System (SMS) authorised by a National Safety Agency (NSA) during a limited period. A SMS offers a systematic and precise description of the company organisation in terms of safety management. It makes explicit the organisation of all the rules, procedures, methods and technical or human means to reach the safety objectives and mitigate railways activities risks. The main challenge is to appropriately assess an “organisational” system which heavily involves human factor aspects. Depending on a lot of factors such as, for instance, the size, the operation field or the experience of the applicant, the SMS can vary significantly from one operator to another. A pure conformity check is definitely inadequate to apprehend the safety performances of a SMS, since the assessment has to be considered in a systemic way. Moreover, as every accident could show a failed organisation in the SMS, the SMS also has to be reactive and scalable during its life period. The authorisation is an a priori check; hence in practice, a future Railways Undertaking (RU) or Infrastructure Manager (IM) could engage a third party which elaborates on an effective and conformed SMS, without any know-how. A NSA must detect this situation. Indeed, this poses a risk that the procedures described in this SMS could not be (appropriately) used in practice. However, a NSA has also to supervise the SMS in the field, and tools are needed to evaluate safety performance of all stakeholder in a fair and equitable way. Namely, it is crucial to identify what does a NSA need to reach this objective, to have a whole overview of the operator in terms of conformity/respect of the regulation, safety effectiveness, maturity/anticipation capability. This paper proposes a tool based on a compliance (other called conformity), effectiveness and maturity assessment model. This model is inspired by ISO standards especially concerning the management science domain that qualifies the performances of an organisation.