Software Catalog Scorecards: Ownership Metadata, Backstage catalog-info and Engineering Standards at Scale

Every engineering organisation past about fifty services eventually asks the same question during an incident: who owns this thing? The honest answer, in most companies, is a wiki page last edited two reorganisations ago. Software catalog scorecards are the mechanism that turns that unreliable answer into a continuously verified one. A catalog records what exists and who is responsible; a scorecard grades each entry against the standards you claim to hold, and keeps grading it every few hours instead of once a year during an audit.

This matters now because platform teams are being asked to prove that golden paths are actually followed, that on-call coverage is real, and that supply-chain controls reach every repository, not just the showcase ones. A catalog without scorecards is a directory. Scorecards without trustworthy ownership metadata are noise sent to nobody. The two only work as a pair.

You will leave with a concrete entity model, a working catalog-info.yaml, an ingestion design that resists ownership drift, scorecard rules written as data, a level scheme that teams accept, wiring to CI, incident and cost systems, and a list of anti-patterns that turn scorecards into vanity dashboards.

What this covers: the catalog entity model, discovery and ownership drift, scorecard rule and level design across Backstage, Cortex, OpsLevel and Port, data wiring, rollout mechanics, and failure modes.

Context and Background

The idea of a service catalog is older than the tooling. Spotify open-sourced Backstage in 2020 and the project later entered the CNCF, where its software catalog became the best-known implementation of a simple principle: metadata about software should live next to the software, in version control, and be aggregated into one queryable graph. Commercial products such as Cortex, OpsLevel and Port took the same idea and added managed ingestion, integrations and, importantly, native scorecards. If you are still choosing between them, the comparison in Backstage vs Port vs Cortex for internal developer platforms covers the product-level trade-offs; this article stays on the data model and the standards layer that all of them share.

The scorecard concept sits on a different lineage: maturity models. OpsLevel’s documentation describes a rubric as a grid in which categories form the rows and levels form the columns, with each check assigned one category and one level. Cortex offers level progression, where rules are grouped into levels with the minimum requirements at the lowest, and a point-based alternative. Port ships default levels of Basic, Bronze, Silver and Gold. Spotify’s commercial Soundcheck plugin for Backstage organises checks into levels inside tracks and issues certifications when a level passes. The shared shape is clear: atomic pass or fail checks, grouped into ordered levels, evaluated per entity.

Why is this a distinct engineering problem rather than a reporting task? Because the inputs are heterogeneous and the audience is hostile by default. Checks pull facts from Git hosting, CI, incident tooling, cloud billing and security scanners. The audience is hundreds of engineers who did not ask to be graded. A scorecard that is wrong once, say by flagging a service as lacking on-call coverage when the schedule exists under a different key, loses credibility that takes quarters to rebuild. So correctness of the underlying metadata, and the transparency of each rule, is the product.

There is also a measurement context. Delivery metrics such as those in DORA metrics and the SPACE framework for platform teams tell you how a team performs. Scorecards tell you whether the team is following the practices that are believed to produce that performance. The distinction is important: confusing practice compliance with outcome is the root of most vanity scorecards, a theme we return to in the trade-offs section.

Finally, the platform context. In a mature internal developer portal the catalog is the substrate for everything else: software templates stamp out entities that are born compliant, TechDocs attach to entities, and cost and incident views are scoped by owner. The golden-path approach discussed in platform engineering golden paths and the Crossplane maturity model assumes this substrate exists. Scorecards are how you verify that the golden path was followed and notice when a service drifted off it afterwards.

For setting up the Backstage side itself, see Backstage 1.30 IDP production setup. The official reference for the descriptor format is the Backstage catalog documentation, and every field named in this article is taken from it unless stated otherwise.

The Catalog and Scorecard Reference Architecture

A software catalog is a graph of typed entities with ownership edges, populated by automated ingestion from source repositories and identity systems, and exposed through an API. Scorecards are an evaluation layer on that graph: each rule is a predicate over an entity and its facts, grouped into ordered maturity levels, and re-evaluated on a schedule.

The entity model: components, systems, domains and resources

Backstage defines nine kinds: Component, Template, API, Group, User, Resource, System, Domain and Location. Most catalogs are dominated by five of them. A Component is a unit of software, typically a service, website or library, with a type (well-known values are service, website and library) and a lifecycle (well-known values are experimental, production and deprecated). A Resource is infrastructure a component needs, such as a database, a queue or a bucket. An API describes an interface a component provides or consumes. A System groups components and resources that together deliver a capability, and a Domain groups systems that serve a business area.

The hierarchy matters for scorecards more than it first appears. Ownership is the obvious reason, but the second is blast-radius reasoning. When a Resource such as a shared Kafka topic fails, dependsOn and consumesApis relations let you compute which components, and therefore which owning teams and which systems, are affected. A scorecard rule like “every production component declares its dependencies” is only meaningful if the relation graph is the thing consumers of the catalog actually query during incidents.

Figure 1: The catalog entity graph. Containment flows from domain to system to component; ownership edges connect components to Groups, and Users belong to Groups.

The figure shows containment and ownership as two orthogonal hierarchies. Containment is about what the software is part of; ownership is about who answers for it. Teams cross system boundaries (the payments team owns both checkout-api and ledger-writer), and systems cross team boundaries. Modelling them separately is what lets you ask both “which components in the Checkout system lack an owner?” and “which components does the payments team own that are still at lifecycle experimental in production traffic?”.

Ownership metadata: the only field every rule depends on

In the Backstage descriptor, spec.owner is a required field on a Component, written as an entity reference to a Group or User. The shortened form payments-team resolves to group:default/payments-team; the catalog produces an ownedBy relation and its reverse ownerOf. The documentation explicitly advises that plugins should consume the relation rather than spec.owner directly, because the relation is the normalised, resolved form.

The naming rules are worth knowing because they cause real ingestion failures. metadata.name must be 1 to 63 characters made of alphanumeric sequences that may be separated by -, _ or ., and is unique per kind within a namespace, case-insensitively. Tags are restricted to lowercase [a-z0-9:+#] sequences separated by dashes. Annotation and label keys follow Kubernetes-style prefix rules, and the backstage.io/ prefix is reserved for Backstage core. Two services that differ only by case collide silently from a human’s point of view, which is a classic source of duplicate-registration confusion.

Ownership metadata is more than one field. A usable service ownership metadata set carries at least: the owning team (spec.owner), a lifecycle, a criticality tier, an on-call escalation reference, a source repository, a documentation pointer and a Slack or equivalent contact channel. Only the first two are first-class Backstage fields. The rest are conventionally expressed through annotations and labels, and the key insight is that you should define your own namespaced annotation keys for organisation-specific facts, for example acme.example/tier, rather than overloading free-text descriptions that no rule can parse.

A working catalog-info.yaml

Here is a realistic descriptor for a production service. The annotations shown (github.com/project-slug, backstage.io/techdocs-ref) are documented well-known keys; the acme.example/* keys are illustrative custom annotations that your scorecard rules would read.

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: checkout-api
  description: Public REST API for cart and checkout
  annotations:
    github.com/project-slug: acme/checkout-api
    backstage.io/techdocs-ref: dir:.
    acme.example/tier: "1"
    acme.example/oncall-schedule: payments-primary
    acme.example/slo-doc: docs/slo.md
  tags:
    - java
    - payments
  links:
    - url: https://runbooks.acme.example/checkout-api
      title: Runbook
      type: runbook
spec:
  type: service
  lifecycle: production
  owner: payments-team
  system: checkout
  providesApis:
    - checkout-rest
  dependsOn:
    - resource:orders-db
    - component:ledger-writer

Several design choices are visible. The system field attaches the component to the Checkout system and generates partOf and hasPart relations. providesApis and dependsOn generate their own relation pairs. links carries the runbook with a type, so a scorecard can test for the presence of a runbook link rather than guessing from a title string. And because annotation values are strings of any length, the tier is quoted: an unquoted 1 would parse as an integer and be rejected where a string is expected.

The Group side: where ownership actually breaks

Teams are modelled as Group entities, usually not hand-written but ingested from an identity source. Backstage documents annotations recording LDAP attributes (backstage.io/ldap-rdn, backstage.io/ldap-uuid, backstage.io/ldap-dn) and Microsoft Graph identifiers (graph.microsoft.com/tenant-id, graph.microsoft.com/group-id, graph.microsoft.com/user-id) that are written at ingestion time. For GitHub-sourced organisations there are github.com/team-slug on Groups and github.com/user-login and github.com/user-id on Users. The immutable user ID is documented as the one used for sign-in matching, which is the right instinct: logins can be renamed, node IDs cannot.

The practical point: spec.owner: payments-team is only as good as the existence and freshness of the Group payments-team. A component that names a Group that no longer exists is a dangling reference. It will register, show an unresolved owner in the UI, and a naive “has owner” scorecard rule that tests for a non-empty field will pass it. This is the first and most common false-green in the whole system, and the rule in the next sections tests resolution, not mere presence.

Figure 2: The ingestion pipeline. Providers emit unprocessed entities, processors validate and emit relations and child entities, the stitcher assembles final entities, and the scorecard engine reads the stitched result through the catalog API.

How ingestion works, and why it determines scorecard honesty

Understanding the Backstage processing loop explains most surprising catalog behaviour. Per the official “life of an entity” documentation, entity providers implement the EntityProvider interface and emit streams of changes that add or remove unprocessed entities. The database tracks which provider owns each entity, and two providers cannot emit the same entity. Ingestion applies only coarse checks such as the presence of kind and metadata.name; stricter validation happens in processing.

Processors then run in order. A processor can mutate the entity and emit further entities, errors or relations. If errors are emitted, the previous error-free final version is kept, which is a deliberate stability choice: a broken edit to catalog-info.yaml does not blank the entity, it freezes it at the last good state and records the error in the entity status. For scorecards that has a subtle consequence. A service whose descriptor became invalid last week is being scored against last month’s metadata, and unless your portal surfaces processing errors prominently, nobody knows. Treat a processing error as a scorecard failure of its own.

The stitcher then merges the processed entity, its errors and all incoming and outgoing relations into the final entity served by the catalog API. Stitching runs only when the entity’s hash changes, and the hash covers the body, relations, errors, referenced entities and parents. The docs note that even array order changes the hash, so a tool that regenerates YAML with shuffled tag order causes needless re-stitching. Stitching itself is fixed and cannot be extended; customisation belongs in ingestion or processing.

Discovery is where adoption is won or lost. The GitHub catalog provider is configured under catalog.providers.github, with an organization, a catalogPath (default /catalog-info.yaml), optional filters on branch, repository regex, topics, visibility and allowArchived (default false), and a schedule with frequency, timeout and an optional scope. The docs advise lengthening the frequency to avoid GitHub API rate limits. A representative configuration:

catalog:
  providers:
    github:
      acmeOrg:
        organization: 'acme'
        catalogPath: '/catalog-info.yaml'
        filters:
          branch: 'main'
          repository: '.*'
          topic:
            exclude: ['experiments']
          allowArchived: false
        schedule:
          frequency: { minutes: 30 }
          timeout: { minutes: 3 }

Auto-discovery of catalog-info.yaml files at a fixed path means a team registers a service by merging one file; there is no central form to fill in. That is the single biggest factor in catalog coverage, and coverage is the denominator of every scorecard statistic. A scorecard that reports 94 percent of catalogued services are Silver means little if the catalog holds only 60 percent of the real estate. Track catalog coverage against an independent inventory (the list of repositories, container images or deployed workloads) as the zeroth scorecard.

Designing Engineering Scorecards: Rules, Levels and Scoring Models

Engineering scorecards are sets of automated pass or fail rules grouped into ordered levels. A service sits at the highest level for which it passes every rule at that level and all levels below. The design questions are what a rule may read, how levels are ordered, and whether to score by level progression or by accumulated points.

Rules are predicates over facts, and the fact source decides everything

Every rule reduces to: take a fact about the entity, compare it with a threshold or pattern, return pass or fail with a message. The vendors differ mainly in where facts come from and how rules are expressed.

In Port, scorecards are defined against a blueprint (Port’s term for an entity type). Each rule has an identifier, title, level and a query made of a combinator (and or or) and conditions. A condition has an operator, a property or a relation (not both) and a value. Supported operators include =, !=, <, >, contains, in, beginsWith, isEmpty, isNotEmpty and between. Port’s documentation is explicit about two limitations that shape design: rules compare against static values only, so comparing two properties requires a calculation property first; and rules see only entity properties and relations, not file contents, so file data must be extracted into a property by an integration or automation. A minimal ownership rule from Port’s documented structure looks like this:

{
  "identifier": "has-resp-team",
  "title": "Has responsible team",
  "level": "Bronze",
  "query": {
    "combinator": "and",
    "conditions": [
      { "operator": "isNotEmpty", "property": "$team" }
    ]
  }
}

In Cortex, rules are built with a form builder pre-configured per integration or with the CQL editor for complex rules and multiple data sources. Each rule carries a failure message, optional group restrictions to include or exclude entities, and optionally a point value and a scheduled start date in UTC. The scheduled start date is a genuinely good design: scheduled rules are visible to all users and appear in weekly reports before they take effect, which is exactly the advance notice that prevents rollout resentment. Cortex evaluates every four hours by default, can be triggered manually, and re-evaluates affected entities when a scorecard is saved.

In OpsLevel, checks are the atomic unit, assigned to a category and a level on the rubric. The documentation describes a beginner level that is a catch-all for services that have not passed every check in the lowest level; it cannot contain checks, so it never appears on the rubric. In Spotify’s Soundcheck, a check has an id, name, description, optional filter, and a fact rule with a schedule, and fact collectors gather data from the catalog, from source control providers and through a Facts API.

The common structural lesson: a scorecard engine is a fact pipeline with a rule language on top. The rule language is the easy part. Whether a fact is fresh, correctly mapped to the entity, and available at all is where projects stall. Before writing a rule, answer three questions: which system holds the fact, how is it joined to the catalog entity (by repository slug, by service name tag, by an explicit annotation), and what does “unknown” score as. A missing fact should usually be a failure for mandatory rules and a skip for informational ones, but the engine must distinguish them, or integration outages will mark whole fleets as non-compliant overnight.

Level progression versus points

Two scoring models dominate. Level progression, offered by Cortex, OpsLevel, Port and Soundcheck, grants a level only when every rule at that level passes, and every lower level also passes. Port states this precisely: passing one of two Bronze rules keeps an entity at Basic, a level with no rules is skipped, and all rules at the same level count equally with no weighting. Point-based scoring, an option in Cortex, assigns each rule a value and sums.

Figure 3: The scorecard engine. Catalog, CI, incident, cost and security facts feed rules arranged in three levels; results drive dashboards, nudges and pull request checks.

Level progression wins for standards because it communicates a clear next step: “you are Bronze; these three rules stand between you and Silver.” Points are better for continuous quantities and for portfolio-level triage, where you want to sort hundreds of services by how far they are from a target. A hybrid works well in practice: levels for the binary, non-negotiable hygiene controls, and a separate informational score for gradient measures like test coverage. Avoid putting a gradient measure in a level rule with an arbitrary threshold unless you can defend the number. A rule such as “coverage at least 80 percent” produces gaming (coverage-padding tests) and arguments about generated code, and the threshold’s link to quality is weak.

A level scheme that teams accept

The biggest design decision is which rules go in which level. A defensible layout, shown here as a starting proposal rather than an industry standard, uses three levels with distinct purposes:

Level	Purpose	Example rules	Typical data source
Bronze	The service is findable and attributable	Owner resolves to an existing Group; lifecycle set; source repo linked; runbook link present	Catalog
Silver	The service is operable	Passing default-branch build; on-call schedule linked; alert routing configured; docs present; dependencies declared	CI, incident tool, catalog
Gold	The service is resilient and defensible	SLO defined with an alert; no critical vulnerabilities older than policy; dependency scanning enabled; tested restore for stateful resources	SLO tooling, scanners, backup system

Bronze should be achievable in an afternoon by any team, because its purpose is to make the catalog reliable, and everything else depends on that. Silver should require real but bounded effort and map to what an incident responder needs at 3 a.m. Gold is aspirational and should be scoped by tier: requiring Gold of a tier-3 internal batch job wastes effort that the tier-1 payment path needs. All of the major tools support filters or group restrictions so a rule can apply only to selected entities, for example only lifecycle: production components with acme.example/tier of 1 or 2.

Make each rule trace to a failure you have actually had. “On-call schedule linked” exists because a page once went to a former employee. “Runbook link present” exists because responders lost twenty minutes finding the restart procedure. Rules that cannot cite an incident or a control requirement are candidates for deletion, and the incident process in incident management architecture with severity levels, on-call and blameless postmortems is the best source of such justifications. Postmortem action items are the natural origin of new scorecard rules, and the best scorecards grow this way.

Scorecard rules as code

Rules that live only in a web UI cannot be reviewed, diffed or rolled back. Cortex supports “Scorecards as code” through GitOps and an API, Port exposes scorecards through its API and an infrastructure-as-code provider, and Soundcheck tracks can be defined in YAML. Whatever the tool, keep definitions in a repository with code review, so that raising a bar is a pull request with a description and a date, and so that the history explains why Silver changed.

A rule file in a neutral, illustrative format (not any vendor’s literal syntax) makes the structure clear:

scorecard: production-readiness
applies_to:
  kind: Component
  where:
    lifecycle: production
levels:
  - name: bronze
    rules:
      - id: owner-resolves
        fact: relations.ownedBy
        assert: exists_and_target_active
        message: "Owner group missing or deactivated"
      - id: runbook-linked
        fact: metadata.links
        assert: any(type == runbook)
  - name: silver
    effective_from: 2026-12-01
    rules:
      - id: oncall-linked
        fact: annotations["acme.example/oncall-schedule"]
        assert: schedule_exists_in_incident_tool
      - id: main-build-green
        fact: ci.default_branch_status
        assert: equals(success)
        max_age_hours: 48

Three details deserve emphasis. The effective_from date implements the advance-notice pattern. The max_age_hours field makes freshness a first-class property, so a stale build fact fails loudly instead of passing quietly. And target_active in the ownership rule tests that the Group resolves to something live, closing the dangling-owner false-green described earlier.

Comparing the tools on the dimensions that matter

The vendor choice is secondary to the design above, but the differences are real. Based on the vendors’ own documentation: Backstage’s catalog is open source and gives you the entity model, ingestion and relations, but not a native scorecard engine, so scorecards come from a plugin (Soundcheck is a commercial option) or from your own code reading the catalog API. Cortex, OpsLevel and Port are managed products where scorecards are native, with level progression in all three and a points option in Cortex. Port’s data model is configurable blueprints rather than a fixed set of kinds, which suits non-service entities such as environments and cloud accounts. Cortex’s rule authoring spans a form builder and CQL. OpsLevel’s rubric grid of categories by levels makes the maturity structure visible at a glance.

I have not benchmarked any of these for evaluation latency or scale beyond what their documentation states, and you should treat pricing and entity limits as things to verify with each vendor at purchase time. One documented limit worth knowing: Port reports support for up to 5 million rule result entities, which is a useful reminder that results are themselves data. Fifty rules across ten thousand entities is already half a million result rows, and that figure grows if you keep history.

Wiring Scorecards to CI, Incidents, Cost and the Delivery Workflow

Scorecards earn their keep when rules read live operational data and when results flow back into the places engineers already work: pull requests, chat, and team dashboards. This section covers the integration patterns and the rollout mechanics that determine adoption.

Pulling facts from CI and source control

The cheapest high-value facts come from Git hosting and CI. Is the default branch protected? Does the repository have a CODEOWNERS file? Does the last default-branch build pass? Is there a lock file and a dependency update configuration? These are all retrievable through the hosting provider’s API and map to catalog entities through the github.com/project-slug annotation, a documented well-known key. Other documented annotations map to additional systems: sonarqube.org/project-key, sentry.io/project-slug, circleci.com/project-slug, jenkins.io/job-full-name and backstage.io/techdocs-ref. Using the well-known keys wherever they exist keeps your catalog compatible with plugins you may adopt later.

A frequent subtle failure is identity joining. If the CI system knows a service as checkout-api-prod and the catalog knows it as checkout-api, the join fails silently and the rule reads “no data”. Prefer explicit annotation-based joins over name matching, and add a scorecard rule that the join keys exist, so mapping gaps become visible work items rather than invisible false-negatives.

Incident and on-call facts

On-call coverage is the fact with the highest value per rule, and also the one most prone to staleness. The rule should check that the referenced schedule exists in the incident tool, that it has at least a minimum number of distinct humans in rotation, and that the escalation policy has more than one tier. Checking only that an annotation is non-empty reproduces the dangling-owner problem one level down.

The reverse flow is equally valuable. When an incident fires, the incident tool should resolve the owning team from the catalog rather than from a separately maintained mapping, which gives you a single source of ownership truth and means every incident exercises the ownership data. Ownership errors then surface at the cheapest possible moment, when someone is paged wrongly, and can feed a rule such as “no misrouted page in the last 90 days”. SLO status is another natural fact: a Gold rule that a tier-1 service has a defined SLO with a burn-rate alert aligns the scorecard with the practices in SLO and error budgets with OpenSLO and Sloth, though that article uses an IoT platform as its running example.

Cost and security facts

Cost attribution is an ownership problem in disguise. If cloud resources carry a tag naming the owning team, and that tag resolves to a catalog Group, then spend can be shown per component and per system. A scorecard rule “all provisioned Resources carry an owner tag that resolves in the catalog” closes the loop, and the share of untagged spend is a more persuasive number than any exhortation. The same applies to security: vulnerability scanners report per repository or image, and the catalog turns that into per-owner queues. Rules should reflect policy windows rather than raw counts, for example “no critical finding older than the number of days your policy allows”, because raw counts punish large services for being large.

Figure 4: The feedback loop. A team commits a descriptor, discovery ingests it, the engine evaluates and reports failing rules, and the platform team raises bars only with advance notice.

Rollout: the sequence that avoids a revolt

Adoption failures are social, not technical, and the mechanics above exist to support a specific rollout sequence.

First, make registration free. Auto-discovery of catalog-info.yaml plus a scaffolder template that generates a pre-filled descriptor means a new service is born in the catalog with an owner. Backstage templates can carry the backstage.io/source-template annotation, documented as the Scaffolder template reference used to create the entity, so you can later measure how many services came through the golden path and compare their scorecard levels with hand-built ones. That comparison is the most credible argument for the golden path you can produce.

Second, backfill ownership before introducing levels. Run a read-only pass that lists components whose owner is missing or unresolved, and let team leads claim them. Expect a long tail of orphaned software; deciding to deprecate or assign it is the first real value the catalog delivers.

Third, launch Bronze only, in report-only mode, with a published date. Cortex’s scheduled-rule feature and its draft state, which hides drafts from reports and notifications, support exactly this. Fourth, add Silver and Gold rules one at a time with effective dates at least a few weeks out, and use initiatives or equivalent time-boxed campaigns (Cortex describes initiatives that require all rules in a level to be completed by a set date) for the rules that need coordinated work. Fifth, put the results where engineers are: a pull request comment when a change would lower a service’s level, a weekly team digest, and the service page in the portal.

Exemptions are a feature, not a loophole

Some services legitimately cannot meet a rule: a vendor appliance has no CI pipeline, a legacy batch job will be retired next quarter. Cortex documents rule exemptions with optional approval workflows and notes that auto-approval defaults to disabled. Whatever tool you use, require exemptions to carry a reason, an approver and an expiry date, and report on active exemptions as a metric. A scorecard without an exemption path trains people to game rules; one with an unbounded path stops meaning anything.

Trade-offs, Gotchas, and What Goes Wrong

Scorecards fail in predictable ways. Most of them are visible in the first quarter if you know where to look.

Ownership drift. Teams reorganise; Groups are renamed or deleted; people leave. Metadata in catalog-info.yaml files does not change when the org chart does, so ownership decays at the rate of reorganisation. Backstage’s orphan handling is relevant but easy to misread: per the official documentation, an entity becomes orphaned when its parent stops emitting it and nothing else points to it, the stitcher adds the backstage.io/orphan: 'true' annotation, and by default (orphanStrategy: delete) orphans are deleted. Moving a catalog file or editing a parent location can therefore delete entities you meant to keep, and setting orphanStrategy: keep disables cleanup but lets dead entities accumulate. Notably, deleting or corrupting a file does not orphan entities; they get a hard error instead. Pick the strategy deliberately and alert on orphan counts. The durable fix for drift is to derive ownership from a system that people must keep correct for other reasons (the identity provider, the on-call tool) and to run a rule that the owner Group resolves and has at least one active member.

Vanity scorecards. A scorecard becomes vanity when it measures what is easy rather than what is protective. The signature symptoms are a fleet that is 95 percent Gold within a month and no corresponding improvement in incident outcomes. Illustrative arithmetic: suppose 400 services and a Silver rule requiring a linked runbook. If linking any URL passes, compliance reaches 100 percent in a week, while the share of runbooks that a responder could follow remains unknown. The remedy is a mix of rule quality (test that the link resolves and the document was updated within a period) and outcome cross-checks (does the Gold cohort have fewer paging incidents and faster recovery than the Bronze cohort?). If it does not, your rules are not measuring what you think.

Goodhart effects. Any numeric threshold on a gameable measure will be gamed. Coverage percentages, ticket counts and “documentation exists” checks are easy to satisfy hollowly. Prefer binary controls whose cheating is more expensive than compliance, and use gradient measures for conversation, not gating. Equally, resist ranking teams against each other. A leaderboard motivates the already motivated and humiliates the under-resourced; show each team its own gap to the next level instead.

False negatives from integration gaps. When a fact source is down or a join key is missing, rules fail for reasons the team cannot fix. Each false negative costs trust. Distinguish “failed” from “unknown” in the UI and the reports, and have the platform team own integration health as a service-level objective of its own.

Evaluation lag and surprise. Cortex documents a default evaluation window of four hours, and Backstage’s GitHub provider examples poll on the order of tens of minutes, so a fix does not show immediately. Engineers who push a fix and see a red rule an hour later assume the scorecard is broken. Display the timestamp of the last evaluation next to each result, and offer a manual re-run where the tool allows it.

Rule inflation. Platform teams keep adding rules because each is individually reasonable. Fifty rules per service is a compliance regime, not a maturity ladder, and engineers stop reading it. Cap each level at a number a person can remember, retire rules whose pass rate has been 100 percent for two quarters, and require every new rule to name the incident or control it addresses.

Tooling coupling. Rules written in a vendor language are not portable. Keep the intent, the justification and the owning person in a vendor-neutral document or repository, and treat the vendor syntax as a compile target. That way a migration is a translation exercise rather than archaeology.

Metadata sprawl in a single file. Every new annotation is a new burden on every team. Before adding one, ask whether the fact can be derived (from the repository, the deployment manifest or the infrastructure state) instead of declared. Declared metadata rots; derived metadata only breaks when the underlying system does. A good rule of thumb is that a descriptor should contain what a human must decide (owner, tier, lifecycle) and nothing a machine can look up.

Practical Recommendations

Start with the problem you can fix this quarter, which is almost always ownership. Make the catalog complete and attributable before measuring anything else. Then add scorecards in thin slices, each justified by a past incident, each announced before it bites.

Treat the descriptor as a small, stable contract. Keep spec.owner, spec.lifecycle, spec.type and spec.system mandatory; add one or two namespaced annotations for tier and on-call; derive the rest. Prefer the documented well-known annotations where they exist so your catalog stays compatible with community plugins.

Test resolution, not presence. For ownership, on-call, runbooks and SLO references, the rule should confirm that the thing referenced exists and is current. Presence checks are the source of nearly every false-green.

Use levels for hygiene and a separate informational view for gradients. Scope Gold by tier. Keep rule definitions in version control with dated changes. Measure the scorecard itself: catalog coverage against an independent inventory, share of unknown versus failed results, active exemptions, and the correlation between level and operational outcomes.

A short checklist you can adopt:

[ ] Auto-discovery of catalog-info.yaml is running across all repositories, with an independent inventory to measure coverage.
[ ] Every production component has a resolving owner Group with active members.
[ ] Scaffolder templates generate compliant descriptors, with the source-template annotation recorded.
[ ] Bronze, Silver and Gold each have a one-sentence purpose and a capped rule count.
[ ] Every rule cites the incident or control that justifies it.
[ ] Rule changes ship as reviewed pull requests with an effective date and advance notice.
[ ] Unknown and failed results are reported separately, with evaluation timestamps shown.
[ ] Exemptions require a reason, an approver and an expiry.
[ ] Processing errors and orphan counts are alerting, not just logged.
[ ] Level is periodically compared against incident and delivery outcomes to catch vanity drift.

Frequently Asked Questions

What is a software catalog scorecard?

A software catalog scorecard is a set of automated rules evaluated against each entity in a service catalog, such as a component, grouped into ordered maturity levels like Bronze, Silver and Gold. Rules read catalog metadata and external facts from CI, incident, cost or security tools. The result shows each team what level its service has reached and exactly which failing rules block the next level.

What does a Backstage catalog-info.yaml contain?

It contains an apiVersion (backstage.io/v1alpha1 for most kinds), a kind such as Component, metadata including a required name, optional annotations, labels, tags and links, and a spec. For a Component the spec requires type, lifecycle and owner, and optionally system, subcomponentOf, providesApis, consumesApis and dependsOn. The owner is an entity reference to a Group or User.

Does Backstage include scorecards out of the box?

Not as a native scorecard engine in the open-source catalog. Backstage gives you the entity model, ingestion, relations and API. Scorecards typically come from a plugin, such as Spotify’s commercial Soundcheck, or from a custom service reading the catalog API. Cortex, OpsLevel and Port, by contrast, ship scorecards as native features. Verify current plugin options and licensing with the vendors before deciding.

How do you keep service ownership metadata from going stale?

Derive it from systems people must keep correct anyway, such as the identity provider and the on-call tool, and ingest teams as Group entities rather than hand-writing them. Add a rule that the owner resolves to an existing Group with active members, alert on orphaned entities and processing errors, and route incidents through catalog ownership so mistakes surface when someone is paged wrongly.

How many levels and rules should a scorecard have?

Three levels are usually enough, with a short rule list in each that a person can remember. Bronze should be achievable in an afternoon and establish findability and ownership. Silver covers operability, Gold covers resilience and security, scoped by service tier. If a level has dozens of rules, engineers stop reading it. Retire rules that have passed everywhere for several quarters.

How do I stop scorecards becoming a vanity metric?

Tie every rule to a past incident or control requirement, test that referenced artefacts resolve and are current rather than merely present, avoid arbitrary numeric thresholds on gameable measures, and compare level cohorts against real outcomes such as incident frequency and recovery time. Do not rank teams publicly. If higher levels do not correlate with better outcomes, revise the rules.

Software Catalog and Service Scorecards: Ownership Metadata, Backstage catalog-info and Engineering Standards at Scale (2026)