Appverse Review Rubric

Related Docs: Appverse Reviewer Process | Appverse Contributor Guide | Appverse Best Practices | Appverse README Template

The criteria for Appverse app review. The automated review (Appverse Review) and the human reviewer both work from this document; the Reviewer Process says what to do with it and in what order. This page replaces the earlier Security Rubric, which is now the Security section below.

How to read this rubric

A review produces three kinds of content. Keeping them apart is the point of this document.

1. Gate criteria (pass / fail). Hard requirements an app must meet to be listed. They live in the Structure section. An app that fails one cannot be accepted until it is fixed.
2. Signals (levels). A per-dimension reading for a deployer on three axes, each with its own three level words, from the good end to the end with the most to read before deploying: Portability is Portable, Needs site config or Site-specific; Documentation is Complete, Adequate or Minimal; Upkeep is Well maintained, Active or Inactive. Each axis's section below maps its grades to these words. Signals describe; they never decide. A Site-specific, Minimal or Inactive level on its own does not reject an app, and a Portable, Complete or Well maintained one does not accept it.
3. The decision. Accept, accept with suggestions, request changes, or reject, derived from the gate criteria and from properties of individual findings. The rules are in the Decision rubric section at the end.

Signals do not gate; the decision rubric does. Two consequences worth stating: a fixable High-severity security finding is "request changes", not "reject"; and a Documentation or Portability signal below the target for inclusion lands in "accept with suggestions", not "request changes".

Code Quality is a fourth kind of content: findings that are neither a gate nor a signal. They are reported as findings, and they never decide. Two of them (error handling and input validation) are strong suggestions; the rest are suggestions.

The README has two roles. "README present and substantive" is a Structure gate (a stub fails). README depth (Minimal / Adequate / Strong / Exemplary) is the Documentation signal. Same file, two different legs.

Each finding the automated review records carries a rule code (STR-, OODT-, QUA-, MNT-), a severity, and file:line evidence. The severity scale is defined in the plugin's finding-codes.md. Each finding also carries a result: FAIL means the criterion is not met; WARN means the tool could not confirm it or the defect is advisory; PASS means the check ran and found nothing; NOT CHECKED means it could not run. A WARN or NOT CHECKED on a gate row is not a pass: the reviewer confirms that row by hand, from the cited evidence, before deciding (Reviewer Process, Step 3).

The reviewer can change any of the tool's FAIL or WARN findings on the review page: a new severity, higher or lower; "Not a finding", which dismisses it; or their own wording of its summary and evidence. A new severity or a dismissal needs a reason, given as the finding's note to the contributor. The tool's values stay on the record, and the reviewer's change applies to that round only. Everything that reads a finding uses the reviewer's values: the decision floors, the counts, and the catalog's Security count. A dismissed finding counts as no finding. The rule code cannot be changed; a finding with the wrong code is dismissed and replaced by a reviewer's own. The reviewer can also change a signal level, with a reason.

Repo shapes

A repo with a root appverse.yml is declared: the contributor states how their app(s) appear in the catalog. A repo with only a root manifest.yml is inferred: the catalog derives a single app from that file plus GitHub metadata.

A declared repo whose appverse.yml has an apps: list is a Monorepo. Review every entry in apps[] as its own app — each gets its own structure, security, and quality assessment, and its own decision. Per-app fields resolve with this precedence:

1. Inline in the root appverse.yml apps[] entry (highest)
2. <subpath>/appverse.yml
3. <subpath>/manifest.yml (name/description fallback only)

Directories listed in shared_paths are shared code: review them once at repo level and include them in the security review.

Installing any one app clones the whole repo, so a monorepo is gated as a whole before its apps are decided. Two kinds of problem decide every app the same way:

• A repo-level problem: a failed repo-level gate criterion (Request changes when it is fixable) or an unmaintained repo (Reject), as the Decision rubric below says for any repo.
• A High or Critical security finding, wherever it is: in shared_paths, at the repo root, or inside one app's directory. High sets at least Request changes and Critical sets Reject for every app (see the severity scale in finding-codes.md).

Past that gate, decisions can differ per app. For example, three apps can be accepted while a fourth gets Request changes for missing metadata. The Overall recommendation rolls the apps up: it is the strictest Per-app decision.

shared_paths is scope input for the security review; it is not a pass/fail criterion.

appverse.yml is optional when a root manifest.yml is present: the catalog infers a single app from the manifest. A repo with neither file has no identifiable shape and fails the Structure gate below.

Structure (gate criteria)

All of these are gates: every one must pass for the app to be listed. The tool records each as PASS or FAIL; where it could not confirm a row it records WARN or NOT CHECKED, and the reviewer confirms that row by hand, checking the cited evidence, before deciding (see "How to read this rubric"). A FAIL on any gate row sets at least Request changes, whatever its severity (see Decision rubric), and is always a fix-item in the feedback to the contributor. A Structure note the tool files as other: (something else worth a look) is not a gate row.

Repository structure

Every repo, regardless of shape:

README and LICENSE are checked once at repo level. In a monorepo the apps share them; an app-level README at a subpath is welcome but is not required by this gate.

Inferred repos (root manifest.yml, no appverse.yml):

Declared repos (root appverse.yml), checked per app after field-precedence resolution. The same fields are required whether the repo declares one app or several:

Batch Connect layout

Standard OOD structure: form.yml or form.yml.erb, submit.yml.erb, and a template/ directory containing the job script (script.sh.erb, or script.sh when nothing is templated). OOD renders form.yml.erb at request time, so an app shipping only the .erb variant is complete; shipping both is unusual and worth a look.

Passenger and companion apps substitute their own layout check: an entry point that exists and parses (config.ru for Ruby/Rack, passenger_wsgi.py for Python/WSGI), a manifest.yml role consistent with that layout, and a dependency manifest (Gemfile.lock, package-lock.json, requirements.txt) consistent with the dependency file (STR-08 if missing or inconsistent).

Validity

Security

Security is a list of findings, never a rating. The tool does not assert a security level; an app with no FAIL or WARN findings is reported as having no tool-detectable issues in the checked tiers, which is not a claim that it is safe. The review page and the catalog show Security as a count of findings to review, and as "No findings to review" when there are none; that is never a claim of safety either. Some finding properties feed the decision rubric directly. Every finding is classified under the OODT taxonomy and rated on the five-level severity scale.

Decision role: a finding tagged potentially malicious, or one that cannot be fixed without redesigning the app, is a reject trigger. A fixable High finding is "request changes". Low and Medium findings do not by themselves block "accept with suggestions" (see Decision rubric).

How the automated review checks security

The security review has three tiers, named in every report so a thinner review looks thinner rather than identical to a full one:

• Tier 1 — Static. Source-level reading: the candidate loop and the open-ended pass below, then the capability profile. Runs anywhere, including CI on a submitted pull request.
• Tier 2 — Tooling. Static analysis tools (shellcheck, bandit, semgrep, trivy, and others per the plugin's security-tools.md). Needs installed binaries, not a running app. Best-effort: the review proceeds without them. Run before the review by the plugin's pre-review script; the report's tool table lists what ran.
• Tier 3 — Runtime. Boot the app and exercise it, only in an isolated, disposable environment (never on a reviewer's own machine or a shared host). Reported as NOT CHECKED — no isolated execution environment when none is available, which is the usual case.

Tier 1 is a loop, not an open-ended read. The plugin's pre-review script lists every candidate site in the app: each ERB interpolation of a form value in submit.yml.erb and template/ (with whether it is quoted and guarded), each unquoted expansion of a form value, each eval/exec, credential-shaped string, chmod/chown/umask, network call, write outside the job directory, security-relevant config flag (--no-auth, --disable-ssl, 0.0.0.0, a CORS wildcard), and binary file under template/. Each candidate names its file, line and the OODT code it would fall under. The review answers every candidate with its own row in the report: FAIL or WARN with category and severity, or PASS with one line of why it is acceptable. The candidate list is the same on every run, so every run is asked about the same sites. After the loop, an open-ended pass looks for what the enumeration cannot see (a missing CSRF defense, partial authentication, container isolation, an unusual way of phoning home) and records each under "Additional observations (review)", saying the enumeration missed it. "No tool-detectable issues in the checked tiers" is the claim when no row in either table is FAIL or WARN; never write it when any row is FAIL or WARN, and never write "safe".

Two complementary methods feed the same classification:

• Capability profiling catalogs what an app does — what system access it needs, what network calls it makes, what files it reads and writes, what processes it spawns. For batch connect apps, with their narrow expected profile, unusual capabilities are a strong signal of malicious intent. For Passenger apps, with legitimately broad capabilities, the profile is a transparency tool — it tells reviewers and deployers exactly what they are putting on their system so they can make an informed decision. An app is never penalized for doing what it is designed to do.
• Pattern checks catch capabilities used unsafely, regardless of app type. Form values reaching shell commands without sanitization, eval() on config-supplied code, a service with --no-auth, credentials stored in plain text. Normal tools, unsafe usage.

Capability baseline: Batch Connect apps

The expected profile is narrow. Anomalies are a strong signal.

Capability baseline: Passenger apps

The expected profile is wide. The full profile is reported for transparency. Capabilities in the "Flagged" column generate findings.

Dashboard apps and widgets — no detailed baseline yet; there are not enough examples in the catalog. Apply the pattern checks and report whatever capabilities are found. Baselines will be built out as more of these enter the catalog.

Pattern checks (all app types)

These catch capabilities used unsafely, across all app types.

OODT — Open OnDemand App Threats

OODT is local to Appverse review and describes threats specific to the Open OnDemand execution model. It is not related to OWASP's separate OAT catalog.

Every finding is classified under one of these eight threat types. The risk level is about blast radius, not raw severity: the Per-User Nginx (PUN) architecture already contains a lot of single-user damage, so threats that escape the user's own sandbox (cross-user, cross-site, cross-session) rank above self-harm.

Rating findings

For on-demand review, rate each finding's severity using the five-level scale defined in finding-codes.md:

• Critical — unfixable without redesigning the feature, or tagged potentially malicious (e.g., curl|bash on user-supplied URLs). Warrants a Reject recommendation.
• High — a real vulnerability or gate-criterion failure, fixable with targeted changes (e.g., CORS open to all origins). Warrants Request changes.
• Medium — a genuine concern but lower blast radius or harder to exploit.
• Low — defensive-coding gap or minor hygiene issue.
• Info — maintenance signals, suggestions, positive observations.

Base the rating on blast radius (cross-user or cross-site outranks self-harm) and how easily it is triggered. Tag each finding as unintentional or potentially malicious.

A finding is a capability used unsafely, not a capability that is part of the app's design. A job composer that runs shell commands is doing its job; running them with CORS open to all origins is a finding. Report designed capabilities in the capability profile; score only the unsafe usage.

Severity does not mechanically determine the review decision. A fixable misconfiguration — even High severity, such as CORS open to all origins — points to "request changes," not "reject." Reserve Critical and Reject for findings tagged potentially malicious, or exposures that cannot be fixed without redesigning the app.

The numeric severity/exploitability scoring and the commit-polling audit pipeline are defined separately, not in this rubric. Both the on-demand skill and that pipeline classify findings with the same OODT taxonomy and pattern checks above.

The catalog shows each listed app's review as four chips: Upkeep, Documentation and Portability in the level words this rubric defines, and Security as the count of findings to review (FAIL or WARN, after the reviewer's changes), or "No findings to review".

Portability

How much a deployer at another site has to change before the app runs.

Portability signal: Portable is shown as Portable; Partially portable as Needs site config; Not portable as Site-specific.

Target for inclusion: Partially portable or above. Below the target is "accept with suggestions" when the gate criteria pass; suggest the contributor move site-specific values into form attributes so a site can set them in its own config.

Where to look: submit.yml.erb, form.yml, form.yml.erb, and template/ scripts, for cluster names, partitions, accounts, absolute site paths, and module versions.

Named checks (recorded as QUA-02 findings with file:line evidence):

• check: hardcoded-site-paths — scratch, project, or home paths, cluster or partition names, or a module version that appears in a script or template rather than in a form.yml attribute or a documented configuration value. Documented site-specific values in the README's configuration table count toward Partially portable; undocumented ones count against it.

Documentation

How well the README answers a deployer who has never seen the app. The README should follow the Appverse README Template.

Documentation signal: Strong or Exemplary is shown as Complete; Adequate as Adequate; Minimal, Below minimal, or a stub README as Minimal.

Evidence: the rating rests on one evidence line per requirement in the table above (what it launches through architecture), each citing the README heading and line the pre-review's readme.json maps to that requirement, or none. A heading whose section is only the README template's placeholder text counts as none. A screenshots or environment-variables heading with no image or variable under it is "heading only", weaker evidence the reviewer checks. The reviewer may judge that a listed section does not satisfy its requirement. A requirement readme.json shows no heading for is met by a README line that delivers it, cited as content: README.md:N (the path is readme.json's file, the app's own README in a monorepo); the line must be text, not a heading, code, placeholder, contact or badge line, one line meets one requirement, and without one the requirement is none. This content search applies even when the requirement's own heading exists but doesn't deliver it — check other headings' content (configuration met by a line under Environment Variables, say) before settling on none. With no Overview-type heading, a descriptive paragraph under the README's title satisfies "what it launches" (readme.json records it as the intro). The level is the highest one whose requirements, and every lower level's, all have evidence. A README that is not a stub but has no evidence even for Minimal is rated Below minimal, with a QUA-01 docs-minimal FAIL, as for Minimal (both miss the Adequate target). Whether a README is a stub is decided once, by the pre-review facts (readme.json stub: fewer than 100 characters of content, or every section body template placeholder text with no real text above the first heading), never by the rating: a stub is the Structure gate failure (STR-01 readme-not-substantive), not a rating. The Documentation side records it as QUA-01 docs-stub, and the report says "Minimal — not supported (stub README; see QUA-01)".

Target for inclusion: Adequate or above. Below the target is "accept with suggestions" when the gate criteria pass. A README that fails the Structure gate (stub or unfilled template) is a gate failure, not a Minimal rating. The four questions above are how the signal is rated; they are not a gate. This was previously listed under "Required Criteria" as Documentation Minimum, and the earlier reviewer checklist and report disagreed about it; the split here is the resolution.

Red flags (recorded as QUA-01 findings):

• README references a different institution's paths without noting they need to change
• No Configuration section — deployers can't tell what to customize

Code Quality

Findings about how the app is written. They are not a signal and not a gate, and they never set a decision. They are reported with file:line evidence. The first two are strong suggestions, worded in the feedback as recommended rather than optional, since a script that runs on after an error, or a form that accepts any value, fails deployers silently. The rest are suggestions.

Correctness and polish defects are also Code Quality findings: copy-paste artifacts from the template an app was cloned from (a MATLAB reference in a SAS app, a CHANGELOG describing a different app), duplicate YAML keys (valid YAML, but last-wins, so the parser does not catch them and the author almost certainly did not intend it), wrong help text, and README typos.

Draft feedback guidance. Never advise removing a comment or help text that states a real constraint (a partition that requires a GPU, a limit); advise rewording it.

Named checks:

• check: numeric-field-bounds — every numeric form field (number_field, or a text field that is parsed as a number in submit.yml.erb) has min and max, and free-text fields that reach the scheduler have a pattern or are otherwise constrained. A missing bound is an input-validation finding.
• check: icon-matches-target-os — desktop and panel configuration under template/ (for example xfce4-panel.xml button-icon, .desktop Icon=) references an icon that exists on the OS the README says the app was tested on. A Fedora icon in an app documented for Rocky Linux is a polish finding.

Upkeep

Signals about the project's health. Upkeep is a signal on the repo and, in a declared monorepo, on each app as well: apps in one repo can be maintained by different people at different paces. An app's upkeep reads its own folder at the reviewed commit (its last change by a person, its human contributors, a CHANGELOG of its own) by the same rule below, with releases, CI and issues taken from the repo. Bot commits do not count. It says whether people have changed the folder lately, not why, so a repo-wide change by a person (a license header, a lint pass) counts for every app it touches. An app's upkeep is a signal for deployers only; the decision role below stays with the repo's.

Upkeep signal: active within 12 months with two or more good-practice signals is Well maintained; active within 12 months is Active; inactive over 12 months (an MNT-01 finding) is Inactive. The good-practice signals are releases, a current CHANGELOG, CI, multiple contributors, and responded-to issues, and each counts once toward the two. A repo with no open issues is neutral on that last signal: it counts neither for nor against, because there is no evidence of responsiveness either way. A brand-new app under the waiver below is Active, unless it also has an MNT-01 finding, which makes it Inactive.

Target for inclusion: active within 12 months. For a brand-new app, waive the history requirement and say so in the evidence phrase. A follow-up review of brand-new apps is not scheduled automatically yet.

Decision role: an abandoned or unmaintained repo is a reject trigger. Inactive is not the same as abandoned: an MNT-01 failure at High sets at least Request changes, not Reject. Whether a repo is abandoned is the reviewer's judgment, for example a repo that is inactive, leaves issues unanswered, and has no maintainer who answers at the support URL. An archived repo fails the Structure gate instead. Good-practice signals never push the axis down and never fail an app.

The other signals (tagged releases, issue responsiveness, contributors, CHANGELOG, CI) are good-practice indicators, not requirements. There is no requirement to cut releases or use versioning; a release is a positive signal that the team follows good practices, and its absence is worth a suggestion, never a failure. CI is the weakest of these as evidence for an app repo: most Appverse apps are config-and-template repos with little to test beyond YAML validity and shell/ERB lint, which Appverse Review already checks. It still counts once toward the level like the others, but a missing CI workflow is the last suggestion to make and never a reason to hold an app back.

Decision rubric

The reviewer's decision. Derived from the gate criteria and from properties of individual findings, never from the signal levels. The automated review suggests a decision; the reviewer makes it.

In a monorepo, apply the repo-level criteria and every High or Critical security finding to the whole repo first, then decide each app (see Repo shapes). A failed Structure gate sets at least Request changes whatever severity the finding carries, since every Structure row is a gate; at Critical it sets Reject. A Structure note tagged other: is not a gate row and sets a floor only at High or Critical. A security or upkeep (MNT-01) FAIL at High sets at least Request changes and at Critical sets Reject. A WARN sets no floor, and a finding the reviewer dismissed sets none; a severity the reviewer changed sets the floor at the new severity. check-decisions.py fails a review whose decisions fall below the floors those findings set, or whose recommendation is milder than its strictest Per-app decision, and the review page does not offer a decision below them. Documentation, portability and code-quality findings never set a floor.

Accepting an app (with or without suggestions) needs its duplicate check recorded on the review page; the outcome itself does not limit the decision. The Software entry and the vocabulary terms are the reviewer's to settle before accepting. The automated review performs the catalog reads and records what they returned; the duplicate decision and any Software entry creation stay with the reviewer, as the Reviewer Process describes.

Appendix: common issues in real apps

Based on analysis of apps currently in the Appverse: