Перейти к основному содержимому

Тестовая матрица

Цели​

  • Проверить immutable published versions.
  • Проверить answer schema validation и checking rules.
  • Проверить скрытие solutions, answer keys и teacher notes.
  • Проверить usage с LMS/competitions.
  • Проверить activity/program templates и locked snapshots.
  • Проверить public catalog projection.
  • Проверить relation semantics and prerequisite cycles.
  • Проверить attempts, answer checks, manual review и evidence.
  • Проверить idempotency проверок и интеграционных команд.

Unit tests​

Versions​

  • draft version can be edited;
  • published version cannot be edited;
  • publishing requires statement, answer schema, checking rule and taxonomy minimum;
  • new version preserves problem identity;
  • retired version remains readable by historical attempts.

Answer schema​

  • exact value validation;
  • numeric tolerance validation;
  • single choice options;
  • multiple select normalization;
  • text normalization;
  • file upload constraints;
  • composite parts validation.
  • unit value validation;
  • proof/free text manual review requirement.

Checking​

  • deterministic auto check returns stable result;
  • manual-only problem creates manual review;
  • hybrid check falls back to manual review;
  • scoring rule max score applied;
  • override requires reason and permission.

Taxonomy and sets​

  • taxonomy node uniqueness;
  • problem taxonomy weight;
  • public task requires primary topic;
  • source hierarchy has no cycles;
  • problem set item ordering;
  • published set requires items;
  • locked usage cannot change version.

Relations​

  • no self-link;
  • duplicate/analog/similar/same_method symmetric normalization;
  • prerequisite directed;
  • prerequisite cycle rejected;
  • machine-suggested similar requires moderation before becoming analog.

Activity/program templates​

  • activity template requires section and valid items before publish;
  • activity template rejects roster/schedule/attendance fields;
  • published activity template immutable;
  • program template supports two tracks;
  • program template supports 32 activities in a track;
  • program template rejects enrollment/calendar/progress fields.

Export snapshots​

  • snapshot includes problem version ids, order, roles and payload hash;
  • snapshot excludes hidden answer key unless target scope allows;
  • locked snapshot cannot be mutated;
  • source template change creates new snapshot;
  • duplicate export with same idempotency key returns same snapshot.

Integration tests​

LMS usage and attempt​

  1. LMS creates usage for lesson block.
  2. Student starts attempt.
  3. Student submits answer.
  4. Task-bank checks answer.
  5. Evidence is created.
  6. LMS receives result event.

Ожидание: attempt references exact problem version.

LMS imports activity template​

  1. Publish activity template with sections and items.
  2. Create export snapshot for LMS lesson.
  3. Lock snapshot.
  4. LMS creates lesson instance.

Ожидание: snapshot keeps problem versions and LMS owns schedule/roster/attempts.

LMS imports program template​

  1. Create program with two tracks and 32 activities in one track.
  2. Export program snapshot to LMS.
  3. LMS creates course plan.

Ожидание: task-bank has no enrollment/schedule/progress fields.

Competition locked usage​

  1. Competition creates locked usage for tour.
  2. Student submits answer before publication.
  3. Solution endpoint is requested by student.

Ожидание: solution hidden until competition visibility policy allows it.

Competition locked snapshot​

  1. Create competition problem set or activity template.
  2. Assign main/reserve/tie-breaker/sample roles.
  3. Create locked snapshot.
  4. Update source problem after lock.

Ожидание: tour still renders locked version and scoring/rubric snapshot.

Public catalog projection​

  1. Publish problem version.
  2. Mark public profile as published.
  3. Request /public/tasks/{taskId}.
  4. Request hidden answer key through public endpoint.

Ожидание: projection has canonical URL /tasks/<taskId> and hidden key request fails.

Manual review​

  1. Proof answer submitted.
  2. Auto check returns needs manual review.
  3. Reviewer applies rubric.
  4. Answer check completed.
  5. Evidence created.

Version immutability​

  1. Publish version 1.
  2. Create attempt against version 1.
  3. Create version 2 with changed statement.
  4. Read old attempt.

Ожидание: old attempt still renders version 1 statement.

E2E scenarios​

Автор публикует задачу​

  • создаёт problem;
  • добавляет version;
  • заполняет statement, answer schema, key, solution, taxonomy;
  • preview check;
  • отправляет на review;
  • methodist publishes.

LMS использует задачу в домашке​

  • LMS registers usage;
  • student solves;
  • teacher sees allowed solution;
  • result goes back to LMS;
  • evidence generated.

Олимпиада использует закрытую задачу​

  • competition creates locked usage;
  • answer key hidden from students;
  • checker can review;
  • after publication training set can be exposed by separate rule.

Security tests​

  • student cannot read answer key;
  • student cannot read hidden solution before release;
  • public training cannot include closed competition problem without publish rule;
  • public catalog cannot include hidden answer key, teacher notes, hidden rubric or embargoed source;
  • snapshot payload cannot include sensitive fields without target scope;
  • teacher reads solution only in assigned context;
  • answer key access writes audit;
  • raw answer absent from public events;
  • manual override requires permission and reason;
  • export of hidden solutions requires admin permission.

Checking and idempotency tests​

  • duplicate submit with same idempotency key returns same attempt;
  • duplicate check returns same answer check;
  • checker failure does not lose attempt;
  • external checker timeout creates failed check or manual review;
  • scoring changes require new checking rule version.
  • snapshot lock with same payload hash is idempotent.

Data quality tests​

  • broken asset prevents publish;
  • missing taxonomy warns or blocks by policy;
  • duplicate imported problem creates review queue, not silent merge;
  • prerequisite import with cycle creates review error;
  • analog/similar import keeps separate relation types;
  • answer schema migration keeps old attempts renderable.

Acceptance gates​

  • All state transitions covered by unit tests.
  • Published version immutability enforced.
  • Hidden solution/key negative tests exist.
  • Auto, manual and hybrid checking covered.
  • LMS and competitions usage integration covered.
  • Activity/program snapshot integration covered.
  • Public catalog projection security covered.
  • Evidence generation traceable from answer check.