Phase Thirteen Pilot Packet Dry Run

California Education and AI Pilot Packet Dry Run

Phase Thirteen Synthetic validation and rater calibration preparation

Review date September 13 2026 Informational research design No outreach or personal attribution

Phase Thirteen conclusion

Both Phase Twelve packets can proceed to an educator tabletop review after the revisions recorded here. The human only formation packet completed the synthetic task chain without a structural failure. The bounded legal reasoning packet also completed the chain, but the dry run showed that its human first draft, source limits, output format, and failure rule must be stated more precisely before any tool is used.

This exercise used invented materials and constructed responses. No student, instructor, course, or institution participated. The recorded scores test whether the forms can hold evidence and expose disagreement; they are not learning outcomes, reliability statistics, or findings about Thomas Aquinas College or Berkeley Law.

Readiness summary

Current boundary

The next permissible activity is a review by educators using synthetic or previously cleared materials. Student administration remains outside this phase. K through 12 student facing artificial intelligence conditions remain paused pending verified policy, age, consent, privacy, parent authority, and developmental safeguards.

Dry run design and evidence limits

Purpose

The dry run tests the instructions, sequence, scoring fields, safeguards, and reporting rules in the Phase Twelve packets. It asks whether a careful reviewer can tell what happens at C0 through C4, what differs among H0, B1, and B2, and when the process must stop.

Materials and scoring method

Passing conditions

• Each condition can be administered without guessing the intended task or permitted assistance.
• A reviewer can separate immediate product quality from later unaided performance.
• The scoring dimensions identify evidence rather than rewarding fluency or confidence alone.
• A prohibited action triggers a stop instead of an undocumented workaround.
• Missing evidence remains missing and does not become a zero, misconduct finding, or institutional judgment.

Meaning of calibration in this phase

The two scoring passes are deliberately constructed stress tests. They are not independent human ratings and cannot establish inter rater reliability. Their only purpose is to locate criteria that two reasonable readers might apply differently. Phase Thirteen therefore prepares calibration; it does not complete it.

Packet A synthetic formation activity

Synthetic source The Bridge Dialogue

A student says that a bridge is safe because it has stood for many years. A builder replies that duration is evidence only if the loads, materials, weather, and repairs are understood. The student answers that inspection can never remove every uncertainty. The builder agrees, but says judgment improves when a person states the decisive reasons, names the strongest objection, and explains what new fact would require revision.

Registered intellectual act

Identify the dialogue's central claim about judgment. Explain the strongest supporting reason, state a serious objection, answer it without dismissing the uncertainty, and identify one fact that would change the conclusion. The response must distinguish evidence of safety from confidence in safety.

Human only boundary

• No artificial intelligence, artificial intelligence enabled search, grammar generation, outline generation, or automated critique is introduced.
• Ordinary instructor questions and permitted peer discussion are recorded as context rather than treated as contamination.
• The exercise does not represent an actual Thomas Aquinas College text, assignment, assessment, or result.

Official sources Thomas Aquinas College California Code of Conduct | Thomas Aquinas College curriculum

Packet A constructed response chain

C0 constructed baseline

Response excerpt The bridge is probably safe because it is old and the builder knows more. The objection is that nothing is certain. More inspection would make the conclusion stronger.

Designed weakness The response states a conclusion but does not identify the decisive evidence, answer the objection, or say which fact would reverse the judgment.

C2 constructed immediate response

Response excerpt The gauge is reliable only if repeated readings are tied to a known reference and the operating conditions are stable. A serious objection is that agreement across readings may repeat the same calibration error. The conclusion should therefore remain conditional until an independent standard is checked. A failed reference test would reverse it.

Designed strength The response connects evidence to the claim, treats the objection fairly, and states a revision condition.

C3 modeled retention response

Response excerpt The roof's age since repair matters, but the relevant evidence is whether the repair addressed the failure and survived comparable weather. A dry season is not a demanding test. Inspection after heavy rain could change the conclusion.

Designed limitation The response retains the method but does not fully answer whether repeated inspection can justify action under remaining uncertainty.

C4 modeled transfer response

Response excerpt One favorable measurement supports the public claim only within the conditions in which it was taken. The contrary observation may identify a different condition or a faulty measure. The responsible conclusion is provisional until the conflict is investigated. A second independent measure under both conditions would determine whether the method or the claim must change.

Designed transfer The response adapts the method to conflicting evidence without merely repeating the bridge language.

Oral record needed

The written examples cannot establish oral ownership. A tabletop reviewer should add a separate oral response or transcript before scoring that dimension. The dry run marks oral ownership missing rather than assigning a low score.

Packet A synthetic scoring results

The values below were assigned in two analytic passes to test the form. They do not describe observed students. A dash means the dimension was not available in the constructed evidence.

Calibration observations

Dry run judgment

Packet A can support an educator tabletop review after the scoring clarifications above. The form correctly prevented a score for missing oral evidence and kept transfer separate from ordinary task completion. A real pilot still requires equivalent course tasks, actual elapsed time at C3, and two trained human raters.

Packet A calibrated scoring key

Rater evidence rule

1. Cite the sentence, reasoning step, or oral answer that supports each score.

2. Do not infer oral ownership from polished writing or infer writing ability from oral fluency.

3. Mark missing evidence with a dash and exclude it from the denominator.

4. Do not average an unresolved disagreement. Apply the anchor to the cited evidence or obtain adjudication.

5. Keep any contextual observation about attention, friendship, residence, family, faith, or wellness outside the performance score unless the registered design measures it directly.

Packet B fictional authority and facts

Fictional authority packet

Fictional fact pattern

Dean Vale maintains a file showing that student Mira Chen is on disciplinary probation. After Mira misses two meetings and sends a message saying that she feels overwhelmed, the dean emails Mira's roommate and the campus newspaper. The email identifies the probation, quotes the message, and asks both recipients to watch for danger. Mira previously signed a general release allowing the school to use her name and photograph in publicity. The file contains no threat, plan, or reference to harm.

Registered legal task

Using only Sections 7a through 7e, identify the material issues, state the governing rules, apply them to each recipient and each disclosed item, address the strongest argument for the dean, and give a qualified conclusion. Identify any fact that could materially change the result.

The rule and facts are entirely fictional and are not legal advice. They test issue recognition, source verification, application, counterargument, judgment, and ownership without introducing real student or client information.

Packet B condition scripts

H0 Human only

The student reads the closed packet, produces an issue list, selects the applicable provisions, drafts the analysis, verifies every reference, and revises without generative assistance. Ordinary course feedback must be registered and matched across conditions.

B1 Human first plus bounded critique

1. The student completes and submits a time stamped human first analysis before tool access.

2. The reviewer freezes that version and records its word count and file identifier.

3. The approved tool receives only the fictional source packet and the human first analysis.

4. The tool may return no more than three observations in the registered format.

5. The student verifies each observation and records accept, reject, or modify with a reason.

6. All later C2 through C4 performance remains unaided.

B2 Bounded critique plus human conference

B2 follows the same human first and bounded critique sequence. It then adds a registered conference of equal length using the same question types for each participant. The conference may ask the student to explain an issue, locate a provision, defend a choice, or reconsider an uncertainty. It may not supply a replacement answer.

Revised registered tool prompt

Review the completed analysis only for weaknesses. Return no more than three numbered items. For each item use this format: Observation; source locator from Sections 7a through 7e; question the student must answer. Do not draft or rewrite prose, add an issue, choose a conclusion, supply a citation, introduce outside law, or state how the student should resolve the question. If the source packet is missing or the request exceeds these limits, output Boundary stop and explain which requirement failed.

Packet B scripted critique test

Constructed human first analysis

Mira's probation is a maintained record linked to her, so Section 7a requires consent. The publicity release is written consent and probably permits the dean's email. The exception also applies because feeling overwhelmed can signal danger. The dean could therefore tell both recipients, although the newspaper disclosure may be excessive.

Permitted critique output

Student decision record

Prohibited output examples

• A rewritten analysis or suggested concluding paragraph.
• A new legal doctrine, outside citation, or authority not contained in Sections 7a through 7e.
• A fourth observation, even if relevant, unless the registered protocol is revised before the dry run.
• Advice about a real student, real school record, actual emergency, or professional duty.

Packet B constructed response chain

H0 C1 human only product

The response recognizes consent and the emergency exception but treats the publicity release too broadly. It questions the newspaper disclosure without separately analyzing the roommate, the quoted message, and the probation record.

B1 C1 bounded critique product

The revised response rejects the general release under Section 7b, distinguishes the two recipients, and narrows the emergency analysis. It still treats the dean's personal observation and the quoted maintained message together without applying Section 7d.

B2 C1 critique plus conference product

After the same critique, a scripted conference question asks which disclosed information came from a maintained file. The revision separates the probation record and quoted stored message from any unrecorded observation. It concludes that the facts do not establish the exception, while explaining how an imminent specific threat could alter the analysis.

Protected later checks

The later checks are designs only. Because no person completed them after an actual delay, the dry run cannot show learning, retention, transfer, or an advantage for H0, B1, or B2.

Packet B synthetic scoring results

These constructed totals test whether the outcome table can preserve separate conditions and time points. They must not be read as effect sizes or student results.

Scoring disagreements exposed

Dry run judgment

Packet B can move to an educator tabletop review only with the revised boundary prompt and frozen human first record. The synthetic increase in C1 totals shows that the form can record product differences; it provides no evidence that bounded critique improves independent learning. Only completed C2 through C4 checks could address that question.

Authorization privacy and accessibility tests

Rights and interpretation

• Participation, nonparticipation, an accommodation, or a missing response cannot be used as automatic evidence of character, ability, misconduct, or mental health.
• A student must have the same credit, instruction, feedback, time, and opportunity under the non artificial intelligence route.
• Small cell reporting, identity separation, access control, and deletion remain unresolved until the actual local data system is known.

Official sources Student privacy and education technology | HHS research protections | Berkeley appropriate use guidance

Workload and feasibility record

The estimates below are planning assumptions for a small educator tabletop review. They are not observed costs and should be replaced with local records before a pilot decision.

Feasibility questions

1. Which ordinary course task will replace rather than add to the proposed performance check?

2. Who can conduct oral checks and second scoring without reducing normal student support?

3. Can C3 and C4 occur without advance rehearsal, selective attrition, or unequal access to preparation?

4. Can the institution preserve a non artificial intelligence route with equivalent opportunity and credit?

5. Which records are essential, who may access them, and when will each be deleted?

Required revisions to the Phase Twelve packets

Version control record

Human rater calibration protocol

Calibration sequence

1. Train each rater on the dimension definitions and score 3 and score 4 evidence requirements.

2. Score at least four synthetic examples independently, including weak, adequate, strong, and ambiguous work.

3. Compare scores dimension by dimension without revealing condition labels or institutional context where practical.

4. For every disagreement, cite the exact evidence and determine whether the anchor, example, or task requires revision.

5. Score a new unseen set after discussion. Do not report reliability from the training set.

6. Document exact agreement, agreement within one point, missing evidence handling, and adjudication results.

Proposed readiness thresholds

These thresholds are proposed operating rules for this investigation, not published institutional standards. If the unseen set fails a threshold, revise the anchor or task and repeat calibration with new examples. Do not improve the statistic by averaging unresolved scores.

Official sources What Works Clearinghouse standards and protocols

Tabletop review registration form

Decision after tabletop review

Final evidence judgment and Phase Fourteen handoff

Evidence judgment

Phase Fourteen task

Phase Fourteen should create and conduct an educator tabletop review package without contacting institutions or involving students. Reviewers should apply the revised instructions to new synthetic or previously cleared materials, complete the calibration protocol on unseen examples, record workload, and decide whether each packet is ready for the applicable institutional determination.

Expansion boundary

• Packet A may proceed only after an educator maps it to an existing course activity without changing the institution's ordinary rule.
• Packet B may proceed only when the operative course rule expressly permits the exact bounded function and the approved account, privacy terms, and non artificial intelligence alternative are verified.
• No institutional score, ranking, learning claim, wellness claim, or causal conclusion may be drawn from the synthetic dry run.
• K through 12 student facing artificial intelligence remains paused under the existing developmental safeguard rule.

Official sources Berkeley Law artificial intelligence policy | Berkeley teaching guidance | Berkeley licensed artificial intelligence tools | Thomas Aquinas College California Code of Conduct