CHO-FAM · Month 01 · Week 03 Recorded: 2026-10-06 Evidence: interactive study conversation; not an executed benchmark. Curriculum: Week 03
How should CHO-FAM define observation units, denominators, and uncertainty so that probabilistic AI outputs and operational outcomes support reproducible decisions?
Working hypothesis: explicit outcome definitions and uncertainty reveal limits that point estimates alone conceal. This is a design hypothesis, not an experimentally verified result.
Deadline detection was a teaching example, not a representative definition of company work. The user clarified that general work includes complex domain jobs and referenced gstack as a product-work reference.
An example whole job is delivering a feature against agreed requirements, including planning, implementation, review, testing, and user acceptance. Individual email checks, tool calls, or alerts are subordinate observations.
Tests come before acceptance. Passing tests does not by itself establish that the user need was met. When tests pass but acceptance fails because a requirement was misunderstood, test pass rate remains unchanged while acceptance and end-to-end success decline.
| Metric | Unit and denominator | Interpretation |
|---|---|---|
| Required-test pass rate | Jobs passing required tests / jobs with completed required testing | Technical checks; report jobs not yet tested separately |
| User acceptance rate | Accepted jobs / jobs with completed acceptance decisions after testing | Conditional on reaching acceptance; not the whole workload |
| Confirmed end-to-end success proportion | Jobs passing tests and accepted / all eligible jobs in a fixed cohort | Interim confirmed progress; pending is not automatically failure |
| Finalized end-to-end success rate | Successful jobs / eligible jobs with finalized outcomes | Report pending count and proportion alongside this conditional rate |
| Deadline precision | Correct deadline-positive decisions / adjudicated deadline-positive decisions, deduplicated by task/deadline episode | Separate confirmed alerts and caution alerts |
| Alert burden | Delivered notifications per task/deadline episode | Detect repetitive notifications hidden by deduplicated precision |
| Subsystem measurements | Calls, validation attempts, or proposed actions, as appropriate | Per-call validity, latency, cost, and validation behavior |
A whole job is counted once for job-level outcomes; repeated calls remain visible at subsystem level. Keep run IDs nested under case/job IDs so repeated executions are not mistaken for independent jobs.
Recall measures detected true positives among all adjudicated actual positives; precision measures true positives among adjudicated predicted positives. False-positive rate uses actual negatives as its denominator, not alerts. Low prevalence can produce poor precision even when recall and false-positive rate appear reasonable.
Costs of missed deadlines and false alerts affect policy choices. Priority or caution flags can justify stricter recall requirements. Product policy and user preferences define acceptable trade-offs; the model does not independently change those constraints.
For a flagged task with uncertain deadline evidence, ask the user for confirmation. If confirmation is absent when intervention is needed, send an alert explicitly labeled Unconfirmed caution, with the source, suspected deadline, remaining time, and essential work.
Use estimated work plus review, confirmation, or physical preparation needs to determine useful lead time. A fixed lead time is a fallback when estimation is unavailable, not a substitute for a known estimate.
Insufficient remaining time makes the alert more urgent. It should display essential work and the time shortfall without silently changing scope or assuming permission to omit required review. Investigate why the alert was late afterward; immediate intervention should focus on action.
Caution outcomes are adjudicated later as confirmed, refuted, or unknown. Unknown is neither a false positive nor a true positive. Report unresolved counts and proportions; precision on resolved cases alone may be biased. Keep original alert type and evidence timestamp even after later resolution.
An unconfirmed caution may contribute to notification coverage without establishing a correct factual prediction. Its label alone must not exempt a misleading or unnecessary alert from later evaluation.
The observed cohort result can be calculated without bootstrap. Bootstrap resamples existing observations to estimate sampling uncertainty; it does not rerun the model, create unseen cases, or repair unrepresentative case selection.
For generalization across jobs, resample whole case/job clusters, retaining their repeated runs, alerts, and subsystem observations together. Separately examine model variability through repeated execution of fixed cases. State which uncertainty each analysis addresses.
Preserve job context when sampling: selecting whole jobs randomly is different from randomizing dependent steps within a job.
Users may review easy jobs first. Acceptance among reviewed jobs can then overstate overall performance and may decline as harder jobs are resolved. This decline is possible, not guaranteed. Report review coverage and pending outcomes; a narrow interval does not remove selection bias.
The user clarified that “expected range of error” meant an uncertainty range or margin of error, not job size. The aim is enough evidence for a decision, not maximum statistical precision:
These interpretations depend on representative observations and appropriate interval assumptions. If operating conditions or error rates change, analyze recent conditions and versions separately instead of hiding deterioration in pooled historical results.
Repeatedly checking a conventional interval and stopping whenever it looks favorable can invalidate its coverage. Predefine a fixed sample/stopping rule, or use a sequentially valid method when designing the next experiment.
A hypothetical cohort contained 30 jobs: 24 passed tests; among these, 18 were accepted, 4 rejected, and 2 awaited acceptance.
The final range is an outcome-completion bound, not a bootstrap confidence interval. These numbers are instructional only and must not be placed in raw experiment logs.
Confirmed conversational outcomes: metric-unit separation; tests followed by acceptance; distinct unresolved caution states; urgency under insufficient lead time; selection-bias awareness; decision-relevant uncertainty; expansion to whole domain jobs.
Not evidenced by this conversation: completion of assigned textbook reading, an implemented measurement harness, 30-case execution, raw JSONL, cost/latency measurement, or calculated bootstrap confidence intervals. No synthetic benchmark results are claimed.
The curriculum’s administrative completion/no-carryover convention is separate from empirical verification. This report adds no remediation assignment or extra hours.
Define the target job population, agreed acceptance criteria, evaluation cutoff, and pending-outcome handling before running comparisons. Use the same whole-job cases and comparable conditions for candidate and baseline; preserve approval requirements and record all failures and costs.
Predefine the improvement worth paying additional cost for, safety limits, and stopping rule. A proposed planner with uncertain benefit and higher cost was discussed as a decision exercise, but no adoption choice or performance conclusion was established.
Conclusion: For CHO-FAM, evaluation should preserve whole-job context, distinguish technical tests from user acceptance, and quantify uncertainty when it can change a decision—without converting unresolved outcomes or narrow task examples into evidence of company-wide success.
저장소 원본 · 공개 사본 갱신: 2026-10-12