# Detailed execution plan — full 2019–2021 UTS study

Version 3, 4 November 2026 (IST). Supersedes the 24-date proposal.  
User direction: process all three years and delete files after processing when they are no longer needed.

## Scope and research question

Process **every available normal-session NSE CM order/trade pair in 2019, 2020 and 2021** for an outcome-independent, matched stock sample. The metadata inventory contains 744 complete session pairs before special-session exclusions: 245 in 2019, 252 in 2020 and 247 in 2021. Full-market source sizes total approximately **2.608 TB**.

This means the full available three-year calendar for selected stocks, not every stock on the exchange. The initial target is 20 stocks per year, ten in each liquidity group, fixed within each year. Report missing source dates and special-session exclusions explicitly; do not call unavailable dates analyzed.

Question: how does signed midpoint response vary with relative execution size, and how does that relationship differ with stock liquidity after accounting for obvious price, tick, spread, volatility and timing differences?

The existing four-stock May pilot remains development evidence. It is not the final three-year assignment.

## 1. Freeze the design before bulk outcomes

- Construct each annual cohort from contemporaneous ordinary-equity records and the 20 sessions before its first study date.
- Use prior traded value for liquidity groups; match on prior price and volatility where overlap exists. Review instrument status, identifiers and corporate actions.
- Keep cohorts fixed within year, using deterministic tie-breaks and exclusions.
- Do not choose stocks or dates because their impact results are convenient.
- Primary new-study horizon: 30 seconds. Secondary horizons: 1, 5 and 60; additional: 10 and 300 seconds.
- Primary relative size: shares / prior 20-session average daily shares. Value / prior average daily traded value is an alternative.
- Keep negative and zero responses.
- The main method uses clock-certain, mutually agreeing reconstructed quotes sampled once per second. Store a half-second grid for phase/resolution sensitivity.
- Record a deterministic development/evaluation date split within every year before fitting; exclude already examined pilot outcomes from honest evaluation.
- Report year-specific, pooled, development/evaluation and leave-one-year-out evidence.

## 2. Storage architecture and authorized deletion

Execution remains VPS-only. Keep the existing research Compose and separate progress site running. Enforce **55 GB project storage** and **at least 75 GB free**.

Deleting processed full-market archives makes calendar-wide processing feasible. It does not make retaining every raw message forever feasible.

For each day:
1. Download one archive at a time using verified range checkpoints, or reuse a currently saved source.
2. Verify its full Dropbox content hash and parse the frozen cohort once.
3. Retain lossless selected messages until the day's reconstruction, reconciliation and planned feature production are complete.
4. Preserve compact selected executions, trade-to-aggressor mappings, half-second quote grids, denominators, audit evidence, provenance and the statistical outputs needed for all planned reruns.
5. Retain full selected order replay inputs for a predefined 24-date audit set spread across all years and for material failures that need investigation.
6. On other completed dates, release bulky order replay inputs after the compact analytic data and diagnostics are committed.
7. Delete full-market scratch archives only after verified downstream commit. Cached May originals are also eligible for release, but only after processing the **new annual cohort**, so they will not have to be downloaded again for this study.
8. Record every release and its source/output identities. A released, processed source must not appear as an incomplete download.
9. Never delete the only uncommitted copy. Missing files must not be mistaken for successful work.

The compact executions plus quote grids allow alternate horizons, normalizations, aggregation, controls and resampling without another source transfer. A new matching-engine design or a new nonselected stock could still require original order messages; do not promise otherwise.

Losslessly compress the old pilot's regenerable binary intermediates first, verify decompressed hashes, and preserve restoration instructions. Preserve its reports and failed-run evidence.

## 3. Benchmark, runtime and staged operation

The complete source volume is about 2.6 TB, less sources already retained. This is a **multi-day job**, not a one-day commitment on the current VPS.

Before the long run:
- Verify dependencies and meaningful invariants.
- Freeze annual cohorts and the full session manifest.
- Benchmark at least one day from each year to test layouts, matching behavior, memory, compression, transfer rate and output size.
- Process the already cached May dates early, using the frozen 2020 cohort, to release space without another download.
- Forecast final compact-data size, peak scratch use and runtime from measured results.
- Adjust simultaneous working-set size first. If necessary, choose a uniformly smaller annual cohort before bulk outcomes, with the decision recorded.
- Keep storage gates active during downloads and native processing; pause cleanly on resource or authentication failures.

The runner must continue from capture through analysis and deliverable generation automatically, rather than stopping after downloads.

## 4. Required computation per date and stock

### References and reconciliation

Preserve official daily reference files and the preceding 20-session denominators. Check stock identity, total executions, shares and traded value. Use official adjusted previous-close information where relevant; investigate corporate-action discontinuities and report exclusions.

### Reconstruction and audit

Preserve physical source order and the corrected timestamp-eligible event rule. DAY market-order remainders may passivate; IOC remainders do not.

Check:
- Unique trade IDs and correct incoming order identity.
- No impossible incoming quantities or unexplained execution shortfalls.
- All included continuous executions accounted for.
- Held-out execution-price agreement and price-perturbation invariance.
- Order-only / execution-informed quote agreement.
- Clock ambiguity, one-sided/crossed states and invalid quote gaps.
- Failures by date, stock, group, size and volatility.

Retain the existing exact execution-accounting checks, 99% first-trade price agreement, minimum 95% pre-quote coverage, minimum 95% post-quote coverage and maximum 1% crossed pre-quotes. Do not weaken gates to keep a preferred result.

Independent bid/ask data would strengthen verification if it becomes available. Internal checks must never be labelled independent exchange-quote verification.

### Compact reusable analytic data

Preserve selected trade identifiers, timestamps, quantities, prices, incoming-event mappings and signs; half-second quote grids with validity/agreement flags; and reference metadata. Compute:
- Responses at 1, 5, 10, 30, 60 and 300 seconds.
- Both prior-volume and prior-value normalizations.
- Quoted/effective/realized spreads and their accounting identity.
- Incoming-event aggregation and balanced-horizon samples.
- Correctly lagged price trend and order-flow measures.
- Lagged volatility, relative tick/price measures and time buckets.
- Market-adjusted sensitivity using a clearly labelled, leave-one-stock-out liquid-stock basket where reliable.
- Timing-grid phase/resolution variants and exclusion diagnostics.

Keep full-sample bin statistics and sufficient statistics for the planned controlled models. A deterministic compact regression/inspection sample may be retained additionally, with sampling and weights recorded. It must not silently replace full-sample analysis.

## 5. Empirical analysis

1. Sample construction, overlap, corporate-action and quality tables.
2. Relative-size distributions and common support.
3. Identical-bin conditional signed-response curves, with equal stock-day weighting and an equal-stock sensitivity.
4. Controlled estimates with stock or firm-year, date and time-of-day effects; flexible size terms and group interactions; pre-spread, lagged volatility and other correctly timed controls.
5. Recognize that a fixed group indicator is absorbed by the corresponding fixed effects. Do not report unidentified coefficients.
6. Compare free-power, square-root and linear shapes on the signed scale using held-out prediction, not training fit alone.
7. Repeat across years and evaluation periods; examine event aggregation, alternative definitions, timing, support and leave-one-stock/year-out stability.
8. Address stock/date dependence in uncertainty estimates; do not treat millions of trades as independent observations.
9. Report economic magnitudes in basis points at supported sizes, uncertainty and limitations.

Individual executions are not institutional parent metaorders. Finite-horizon responses are not proven permanent impact. The design is descriptive and cannot establish an isolated causal liquidity effect.

## 6. Final outputs

Working targets, subject to the course's actual instructions:
- An approximately 8–12-page main report with a direct answer.
- Technical appendix covering provenance, validation, worked examples, definitions and robustness.
- Approximately 10–12 presentation slides.
- Reproducibility package: code, environment, frozen specification, manifests, checkpoints, seeds, release log and resume instructions.
- Website with the full-calendar job separate from the May pilot: actual years/dates/stocks, current phase, completion, failures, transfer and storage information, and final downloads.

Essential exhibits: sample/overlap table, attrition/quality table, common support, 30-second curves with uncertainty, controlled group comparisons, balanced response paths, annual/evaluation results and robustness summary.

Every numerical claim must trace to a saved table. Report/slides/figures must agree on sample, units, weighting, exclusions and uncertainty.

## 7. Resumability and completion

Each stage has input/code/config identity and a completion marker written last. Resume skips verified completed stages and restores only required intermediates. Source releases occur only after downstream commit. Keep periodic status/heartbeat updates and bounded retries. A failure must never become a success marker.

Completion requires:
- All available normal-session pairs across 2019–2021 attempted, with unavailable dates recorded.
- Frozen annual cohorts and explicit technical exclusions.
- Credible validation, common support, controls, evaluation and robustness.
- Completed, consistent report, appendix, slides and reproducibility package.
- Storage and free-space limits respected throughout.
- No laptop processing, no library-source writes, and no unsupported claims about missing data or independent quote validation.

The detailed measurement, literature and presentation guidance in plan_v1_may2020.md remains useful. The sampled-calendar plan is retained as plan_v2_sampled_three_years.md, but its 24-day limit and preserve-all-raw policy are superseded by the user's full-calendar and post-processing deletion instructions.