NEMWeb feeds
Where the tables actually come from. AEMO publishes the same data through three channels with different windows, different file naming and different failure modes, and choosing the wrong one is how a backfill quietly loses a week.
https://www.nemweb.com.au. Paths and file patterns below are the ones the simulator used from 2026-05 to 2026-08.
The three channels
1. Current reports
/Reports/Current/<report>/
A rolling window of zipped MMS CSV files, one per run. The window is roughly two days. This is the live feed: fast, small, and it forgets.
File names carry a run timestamp: PUBLIC_DISPATCHIS_<yyyymmddHHMM>_<seq>.zip.
2. Report archive
/Reports/ARCHIVE/<report>/
One zip per day, and each of those contains the day's individual report zips. About 13 months of history, running about two days behind Current. This is what a gap longer than the Current window has to be filled from.
3. MMSDM monthly archive
/Data_Archive/Wholesale_Electricity/MMSDM/<year>/MMSDM_<year>_<mm>/MMSDM_Historical_Data_SQLLoader/DATA/
One zip per table per month. This is where standing data lives: the constraint definitions themselves, and the unit registry.
Current naming: PUBLIC_ARCHIVE#<TABLE>#FILEnn#<yyyymm>010000.zip. Legacy naming: PUBLIC_DVD_<TABLE>_<yyyymm>010000.zip.
The # characters in an MMSDM file name arrive URL-encoded as %23. Decode before matching, and match on the delimiters: looking for GENCONSET anywhere in the name will also match GENCONSETINVOKE. Match #GENCONSET# instead.
The reports, one card each
DispatchIS_Reports
/Reports/Current/DispatchIS_Reports/ and /Reports/ARCHIVE/DispatchIS_Reports/The published dispatch solution. Everything the simulator ever labelled came from here.
Tables: DISPATCHCONSTRAINT, DISPATCHPRICE, DISPATCHREGIONSUM, DISPATCHINTERCONNECTORRES, DISPATCHCASESOLUTION, DISPATCHBLOCKEDCONSTRAINT, CONSTRAINTRELAXATION_OCD and DISPATCH_LOCAL_PRICE.
- The file arrives before the interval ends. NEMDE solves ex-ante at interval start, so the file for the interval ending T uploads at T minus 4 to T minus 1 minutes. Measured 2026-07-14 over 577 files: start plus 1 to 2 minutes typical, plus 3 at p99, plus 4 maximum.
- So "fetch the newest file and stop" is always one interval behind. At a boundary plus 25 seconds the newest file is the roughly 4-minute-old previous label. The freshest label at any instant is the in-progress interval, and a page showing an interval that equals or trails its own generation minute has the bug.
- Absence means not binding. The report carries all inter-regional constraints but only the binding intra-regional ones. Reindex before you count.
P5_Reports
/Reports/Current/P5_Reports/ and /Reports/ARCHIVE/P5_Reports/AEMO's own five-minute pre-dispatch: a fresh run every five minutes, each covering the next twelve intervals, so roughly an hour ahead. P5MIN pass-through is the baseline any hour-ahead model has to beat.
Tables: P5MIN_CONSTRAINTSOLUTION and P5MIN_REGIONSOLUTION, which ride the same zip.
- A P5MIN "gap" older than seven days is almost always a retention prune boundary in your own database, not missing AEMO data.
- Seven days of raw retention makes any P5MIN feature era-limited: it exists only for the tail of a training window, and a walk-forward evaluation correctly punishes that. The cure is a condensation table, not patience.
- INTERVENTION is part of the key: the intervention pricing pass and the physical run are different rows for the same interval.
PredispatchIS_Reports
/Reports/Current/PredispatchIS_Reports/ and /Reports/ARCHIVE/PredispatchIS_Reports/The half-hourly outlook to the end of the next market day. This is the forecast a desk actually watches, and the one a model has to beat at 1 to 24 hour leads.
Tables: PREDISPATCHCONSTRAINT, PREDISPATCHREGIONSUM and the rest of the pre-dispatch package.
- The file is named after its first forecast interval, roughly 30 to 60 minutes after publication, not after the publication time. Watermark on the newest run's minimum DATETIME, never on the maximum LASTCHANGED, or the watermark walks into the future and stops fetching.
- PERIODID is inconsistent with the other pre-dispatch tables. Join on DATETIME.
- Each run overwrites the previous run's view of the same target interval. If you need to know which run first flagged an hour, you have to log it yourself as it arrives.
Short_Term_PASA_Reports
/Reports/Current/Short_Term_PASA_Reports/ and /Reports/ARCHIVE/Short_Term_PASA_Reports/Capacity adequacy at half-hourly resolution for roughly days 2 to 7. Not a dispatch forecast: these are LOR-style adequacy signals.
Tables: STPASA_CONSTRAINTSOLUTION and STPASA_REGIONSOLUTION.
- Only flagged rows are published. Every stored row has a non-zero CAPACITYMARGINALVALUE, so absence means not flagged, not "forecast to be fine".
- A constraint repeats once per STUDYREGIONID the run associates it with, and STUDYREGIONID is in the key. Deduplicate before counting constraints.
- Only about 150 constraints are modelled at all. Treating this as week-ahead coverage of the constraint population overstates it badly.
- RUN_DATETIME equals the filename timestamp, which makes this feed pleasantly easy to watermark.
Dispatch_SCADA
/Reports/Current/Dispatch_SCADA/ and /Reports/ARCHIVE/Dispatch_SCADA/Instantaneous telemetered MW per unit. Small, high-cardinality, and the value source for the T leaves of a right-hand-side term tree.
Table: DISPATCH_UNIT_SCADA.
- About 1,100 rows per five-minute interval. Only recent intervals matter for right-hand-side evaluation, so this is a feed to prune aggressively: the simulator kept seven days.
- A term tree's
SPD_IDfor a T leaf is writtenDUID.CONNECTIONPOINTID, not a bare DUID. Split before joining.
Network
/Reports/Current/Network/ and /Reports/ARCHIVE/Network/The planned-outage book. One snapshot carries the whole multi-year book, so there is nothing to backfill: fetch one file and you have everything.
Tables: NETWORK_OUTAGEDETAIL, NETWORK_OUTAGECONSTRAINTSET, NETWORK_OUTAGESTATUSCODE, plus the equipment and rating tables.
- PUBLIC_NETWORK file names use a 14-digit run stamp (
yyyymmddHHMMSS), where the other current reports use 12 digits. A shared filename parser needs to accept both. - Watermark on the file's run time, not on LASTCHANGED. A LASTCHANGED watermark refetches quiet snapshots forever.
- A snapshot has no history. A status that changed between two snapshots leaves no trace. If you want first-seen status times you must log them as they arrive.
- The outage to constraint-set mapping is thin. Measured 2026-07-27: only 224 of the 1,341 outages scheduled to start within 28 days, 17%, carried any mapping. Foresight built on it must omit the unmapped majority rather than guess.
STPASA_DUIDAvailability
/Reports/Current/STPASA_DUIDAvailability/ and /Reports/ARCHIVE/STPASA_DUIDAVAILABILITY/Per-unit PASA availability and recall period for every scheduled resource, covering days 2 to 7. Roughly 1.6M rows per day raw, which is why the simulator condensed it at parse time instead of storing it.
Table: STPASA_DUIDAVAILABILITY, condensed into the project's own STPASA_UNIT_AVAILABILITY.
- The archive form is MONTHLY bundles, not daily zips like the dispatch feeds. A few missing days cannot be patched the easy way.
- Note the case difference between the Current path and the ARCHIVE path. They are not the same string.
- Because the product covers days 2 to 7, the leak-free as-of view for a target day is the run whose date plus two days is at or before that day. Using a later run leaks the future into a feature.
NEMDE price setter (XML, monthly archive only)
/Data_Archive/.../NEMDE/<yyyy>/NEMDE_<yyyy>_<mm>/NEMDE_Market_Data/NEMDE_Files/NemPriceSetter_<yyyymmdd>_xml.zipPer interval and region, which offer bands set each market's clearing price. 288 inner XML files per day. This is the only non-CSV feed in the whole stack, and it answers a question no MMS table answers.
Table: PRICESETTER, a project table, not part of the MMS model.
- There is no live feed. Verified 2026-07-27: no Reports/Current or ARCHIVE Price_Setter directory exists, and no MMSDM table carries it. Each month appears roughly two weeks after month end; June 2026 appeared on 2026-07-14. Anything built on it is historical by construction.
- The energy price is usually set by a blend, which is why an RRP can match nobody's offer. Each contributing band carries an
Increase: the MW it moves per 1 MW of reference-node demand, negative for counter-flowing units. - A region's price setter can be an MNSP interconnector, for example Basslink setting the Tasmanian price, published as
T-V-MNSP1,TAS1. - Tied bands publish as one comma-joined unit string, for example
GSTONE1,ENOF,7,GSTONE6,ENOF,7. Band number is missing on about 16% of rows. - Files carry a fixed
+10:00offset. Strip it and store naive market time.
MMSDM monthly archive (standing data)
/Data_Archive/Wholesale_Electricity/MMSDM/<year>/MMSDM_<year>_<mm>/MMSDM_Historical_Data_SQLLoader/DATA/The constraint definitions themselves, plus the unit registry. Without this channel a solution table is a stream of ids with no meaning attached.
GENERIC_CONSTRAINT package: GENCONDATA, GENCONSET, GENCONSETINVOKE, GENERICCONSTRAINTRHS, GENERICEQUATIONDESC, GENERICEQUATIONRHS, SPDCONNECTIONPOINTCONSTRAINT, SPDINTERCONNECTORCONSTRAINT, SPDREGIONCONSTRAINT, EMSMASTER.
PARTICIPANT_REGISTRATION package: DUDETAILSUMMARY, DUDETAIL, STATION, PARTICIPANT.
- Monthly archives are change-sets, not snapshots. Standing coverage accretes over months of ingest. One month alone leaves most constraints unresolvable, which is why the GENCONDATA join falls back to the latest held version and the operator falls back to the naming convention (
F_*is>=, network is<=). - Ingest months in chronological order. Backfilling an old month after a newer one lets a stale first draft clobber an amendment under the same key. After any historical backfill, re-ingest the newer months.
- The exception, measured 2026-07-19: the PARTICIPANT_REGISTRATION monthly files for DUDETAIL and PARTICIPANT are FULL SNAPSHOTS. One month gave 751 of 751 active units, so no multi-month backfill loop is needed for those two.
- The CSVs inside use abbreviated package names (SPDRC, GCRHS, GEQRHS and friends), not the table names on the tin. A parser matching on the visible package will silently skip files.
- Versions effective in the current, unpublished month have no public standing row anywhere. That is a coverage boundary, not a bug: it self-heals when the month publishes.
Choosing a channel
- Live operation: Current only, and watermark carefully.
- A gap of hours: still Current, if it is inside the roughly two-day window.
- A gap of days to a year: the daily ARCHIVE zips.
- Constraint definitions, or anything older than about 13 months: the MMSDM monthly archive, ingested in chronological order.
- Price attribution: the NEMDE monthly XML, and accept that it is always about two weeks behind.
The feed layer is what 10. Constraints in the AEMO tables builds on.