Proposed evaluation framework 0.2
How to evaluate Home Power Automation
An HPA evaluation compares coordinated operation with a declared baseline and reports cost, household requirements, constraint violations, manual intervention and action reliability separately. It must state whether the evidence comes from replay, simulation or field operation.
This is a proposed evaluation framework, not a benchmark, certification or leaderboard. GridPassport will use the word “benchmark” only after publishing a versioned dataset, executable procedure and example result that another team can reproduce.
First decision
Choose the evidence track before choosing the metric.
Replay and field measurement answer different questions. Results should never move between the two categories without a clear label.
Deterministic replay or simulation
- Runs multiple control policies against the same versioned input data or model.
- Supports repeatable comparison of planning decisions and counterfactual outcomes.
- Requires explicit models for device losses, thermal response and actions that change later demand.
- Cannot by itself prove field communication reliability, commissioning quality or user behavior.
Field measurement and verification
- Measures operation in an installed home with real devices, integrations and people.
- Can test command delivery, fallback, manual intervention and actual household outcomes.
- Needs an adjusted baseline because weather, occupancy, prices and initial state do not repeat exactly.
- Must report uncertainty and avoid claiming that different periods had identical conditions.
Test manifest
Define the test before running it.
A result is interpretable only when the system boundary, control policy, data and calculation rules are fixed in advance.
- Question and evidence track. State the claim being tested and whether the run is replay, simulation or field M&V.
- Site boundary. List the grid connection, meters, devices, uncontrolled loads and energy flows included or excluded.
- System versions. Record coordinator software, device firmware, integration version and configuration.
- Baseline. Describe the exact alternative control policy, schedules, defaults and user actions.
- Inputs. Version prices, export rules, weather, solar, demand, occupancy assumptions, deadlines and starting device state.
- Constraints. Declare site power, equipment limits, reserve, comfort, EV readiness and manual controls.
- Time basis. Declare timezone, daylight-saving treatment, sampling interval and missing-data rules.
- Metrics. Publish formulas, units, inclusion rules, end-state treatment and uncertainty method before seeing the outcome.
Baselines
The comparator must be an executable policy.
“Without HPA” is not precise enough. The baseline must say what every controlled device would do.
| Baseline | What must be specified | Useful for |
|---|---|---|
| Observed current operation | Existing apps, schedules, defaults, manual interventions and period used to estimate normal behavior. | Field M&V against how the household actually operated before coordination. |
| Device defaults | Exact default mode and version for each battery, charger, thermostat or other controller. | Repeatable laboratory or simulated comparison with manufacturer behavior. |
| Fixed household rules | Every timer, threshold, reserve setting and priority, including conflict behavior. | Comparing adaptive coordination with a practical rules-based setup. |
| Single-signal control | The signal, threshold, response, device scope and any protected household requirement. | Testing whether shared context adds value beyond reacting to price or solar alone. |
Outcome set
Report separate outcomes instead of one composite score.
A lower bill cannot hide an empty EV, lost battery reserve, comfort violation or unreliable control.
| Metric | Proposed calculation | Required disclosure |
|---|---|---|
| Net energy cost |
Sum import energy times interval import rate, subtract export energy times export rate, then add
variable charges included in the test.
cost = sum(import_kWh_i * import_rate_i) - sum(export_kWh_i * export_rate_i) + variable_charges | Taxes, fees, demand charges, export rules and whether unchanged fixed charges are excluded. |
| EV energy shortfall |
Target usable energy minus usable energy available at the declared departure, floored at zero.
shortfall_kWh = max(0, target_kWh - available_kWh_at_deadline) | Target source, deadline, connection period, starting energy and charging losses. |
| Battery reserve shortfall |
Percentage-point-hours below the active minimum reserve.
reserve_shortfall = sum(max(0, reserve_pct_i - actual_pct_i) * interval_hours) | Reserve schedule, state-of-charge source, calibration and periods excluded during an outage. |
| Comfort deviation |
Degree-hours outside the declared lower and upper temperature bounds.
degree_hours = sum(distance_outside_comfort_band_i * interval_hours) | Sensor location, occupancy rule, comfort band, humidity treatment and missing measurements. |
| Peak site power | Maximum interval-average grid import and export, reported separately in kW. | Meter, interval length, phase treatment and whether instantaneous peaks are also reported. |
| Manual intervention | Count and duration of pauses, overrides, schedule edits and recovery actions performed by people. | Which actions count and whether they were requested by the system or initiated independently. |
| Action acceptance | Material commands accepted by the target divided by material commands issued. | Command types, retry policy and how delayed, duplicate or unknown responses are classified. |
| Verified outcome | Material actions whose intended state or energy result was observed divided by actions where verification was possible. | Telemetry used, tolerance, verification window and cases where acceptance could not be physically checked. |
End state matters
A replay cannot claim a cost saving by ending with a depleted battery or undercharged EV. Require comparable end states or assign and disclose a residual value to the energy difference. Battery losses, degradation cost and thermal carry-over also need a declared treatment when material.
Hard constraints
A violation remains visible even when another metric improves.
Device safety, installation limits and manual stops are outside the optimizer's trade space. An intentional violation invalidates the run. An externally caused or measured violation must be reported with cause and response.
Household requirements should be classified before the test as hard, soft or informational. A missed hard requirement is reported as a failure, not converted into a small penalty inside a favorable aggregate score.
Replay procedure
Make counterfactual assumptions executable and versioned.
- Freeze the input dataset, model versions, random seeds and scenario manifest.
- Initialize every policy with the same device state, household requirements and available information.
- Run each policy with identical time steps and rules for forecast updates, missing data and external events.
- Model the effect of each action on later state, including battery efficiency, charging losses and thermal response where relevant.
- Store actions, state transitions, constraint status and metric inputs at each interval.
- Check end-state equivalence, calculate every declared metric and publish machine-readable results.
- Repeat sensitivity cases for forecast error, missing data, device rejection and model uncertainty.
Counterfactual limitation
Measured household demand is not always independent of control. HVAC, hot water and EV actions change later demand and state. A replay must model those effects or limit its claim to the decisions it can evaluate faithfully.
Field procedure
Use an adjusted baseline and report uncertainty.
- Pre-register the question, boundary, baseline model, evaluation period and exclusion rules.
- Collect a baseline period long enough to represent relevant weather, occupancy, prices and device use.
- Commission the system and separate setup faults from normal operation without silently deleting them.
- Measure the reporting period with the same meters, interval treatment and data-quality checks.
- Adjust the baseline only with declared variables and methods; show model fit and residual error.
- Report measured outcomes, adjusted baseline, difference, uncertainty range, missing data and interventions.
- Limit the conclusion to the site, period, equipment and operation actually observed.
No “same conditions” claim
Two field periods do not repeat the same weather, occupancy, prices or starting state. The method should describe normalization and uncertainty rather than imply identical conditions.
Scenario coverage
Test ordinary operation and degraded states.
| Scenario | Primary conflict | Failure variant |
|---|---|---|
| EV departure after a cheap night | Low charging cost versus protected home-battery reserve and site import limit | EV connects late or charger rejects the first schedule |
| Solar surplus before an expensive evening | Battery headroom, EV need, hot water, cooling and export value | Solar forecast is materially too high |
| Heating or cooling during a price peak | Cost and site power versus an occupied comfort band | Room sensor becomes stale or occupancy changes |
| Grid or connection-point power limit | Allocation of limited power among simultaneous flexible loads | External limit arrives late or with an invalid timestamp |
| Loss of a device or cloud connection | Safe degradation while remaining devices continue operating | Delayed acknowledgement creates uncertain command state |
Publication template
A result should carry enough detail to be challenged.
| Report section | Minimum content |
|---|---|
| Claim | Exact question, site, period, evidence track and limits of the conclusion |
| Manifest | Boundary, versions, devices, configuration, baseline, inputs, constraints and time basis |
| Data quality | Coverage, missing intervals, imputation, clock corrections, calibration and excluded periods |
| Results | Every declared metric, hard-constraint status, manual interventions and end-state comparison |
| Uncertainty | Model error, sensitivity cases, measurement limits and confidence or uncertainty interval where applicable |
| Reproduction | Dataset identifier, code or executable procedure, configuration and expected output checksum where publishable |
Current GridPassport evidence
The first deployment is a retrospective scenario replay.
The published first-home case study uses measured household data to compare modeled tariff and control scenarios. It predates this framework and does not demonstrate closed-loop field reliability.
It should be read as early evidence from one home, with the assumptions shown on the page. We will not call it an HPA benchmark result. A future benchmark release needs a versioned dataset, executable method and reproducible example output.
Sources
Measurement and HEMS references.
These sources inform the framework. They do not endorse the HPA term or this draft method.
- ISO 50015:2014: Measurement and verification of energy performance
Current general principles for energy-performance measurement and verification, confirmed by ISO in 2025.
- U.S. DOE: Smart electrical panel-based HEMS
An example of multi-objective HEMS development using laboratory validation and field demonstration.
- ENERGY STAR: Smart Home Energy Management Systems criteria
Includes automated control, overrides, privacy, cybersecurity reporting and population-level field performance.
- IEA 4E EDNA: Residential HEMS and controllers
Provides market context for the range of HEMS functions and integrated residential devices.
Next milestone
Turn the framework into something another team can run.
The next credible release is a small versioned replay dataset, reference policy, metric code and expected result. That work matters more than publishing a score before the method is reproducible.