Project Overview

GreeGrid Solutions launched a Smart Grid Smart Pricing Pilot for 100 residential homes to evaluate time-based electricity pricing. The project was executed using Microsoft Fabric, replacing error-prone Excel workflows with a governed, automated analytics solution. 

• Pilot focused on dynamic electricity pricing based on consumption time

• Higher tariffs during peak hours (6 PM – 9 PM) to promote energy conservation

• Sensor-generated data ingested daily from smart meters

• Strict data privacy and compliance requirements enforced

• End-to-end solution built using Fabric Pipelines, Lakehouse, PySpark, and Delta tables

   

​​

Business Problem

The pilot program was blocked due to operational, analytical, and compliance risks:

• Smart meter data contained invalid values (negative readings, extreme spikes)

• Billing team manually calculated peak charges in Excel, causing frequent errors

• Timestamps were inconsistently interpreted, leading to incorrect peak pricing

• Customer email addresses (PII) were stored in shared analytics files

• Legal team flagged privacy violations, threatening to shut down the pilot

• No automated process existed for data quality enforcement or secure analytics

​

Project Objective

• Automate secure, accurate smart billing calculations

• Enforce data quality rules before billing logic

• Implement privacy-first analytics without losing traceability

• Replace Excel-based processes with scalable Fabric notebooks

• Enable leadership to evaluate the Smart Pricing model confidently

​​​

Project Design

• Designed and implemented a PySpark-based analytics notebook that directly leverages existing Lakehouse files & tables to address data inconsistency.

• Bronze Layer (Raw Ingestion)

  • Ingest JSON and CSV files via Fabric Pipelines

  • Store raw meter readings and customer reference data

• Quality Control Logic (PySpark)

  • Filter invalid readings (kwh <= 0 or kwh > 100)

  • Route bad data to the Quarantine_Readings table

• Security & Compliance Layer

  • Join meter data with customer reference data

  • Hash customer emails using SHA-256

  • Ensure no raw PII flows into analytics tables

• Business Logic Layer

  • Identify peak vs non-peak hours using timestamps

  • Apply dynamic pricing rules

  • Calculate the final bill amount using conditional logic

• Silver Analytics Table

  • Store clean, anonymised, billing-ready data

  • Optimised the Delta table for reporting and validation

• Validation & Audit

  • SQL checks to verify hashed PII

  • Clear lineage from raw → clean → billed data

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​​​​​​​​​​​​​​​​​​​​​​​​

 

​

​​

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

​​​​​​​​​​​​​​

​​​​​​​​Business Problem Solved

• Eliminated manual Excel billing errors with automated logic

• Prevented over-billing caused by sensor glitches and phantom spikes

• Enforced regulatory compliance by removing raw PII from analytics

• Enabled trustworthy peak pricing validation for leadership decisions

• Delivered a reusable, innovative billing framework ready for scale

• Created a production-grade analytics pipeline aligned with real utility-sector constraints

​