Smart Fridge

• ESP32-WROVER microcontroller: chip with built-in WiFi, PSRAM memory and camera connector.
• Image sensors (24-pin connector):
  • OV2640: 2 megapixels, standard lens with 68° field of view.
  • OV5640: 5 megapixels, 160° wide-angle lens (wide view to cover the entire inside of the refrigerator).
  Both modules connect to the same 24-pin connector on the ESP32 (dedicated parallel bus).
• DS18B20 temperature probe: waterproof digital sensor using the 1-Wire protocol (a single data wire + 4.7 kΩ pull-up resistor).
• GPIO & ADC: management of the ESP32 input/output pins, with ADC1 constraints when WiFi is active.
• 3.3 V power supply: voltage regulation and wiring (VCC, GND, data lines).
• Reading schematics & pinout: interpreting datasheets to connect each component correctly.
• Development refrigerator — Whirlpool WRF560SEHZ02: French-door fridge (~20 cu ft, 2022) used as the real-world test bench. Powered at 115 V / 60 Hz, 6.5 A full load — a standard outlet the camera boards share, which drives the low-power, USB-fed design of the electronics.
• Parametric 3D design (OpenSCAD): the camera enclosure is written as code rather than drawn, so every dimension — board size, lens position, shelf thickness, print tolerance — is a variable at the top of the file. Measure the actual hardware, change a number, re-render: the part adapts instead of being redrawn.
• 3D printing of the fridge mounts: parts printed in PLA on a Bambu Lab X1C, designed around the constraints of the process — a spring clip whose opening is deliberately narrower than the shelf so it grips, a fillet at the highest-stress joint so the clip does not snap off, a chamfer around the lens so the wall does not crop the field of view, and vents because a sealed box in a fridge invites condensation on the lens.

• Embedded programming: firmware in C/C++ via the Arduino environment (setup/loop cycle, non-blocking timing with millis()).
• Network communication: WiFi, multipart HTTPS requests to send photos, NTP time synchronization.
• Embedded libraries: OneWire and DallasTemperature (probe), esp_camera camera driver.
• PHP backend: image reception, logging, remote configuration, data API.
• Person detection (Python / OpenCV): local Flask application that analyzes each photo with DNN neural networks (res10 model for faces, MobileNet-SSD for bodies). Any photo where a person appears is rejected — privacy protection, free and instant.
• Artificial intelligence: Claude (vision) API to describe the refrigerator content (food identification).
• Smart image filtering: brightness detection (too dark or overexposed images ignored) and perceptual hash comparison to avoid re-analyzing identical images — reducing API costs.
• Temperature measurement: history, interactive charts (Canvas) with average, CSV/JSON export.
• Web front-end: HTML/CSS/JavaScript, responsive interface (mobile and desktop), touch navigation, smooth scrolling.
• Unique identifiers: each camera identifies itself by its MAC address, for large-scale deployment.
• OTA updates (Over-The-Air): cameras update their firmware wirelessly, with global or per-camera targeting and automatic version detection from the file name.
• Bilingual interface (EN/FR): centralized translation engine, automatic language detection (browser, cookie), and a language selector on every page.
• Centralized event log: cameras and server write to a shared transaction log (uploads, temperature, WiFi quality, OTA events), viewable and filterable on the web.
• Contact form with double opt-in: email confirmation before delivery, IP geolocation for monitoring, and domain authentication (SPF/DKIM/DMARC) for deliverability.
• Remote configuration: photo interval, camera activation, analysis/upload pause, and per-camera image settings, all adjustable from the web without reflashing.
• Server protection & network-load control: spaced-out OTA checks, batched uploads, and slowed retries, with locked minimum values to avoid triggering the host firewall (which blocks IPs generating too many requests).
• Authenticated transactional emails: confirmation and monitoring messages sent via authenticated SMTP (PHPMailer), with styled HTML templates, Geoapify static map images, identity comparison, and a daily summary report by cron.
• Credential encryption: sensitive credentials are stored encrypted (AES-256) rather than in plain text.
• Fleet management & data export: OTA inventory tracking each camera's real version, last update and local IP, a pausable global rollout, and CSV exports (Excel) of the log and the inventory.
• IP geolocation: visitor IP resolved to city, country and ISP via ip-api.com, an interactive OpenStreetMap link, and a Geoapify static map image embedded in the monitoring emails.
• Wireless WiFi provisioning over Bluetooth: no network is hardcoded in the firmware. A dedicated master board receives the WiFi from the official ESP BLE Provisioning mobile app (security via Proof-of-Possession), then closes Bluetooth to free memory.
• Master/slave architecture with ESP-NOW sharing: the master broadcasts the WiFi (encrypted password) to all the fridge's cameras via ESP-NOW, so a single setup configures every camera. Splitting Bluetooth (master) and camera (slaves) keeps each board within its memory budget.
• Per-fridge QR code: each master derives a unique Bluetooth identity from its MAC address, encoded in a scannable QR code stuck on the fridge, for one-tap provisioning in the app without typing any code.
• Robust reconnection: on a temporary WiFi outage the saved network is kept and the camera auto-reconnects, instead of erasing its configuration; the master appears as a connected device in the OTA inventory (flagged as master, never updated).
• Source control on GitHub: the whole project (ESP32 firmware, PHP pages, 3D models) lives in a Git repository, with a full history of every change, the ability to roll back a bad version, and branches to develop a feature without breaking what already runs.
• Continuous integration with GitHub Actions: every push automatically runs checks on the code (PHP syntax, firmware compilation), so a mistake is caught before it reaches the live site or a camera, instead of being discovered in production.
• Task tracking in Jira: features, bugs and improvements are logged as tickets, prioritized and followed through to completion — the same method used on professional software teams, applied here to a personal project.
• Documentation in Confluence: architecture decisions, wiring diagrams, network protocols and setup procedures are written down rather than kept in memory, so the project stays understandable months later and could be handed over to someone else.
• "My fridge" assistant: a page that reads every description Claude has written over a chosen period (week, two weeks, month, or everything) and turns that free text into a structured grocery list — only products you could actually buy in a store, with opaque containers and unidentifiable leftovers filtered out. Each item shows how often it appeared, which surfaces the household staples.
• Recipe suggestions from real sites: Claude searches the web for existing recipes that use what is actually in the fridge, filtered by the household's tastes and constraints. Each suggestion links to the source (Ricardo and other Quebec sites) rather than reproducing the recipe, and the prompt explicitly rewards originality over the obvious dish.
• Self-contained descriptions: each analysis now stores its own timestamp and camera alongside the text, instead of depending on the photo filename. Photos are capped and deleted as new ones arrive, but the descriptions survive — they become the lasting record of what passed through the fridge. Both the old and new storage formats are read transparently, so years of history stay usable.
• Non-blocking page loading: the monitoring page used to wait for every pending Claude analysis before showing anything, which meant several seconds of blank screen. It now renders immediately from cached descriptions and fetches the missing ones in the background, one at a time, filling them in as they arrive — including live inside an already-open photo.
• Data kept outside the web root: photos, JSON files, firmware binaries and API keys were moved out of the public folder. Nothing about the installation — how many fridges, which camera MAC addresses, how many clients — can be inferred from scanning the site. Images are served through a script that validates every request and resolves the path to make sure it cannot escape the photo folder.
• Nearby grocery stores and current deals: the shopping list connects to the real world through browser geolocation (or a postal code) to find supermarkets within five kilometres, and a web search for deals on those exact products, showing the regular price struck through next to the sale price with the computed discount.

• Stateless microservices: split the monolithic PHP backend into independent services (ingestion, analysis, notifications) that can each scale horizontally behind a load balancer.
• Message queue: cameras publish photos and events to a queue (Kafka / AWS SQS) instead of hitting the server directly — smoothing traffic spikes and decoupling capture from processing.
• Auto-scaling & containers: services packaged in Docker and orchestrated with Kubernetes (EKS), automatically adding capacity as the number of cameras grows.
• Content delivery network (CDN): images and static pages served from edge locations close to users, reducing latency and origin-server load.
• Staged OTA rollouts: firmware pushed progressively (canary → percentage → full fleet) with automatic rollback, so a bad update never reaches all 500,000 cameras at once.

• Distributed NoSQL database: camera metadata, versions and events stored in DynamoDB or Cassandra, which partition data across many nodes and scale to billions of records.
• Object storage: photos stored in Amazon S3 (virtually unlimited) with lifecycle policies to archive or delete old images automatically.
• Time-series database: temperature and sensor readings stored in a specialized engine (TimescaleDB / Amazon Timestream) built for high-volume sensor data.
• Sharding & read replicas: data partitioned by region/account and replicated so reads scale independently of writes, with no single bottleneck.

• Managed ingestion: AWS IoT Core to connect and authenticate millions of devices over MQTT, with per-device certificates.
• Serverless processing: AWS Lambda functions triggered per event (analyze a photo, send an email) — pay only for what runs, scales to zero and to millions automatically.
• Managed AI: Amazon Rekognition for person detection and Amazon Bedrock (Claude) for content description, removing the need to run the local Python/Flask app.
• Infrastructure as code: the whole platform defined in Terraform / CloudFormation, reproducible across regions and environments in minutes.
• Observability: centralized metrics, logs and alerts (CloudWatch, Prometheus, Grafana) to monitor the health of the whole fleet in real time.

• Per-device identity: each camera holds a unique X.509 certificate; a compromised device can be revoked individually without affecting the fleet.
• End-to-end encryption: TLS in transit and encryption at rest (S3, database) with keys managed by AWS KMS — no secret ever stored in plain text.
• Signed firmware: OTA binaries cryptographically signed and verified by the device before installation, preventing malicious or tampered updates.
• Zero-trust access & IAM: fine-grained roles and least-privilege permissions for every service and user, with multi-factor authentication for administrators.
• DDoS protection & rate limiting: AWS WAF and Shield at the edge to absorb attacks and throttle abusive traffic before it reaches the services.
• Privacy & compliance by design: on-device person filtering kept, data residency per region, audit trails, and alignment with standards such as GDPR and SOC 2.

Beyond the cloud, the cameras themselves would be hardened in three complementary layers. Each layer defends against a different threat, and the hardware layer protects the secrets used by the other two — so an attacker would have to defeat all three at once.

• Layer 1 — Unique per-camera token (application level): instead of one shared secret, each camera derives its own token from its MAC address combined with a server-side secret (HMAC-SHA256). The server recomputes and verifies it on every request. A stolen token only works for a single camera, and any camera can be revoked individually. Cost is negligible (~2 KB RAM, SHA-256 is built into the ESP32) and it is fully reversible, making it the safe first step.
• Layer 2 — Mutual TLS with signed client certificates (connection level): each camera would carry a unique X.509 certificate signed by a private certificate authority (PKI). The server would reject any connection that does not present a certificate signed by that authority, so a foreign or counterfeit ESP32 simply cannot establish a session. This is the classic « signed certificate » solution; its main cost is RAM during the TLS handshake (~40–50 KB), which the WROVER absorbs thanks to its 8 MB PSRAM. The real effort is operational: running a PKI, issuing one certificate per camera, and handling revocation.
• Layer 3 — Secure Boot + Flash Encryption (hardware level): the ESP32's hardware eFuses (one-time-programmable bits, physically irreversible) would store keys that (a) make the chip run only firmware signed with the manufacturer's private key, refusing any tampered or infected build at boot, and (b) encrypt the flash so the firmware — and the secrets from layers 1 and 2 — cannot be read even by desoldering the chip. This is what truly answers « don't let an infected device in », because it guarantees only the genuine firmware ever executes and its secrets stay inside the silicon.
• Why all three together: layer 3 protects layers 1 and 2 — without it the per-camera token and the certificate's private key could be extracted from the firmware; with it they become inextractable. An intruder would need a firmware signed by us, a certificate signed by our authority, and a valid token, all at once. They are deployed in order (token first, mTLS next, hardware sealing last) because Secure Boot is irreversible and is applied only once the firmware is final — like pouring concrete for the foundations once the plans are frozen.