+---
+title: Ourosboros Flash Reader
+author: Patrick Schönberger
+date: 16.07.2025
+transition: none
+revealjs-url: https://cdn.jsdelivr.net/npm/reveal.js@5.2.1
+slideNumber: true
+---
+
+## Structure
+<!-- structure: start with a problem and reproduce the work leading to the working solution -->
+
+
+- Start with the problem
+- Work our way to the solution step by step
+
+
+
+<!-- access smart home hw on crime scenes -->
+<!-- many shelly devices are based on esp32/esp8266 -->
+<!-- they dont contain any usable data but contain identifiable user data -->
+<!-- this can be used to inquire about the user account -->
+<!-- so we need to extract the content of the esp's flash memory -->
+<!-- we then also need to extract filesystems from the memory -->
+<!-- do it read-only, verifiably -->
+<!-- existing solutions (esptool, mos) can also write and erase memory -->
+<!-- they are also complex, making understanding and changing the code time consuming -->
+<!-- what about writing a custom extraction tool? -->
+
+## The Problem
+
+- How to access data on Shelly devices?
+- Most Shelly devices are based on ESPs
+- They don't contain usable data like logs
+- They do contain user-identifiable data
+
+##
+
+- Extract the flash memory and read the data
+- Do it read-only and document the process
+- Existing solutions (esptool, mos) can modify memory
+- They are very complex, and thus hard to modify
+- What about a custom tool?
+
+
+
+<!-- what is the bare minimum needed to talk to the esp? -->
+<!-- - two modes: bootmode and runmode -->
+<!-- - decided by GPIO0 at start -->
+<!-- - the esp as well as the shelly devices expose uart pins (tx/rx) -->
+<!-- - in runmode they output logging information -->
+<!-- - in bootmode they listen to a custom serial protocol -->
+<!-- so we need a serial connection and the ability to enter boot mode! -->
+<!-- what can the serial protocol do? -->
+<!-- - sync -->
+<!-- - write ram/flash/registers -->
+<!-- - configuration etc. -->
+<!-- - on the esp32 it can read flash, but not on esp8266, esp32c3, esp32c6 -->
+
+## Talking to the ESP
+
+- What is the bare minimum needed to talk to an ESP?
+- Two modes: boot and run
+- Serial pins are exposed on some Shelly devices and accessible on others
+- In boot mode they can be used to communicate using a custom protocol
+<!-- * So we need a serial connection and the ability to enter boot mode -->
+
+##
+
+- What can we do with the protocol?
+ - Synchronize
+ - Write RAM, flash and registers
+ - Configure memory etc.
+ - On the ESP32 it can read flash, but not on the ESP8266, ESP32C3 or ESP32C6
+- How do other tools read flash?
+
+
+
+<!-- how do other tools read flash? -->
+<!-- - we cant directly read flash, but we can write ram -->
+<!-- - write a program, load it into ram, run it and then talk to it instead -->
+<!-- - flash loader/stub -->
+<!-- - esptool uses two different variants, c based and rust based -->
+<!-- - the c based one is older and getting replaced, but it is also dramatically simpler and also supports the esp8266 -->
+<!-- - so we use the c based one and customize it (remove write and erase flash commands) -->
+<!-- technically this means we do have write access until the flash loader is activated -->
+<!-- the extraction tool is also small and runs a fixed number of commands -->
+<!-- -> as sure as we can be -->
+
+## Reading Flash Memory
+
+- We cannot directly read flash, but we can write RAM
+- Write a program for the ESP, load it into RAM and run
+- This is called the flash loader or stub loader
+- Esptool uses two versions:
+ - Legacy C version
+ - New Rust version
+
+##
+
+- The C version is getting replaced but it is much simpler and it supports ESP8266
+- We customize this version by removing write and erase commands
+- This way we technically have write access until the flash loader is activated
+- The extraction tool is small and contains no write or erase commands
+
+
+
+<!-- how does the serial protocol work? -->
+<!-- - data is encoded using SLIP frames -->
+<!-- - the host sends a request and the target (esp) sends a response -->
+<!-- - steps to read flash: -->
+<!-- - sync -->
+<!-- - identify chip -->
+<!-- - read mac -->
+<!-- - (change baud) -->
+<!-- - upload stub -->
+<!-- - read flash -->
+
+## Serial Protocol
+
+- SLIP frames...
+
+
+
+<!-- modifying the flash loader -->
+<!-- - stub consists of 6 .c files: -->
+<!-- - miniz.c // compression -->
+<!-- - slip.c // slip -->
+<!-- - stub_commands.c // handle commands -->
+<!-- - stub_flasher.c // main program -->
+<!-- - stub_io.c // serial communication -->
+<!-- - stub_write_flash.c // write flash -->
+<!-- - so we remove stub_write_flash.c and modify stub_commands.c -->
+<!-- - additionally simplify the makefile -->
+<!-- compiling and uploading the flash loader -->
+<!-- - download toolchains -->
+<!-- - compile the stub using specific toolchains -->
+<!-- - this gives us an elf file -->
+<!-- - use a python script to extract the .text and .data sections from the elf -->
+<!-- - generate a header file and write the raw bytes to a `const unsigned char[]` -->
+<!-- - this header gets compiled with the extraction tool (host) -->
+<!-- - at runtime, after the chip is identified, upload .text and .data using MEM_ ram commands -->
+<!-- - addresses for the sections and for the entry point are in elf file and get written to header alongside the elf sections -->
+<!-- and how do we make the esp enter bootmode? -->
+<!-- - wire two gpio pins to RST and GPIO0 -->
+<!-- - pull both low -->
+<!-- - RST low turns the esp off -->
+<!-- - GPIO0 has to be low when it is turned back on -->
+<!-- - pull RST high to turn it on -->
+<!-- - pull GPIO0 high after the esp has started -->
+<!-- overview: -->
+<!-- - bootmode/serial -->
+<!-- - serial protocol -->
+<!-- - flash loader -->
+<!-- differences between esp versions -->
+<!-- - identification: -->
+<!-- - ESP32-C3 and later use GET_SECURITY_INFO which contains a chip_id -->
+<!-- - previous models have a register with a magic value identifying the chip -->
+<!-- - mac address: -->
+<!-- - different registers -->
+<!-- - esp8266 mac has to be calculated -->
+<!-- - different flash loader versions -->
+<!-- - esp8266 has no data section -->
+<!-- different hosts: -->
+<!-- - linux (usb) -->
+<!-- - rpi (gpio) -->
+<!-- - esp (gpio) -->
+<!-- extracting the file system -->
+<!-- - esp8266 -->
+<!-- - esp32 -->
+<!-- interesting files -->
+<!-- - wifi credentials -->
+<!-- - certificates -->
+<!-- - jwt token -->
+
+
+## Demo
+
+
+## cloc
+
+##
+
+```sh
+$ cloc esp-flasher-stub/
+-----------------------------------------------------------
+Language files blank comment code
+-----------------------------------------------------------
+Rust 12 327 78 1863
+Logos 14 32 0 249
+YAML 3 34 12 214
+Markdown 1 34 0 89
+TOML 4 8 2 61
+-----------------------------------------------------------
+SUM: 34 435 92 2476
+-----------------------------------------------------------
+```
+
+##
+
+```sh
+$ cloc esp-hal
+-----------------------------------------------------------
+Language files blank comment code
+-----------------------------------------------------------
+Rust 492 18739 26120 115809
+Linker Script 51 499 1404 11315
+Markdown 66 2014 12 5262
+TOML 43 505 412 4481
+Logos 44 229 15 1105
+YAML 13 151 67 1054
+Jinja Template 3 52 0 255
+JSON 2 0 0 48
+CSV 3 0 0 21
+SVG 1 0 0 4
+-----------------------------------------------------------
+SUM: 718 22189 28030 139354
+-----------------------------------------------------------
+```
+
+##
+
+```sh
+$ cloc esptool-legacy-flasher-stub/
+-----------------------------------------------------------
+Language files blank comment code
+-----------------------------------------------------------
+Linker Script 32 787 1188 18751
+C 6 1284 636 7689
+C/C++ Header 8 463 557 1687
+make 1 36 33 130
+YAML 4 19 0 114
+Python 2 25 21 80
+Markdown 3 58 0 76
+Bourne Shell 2 9 8 23
+TOML 1 1 0 20
+Jinja Template 1 4 2 14
+-----------------------------------------------------------
+SUM: 60 2686 2445 28584
+-----------------------------------------------------------
+```
projects / ouroboros-slides / commitdiff