TensorFlow Lite Micro on ESP32: Quantized Edge AI, Step by Step

Last Updated: October 2026

A vibration sensor bolted to a pump produces thousands of samples per second. Streaming all of it to the cloud to ask one question, “does this sound normal?”, burns radio energy, bandwidth, and money, and it fails the moment the plant Wi-Fi hiccups. The alternative is to ask the question on the sensor node itself, with a neural network small enough to live in a few kilobytes of RAM. That is the job of TensorFlow Lite Micro, the C++ inference runtime that has quietly become the default way to run neural networks on ESP32-class chips.

It matters now because the ecosystem moved under everyone’s feet. Google rebranded TensorFlow Lite as LiteRT, the microcontroller runtime is now documented as “LiteRT for Microcontrollers”, Espressif ships a maintained esp-tflite-micro component with hand-tuned kernels, and rivals such as ExecuTorch and Edge Impulse are credible choices. Tutorials written in 2021 will still compile, but they will mislead you about arena sizing, quantization, and which kernels actually run fast.

You will leave with a complete, runnable pipeline: design a tiny anomaly-detection autoencoder, quantize it to int8, embed it, size the tensor arena from measurement rather than guesswork, and run it on an ESP32-S3 under ESP-IDF with ESP-NN acceleration.

What this covers: naming and status in 2026, the runtime architecture, model design and quantization math, working Python and C++ code, measured speedups from Espressif, failure modes, and when to pick a different runtime.

Context and Background

TensorFlow Lite Micro (TFLM) began inside Google as an experiment to run TensorFlow models on devices with tens of kilobytes of memory. Its design rules are unusual for a machine-learning runtime: no operating system dependency, no standard library dependency, and no dynamic memory allocation. Google’s documentation says the core runtime “just fits in 16 KB on an Arm Cortex M3”. Everything else, including the model, lives in flash and a single statically sized byte array called the tensor arena.

Naming is the first trap in 2026. Google’s AI Edge documentation now titles the project LiteRT for Microcontrollers, while the code, the GitHub repository (tensorflow/tflite-micro), the C++ namespace (tflite::), and Espressif’s component (esp-tflite-micro) all keep the older name. Industry commentary, such as the Beningo Embedded Group technology atlas, dates the rebrand to 2024 and notes it caused confusion through 2026. For practical purposes the two names refer to the same runtime. When you search for error messages, search for both.

ESP32 is a natural host. The family spans the original dual-core Xtensa ESP32, the ESP32-S3 with vector instructions, the RISC-V ESP32-C3, and the newer ESP32-P4. Google lists ESP32 among the ports it supports, and Espressif maintains the integration, tracking upstream through a sync script and publishing it on the ESP Component Registry. At the time of writing the registry showed version 1.4.1 as the latest release of espressif/esp-tflite-micro, with ESP-IDF releases from v5.1 through v6.0 listed as supported; check the registry before pinning, because this moves quickly.

Why not simply use the cloud, or a bigger edge computer? Because the economics differ by orders of magnitude. A fleet of thousands of battery or loop-powered sensor nodes cannot each run a Linux board. If you are weighing larger accelerators, our comparison of Hailo-10H against Jetson Orin Nano for edge vision covers the other end of the spectrum, and our note on int4, int8, and fp8 quantization for edge NPUs explains why int8 remains the lingua franca for microcontrollers. For authoritative background, read Google’s LiteRT for Microcontrollers overview.

The thesis of this guide is simple and contrarian to most tutorials: on an ESP32, the inference call is rarely your bottleneck or your risk. The risks are the quantization calibration set, the arena you never measured, kernels that silently fall back to slow reference code, and the preprocessing you run around the model. Get those four right and the network itself is almost an afterthought.

How TensorFlow Lite Micro Runs on ESP32: The Reference Architecture

TensorFlow Lite Micro on ESP32 runs a pre-quantized flatbuffer model straight from flash. A static op resolver maps each operator to a kernel, ESP-NN replaces the slow reference kernels with optimized ones on supported chips, and every intermediate tensor lives in one fixed arena in RAM. Nothing is allocated after initialization, so latency and memory use are deterministic.

Figure 1: TensorFlow Lite Micro runtime stack on ESP32. The model stays in flash, kernels come from ESP-NN where available, and all working memory is one arena.

The diagram shows five cooperating pieces. The model flatbuffer sits in flash and is read in place. The MicroMutableOpResolver holds only the operators your model needs. The MicroInterpreter owns planning and execution. Kernels do the arithmetic, either the portable reference versions or ESP-NN’s optimized ones. And the tensor arena holds activations and scratch buffers. The application sits on top, feeding input tensors from a sensor pipeline and reading the output tensor.

The flatbuffer is read in place, not loaded

A TensorFlow Lite model is a FlatBuffers file. FlatBuffers is a serialization format designed so data can be accessed without a parsing step: the file is already laid out the way the reader wants it, and fields are reached through offsets. On a microcontroller this is the key property. A 150 KB model does not need 150 KB of RAM to deserialize into, because the interpreter walks the const array in flash and only copies what it must.

Two consequences follow. First, weights cost flash, not RAM, which is why a model of a few hundred kilobytes is feasible on a chip with 512 KB of SRAM. Second, the buffer should be properly aligned. Alignment matters: the vector load paths on the ESP32-S3 prefer aligned data, and misaligned access is a classic source of slow paths or faults. If you embed with the ESP-IDF EMBED_FILES mechanism, verify alignment yourself; if you generate a C array with xxd -i, add alignas(16) as the code later in this guide does.

The op resolver is your binary-size lever

The interpreter does not know about operators until you register them. MicroMutableOpResolver<N> takes a compile-time capacity, and you call AddFullyConnected(), AddConv2D(), AddSoftmax() and so on. This design means unused kernels are never linked, which matters when the whole firmware must fit in a few megabytes of flash alongside Wi-Fi, TLS, and OTA partitions.

The failure mode is also instructive. If your converted model contains an operator you did not register, AllocateTensors() fails at startup with a message naming the missing operator. That is a gift: you find out on boot rather than in the field. Count the operators in the model with a script (shown later) and register exactly that set, plus one or two spare slots only if you intend to ship model updates over the air.

Static memory planning and the arena

When you call AllocateTensors(), the interpreter’s allocator reads the graph, computes each tensor’s lifetime, and packs the tensors into the arena so that tensors with disjoint lifetimes share bytes. This is a form of register allocation for tensors, and it is why peak memory is the sum of simultaneously live tensors, not the sum of all tensors. A deep network with small feature maps can need far less arena than its parameter count suggests, while a shallow network with a large early feature map can need far more.

The arena also holds persistent data such as per-operator quantization parameters and kernel scratch buffers, and some kernels request temporary buffers at prepare time. This is why a rule such as “arena equals peak activation times 1.3” from older tutorials is only a guess. Measure it. The upstream tree includes a recording interpreter variant that reports how many bytes were actually used; the next sections show how to take that measurement and then add a margin.

Where ESP-NN fits

ESP-NN is Espressif’s library of optimized neural-network kernels. According to its repository, it covers elementwise add and multiply, convolution, depthwise convolution, max and average pooling, fully connected, softmax, and ReLU6-style activation, in int8 variants. On the ESP32-S3 the assembly versions use the chip’s vector instructions, on the ESP32-P4 they use its PIE and QACC SIMD extensions, and on the original ESP32 and the ESP32-C3 the optimizations are generic rather than SIMD. The esp-tflite-micro component compiles ESP-NN into every example, and you can disable it by removing the -DESP_NN compile flag.

How much does it matter? Espressif publishes invoke() times for the TFLM person-detection example in the component README, reproduced below. The numbers measure only the invoke() call with internal memory, so treat them as a model-specific reference, not a universal multiplier.

Target	CPU clock	Without ESP-NN	With ESP-NN	Speedup
ESP32-P4	360 MHz	1395 ms	73 ms	about 19x
ESP32-S3	240 MHz	2300 ms	54 ms	about 43x
ESP32	240 MHz	4084 ms	380 ms	about 11x
ESP32-C3	160 MHz	3355 ms	426 ms	about 8x

Source: Espressif esp-tflite-micro README, person detection example. The speedup column is our arithmetic from those figures.

Two things stand out. The S3 with ESP-NN is not merely faster than the ESP32, it is about seven times faster at the same clock (380 ms against 54 ms), purely because of vector instructions. And without ESP-NN, a vision model on any of these chips is unusable for real time. Silent fallback to reference kernels is therefore the single costliest misconfiguration, and the Trade-offs section shows how to detect it.

Choosing the chip for your model class

A pragmatic reading of the table gives a selection rule. Vibration, audio keyword spotting, and tiny-sensor classification, with models in the tens of thousands of multiply-accumulates per inference, run comfortably on any ESP32 variant, including the C3. Small vision models, such as the person-detection example, need the S3 or P4 to hit a frame rate above a few hertz. Anything with sustained high-resolution video or transformer-class workloads belongs on a bigger accelerator; our overview of TI’s MSPM0G5187 and TinyEngine NPU microcontroller shows how dedicated NPUs are entering the microcontroller class.

Designing, Training, and Quantizing the Model

The worked example for the rest of this guide is a vibration anomaly detector, a common industrial IoT use case. An accelerometer samples at 2,560 Hz. A 128-sample window spans 50 ms. A 128-point FFT yields 64 usable magnitude bins, 20 Hz apart, reaching a 1,280 Hz Nyquist limit. A small dense autoencoder learns to reconstruct healthy spectra; when reconstruction error rises, the machine is behaving differently from how it did during training.

Figure 2: End-to-end pipeline from accelerometer window to FFT features, int8 autoencoder inference, and an anomaly decision with MQTT reporting.

Why an autoencoder rather than a classifier? Failure data is scarce and diverse. You can record hours of healthy operation but you rarely have labeled examples of every bearing fault. An autoencoder needs only healthy data, and the reconstruction error against a threshold is a natural one-class detector. For a deeper treatment of learned models on sensor streams, see our piece on time-series foundation models for industrial sensors; the point here is that even a 4,000-parameter model is meaningful at the edge.

Size the model before you train it

Count parameters by hand. The architecture is 64 inputs, then dense layers of 32, 8, 32, and 64 units.

Layer	Weights	Biases	Int8 weight bytes
Dense 64 to 32	2,048	32	2,048
Dense 32 to 8	256	8	256
Dense 8 to 32	256	32	256
Dense 32 to 64	2,048	64	2,048
Total	4,608	136	4,608

Total parameters are 4,744. In int8, weights occupy 4,608 bytes; biases are stored as int32 in TFLite’s quantization scheme, so 136 x 4 = 544 bytes. That is about 5.2 KB of constants before flatbuffer metadata, which typically adds a few kilobytes (the exact overhead depends on the converter version; measure your file). Multiply-accumulate count per inference equals the weight count, 4,608, which is far below the convolutional models in the vision examples (we have not measured their exact MAC counts here, so compare on your own hardware). This is the regime where inference takes microseconds-to-low-milliseconds even on reference kernels, and where preprocessing dominates.

Training code

The following script trains on healthy spectra. It assumes you have a NumPy array healthy.npy of shape (N, 64) containing log-magnitude FFT bins, already normalized to roughly zero mean and unit variance per bin. Version pins are deliberately omitted; use a TensorFlow release that supports your Keras version and verify the converter behavior shown below on your install.

import numpy as np
import tensorflow as tf

x = np.load("healthy.npy").astype(np.float32)          # shape (N, 64)
split = int(len(x) * 0.9)
x_train, x_val = x[:split], x[split:]

inp = tf.keras.Input(shape=(64,))
h = tf.keras.layers.Dense(32, activation="relu")(inp)
h = tf.keras.layers.Dense(8, activation="relu")(h)
h = tf.keras.layers.Dense(32, activation="relu")(h)
out = tf.keras.layers.Dense(64)(h)                      # linear output
model = tf.keras.Model(inp, out)

model.compile(optimizer=tf.keras.optimizers.Adam(1e-3), loss="mse")
model.fit(x_train, x_train, validation_data=(x_val, x_val),
          epochs=60, batch_size=128, verbose=2)

# Reconstruction error distribution on held-out healthy data
rec = model.predict(x_val, verbose=0)
err = np.mean((rec - x_val) ** 2, axis=1)
print("healthy MSE mean=%.4f p99=%.4f p999=%.4f"
      % (err.mean(), np.percentile(err, 99), np.percentile(err, 99.9)))
model.save("ae_float.keras")
np.save("val_err_float.npy", err)

Notice the linear output layer. A ReLU on the output would clip negative normalized values and corrupt the reconstruction. Also notice that we store the healthy error percentiles now; you will compare them to the int8 model’s error later to confirm quantization did not shift your threshold.

What int8 quantization actually does

TFLite uses affine quantization for activations: a real value r is represented as r = s * (q - z), where q is the int8 value in [-128, 127], s is a positive float scale, and z is an integer zero point. Weights use symmetric quantization with zero point 0, usually per output channel for convolutions, which preserves accuracy better than a single scale per tensor.

A worked example shows why calibration matters. Suppose calibration observes an activation range of [-1.0, 3.0]. The scale is s = (3.0 - (-1.0)) / 255 = 0.015686. The zero point is z = round(-min / s) - 128 = round(63.75) - 128 = -64. Check the low end: s * (-128 + 64) = -1.004. Good. Now suppose a single outlier spike in your calibration data stretches the range to [-1.0, 30.0]. The scale becomes 31/255 = 0.1216, nearly eight times coarser, and the useful range of normal values, mostly under 3.0, is now represented with only about 25 of 256 levels. Accuracy collapses not because int8 is inherently bad but because calibration saw an outlier. Quantization quality is a data-curation problem.

Figure 3: Post-training quantization flow. A representative dataset drives calibration; the converter fixes scales and zero points, then emits an int8 flatbuffer plus a C array.

Converting with a representative dataset

Full-integer post-training quantization needs a generator that yields real input samples so the converter can observe activation ranges. Force int8 for both input and output so the device never touches floating point.

import numpy as np
import tensorflow as tf

model = tf.keras.models.load_model("ae_float.keras")
x = np.load("healthy.npy").astype(np.float32)

def representative_data():
    rng = np.random.default_rng(0)
    idx = rng.choice(len(x), size=min(500, len(x)), replace=False)
    for i in idx:
        yield [x[i:i+1]]

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_data
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
tflite_model = converter.convert()

open("ae_int8.tflite", "wb").write(tflite_model)
print("model bytes:", len(tflite_model))

Use a few hundred representative samples drawn from all operating conditions you want to count as healthy: different speeds, loads, warm and cold starts. If calibration only sees one operating point, a legitimate speed change will saturate activations and look like an anomaly. Restricting supported_ops to the int8 builtin set makes the converter fail loudly if any operator cannot be quantized, instead of leaving a float island that the micro runtime cannot run efficiently.

The Google documentation for the converter and for the microcontroller workflow is hosted under Google AI Edge. Because converter APIs have changed across TensorFlow and Keras versions, treat the snippet as a template and run it on your pinned toolchain.

Validating, Embedding, and Running the Model in ESP-IDF

Before touching hardware, prove on the desktop that the int8 model behaves like the float model. This step costs minutes and saves days of on-device debugging.

Validate the quantized model on the host

The script below lists the operators (so you can build the resolver), reads the input and output quantization parameters (you will need them in C++), and compares int8 reconstruction error against the float baseline.

import numpy as np
import tensorflow as tf
from collections import Counter

interp = tf.lite.Interpreter(model_path="ae_int8.tflite")
interp.allocate_tensors()
inp = interp.get_input_details()[0]
out = interp.get_output_details()[0]
print("input  dtype", inp["dtype"], "quant (scale, zp) =", inp["quantization"])
print("output dtype", out["dtype"], "quant (scale, zp) =", out["quantization"])

ops = Counter(d["op_name"] for d in interp._get_ops_details())
print("operators:", dict(ops))

x = np.load("healthy.npy").astype(np.float32)[-2000:]
s_in, z_in = inp["quantization"]
s_out, z_out = out["quantization"]

errs = []
for row in x:
    q = np.clip(np.round(row / s_in + z_in), -128, 127).astype(np.int8)
    interp.set_tensor(inp["index"], q[None, :])
    interp.invoke()
    y = interp.get_tensor(out["index"])[0].astype(np.float32)
    rec = (y - z_out) * s_out
    errs.append(np.mean((rec - row) ** 2))
errs = np.array(errs)
print("int8 healthy MSE mean=%.4f p99=%.4f p999=%.4f"
      % (errs.mean(), np.percentile(errs, 99), np.percentile(errs, 99.9)))

Expect an operator list of FULLY_CONNECTED (and possibly QUANTIZE or DEQUANTIZE if the converter inserted boundary ops, which should not happen with int8 input and output types). Compare the printed int8 percentiles with the float percentiles saved earlier. A small relative shift is normal; a large one means a calibration problem. The private _get_ops_details() helper is not a stable public API, so if it disappears in your TensorFlow version, inspect the model with a flatbuffer viewer such as Netron instead.

Then generate the C array and record the quantization parameters:

xxd -i ae_int8.tflite > model_data.cc
# Edit the generated file so it reads:
#   alignas(16) const unsigned char g_model[] = { ... };
#   const unsigned int g_model_len = ...;

Project layout and component setup

An ESP-IDF project for this firmware has a small footprint:

anomaly_node/
  CMakeLists.txt
  sdkconfig.defaults
  main/
    CMakeLists.txt
    idf_component.yml
    main.cc
    model_data.cc
    features.cc
    features.h

Pull in the Espressif component using the dependency manager. Pin to the version you have tested; the registry showed 1.4.1 as latest when we checked:

# main/idf_component.yml
dependencies:
  espressif/esp-tflite-micro: "^1.4.1"
  idf: ">=5.1"

# main/CMakeLists.txt
idf_component_register(
    SRCS "main.cc" "model_data.cc" "features.cc"
    INCLUDE_DIRS "."
)

# sdkconfig.defaults
CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ_240=y
CONFIG_FREERTOS_HZ=1000
CONFIG_COMPILER_OPTIMIZATION_PERF=y

Then idf.py set-target esp32s3, build, flash, and monitor. Optimization level matters for kernels you compile from source, including the reference paths, so build with the performance setting rather than the size setting when benchmarking. The ESP-NN option is under menuconfig: its NN_OPTIMIZATIONS entry offers “Optimized versions” (the default) and “ANSI C” (the reference kernels, useful for debugging numerics).

Feature extraction on the device

The model expects the same 64 features the training pipeline produced, so the firmware must reproduce preprocessing exactly. This is the most common source of “works in Python, garbage on device”. Document and unit-test it. A compact version using Espressif’s DSP library would call a real FFT and compute log magnitudes; the sketch below shows the contract, with the FFT call left to your chosen implementation (for example esp-dsp, whose API names you should confirm against its current documentation).

// features.h
#pragma once
#include <stdint.h>
constexpr int kWindow = 128;
constexpr int kBins   = 64;
// Fills 'out' with kBins log-magnitude values, normalized exactly as in training.
void compute_features(const int16_t* samples, float* out);

// features.cc  (contract and normalization; FFT call is implementation-specific)
#include "features.h"
#include <math.h>

// Per-bin mean and std exported from the training script. Placeholder values.
extern const float kMean[kBins];
extern const float kStd[kBins];

void fft_magnitude(const int16_t* samples, float* mag);  // 64 bins, your FFT here

void compute_features(const int16_t* samples, float* out) {
  float mag[kBins];
  fft_magnitude(samples, mag);
  for (int i = 0; i < kBins; ++i) {
    float v = log1pf(mag[i]);
    out[i] = (v - kMean[i]) / kStd[i];
  }
}

Export kMean and kStd from the exact arrays used to train, and ship them as constants. Use the same window function, the same scaling of raw ADC counts, and the same log1p. A mismatch of even a constant factor shifts every bin and breaks the autoencoder, which is trained on a precisely defined distribution.

The inference code

The main application builds the resolver, creates the interpreter in a hand-aligned arena, runs inference, and decodes the int8 output to a real-valued error.

// main.cc
#include <stdio.h>
#include <math.h>
#include "esp_heap_caps.h"
#include "esp_log.h"
#include "esp_timer.h"
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"

#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/schema/schema_generated.h"

#include "features.h"

extern const unsigned char g_model[];

static const char* TAG = "anomaly";
constexpr size_t kArenaSize = 24 * 1024;     // first guess; replaced after measuring
constexpr float  kThreshold = 0.35f;         // from healthy p99.9 on the host. Placeholder.

static tflite::MicroInterpreter* g_interp = nullptr;
static TfLiteTensor* g_in = nullptr;
static TfLiteTensor* g_out = nullptr;

static bool model_init() {
  const tflite::Model* model = tflite::GetModel(g_model);
  if (model->version() != TFLITE_SCHEMA_VERSION) {
    ESP_LOGE(TAG, "schema %lu != %d", (unsigned long)model->version(), TFLITE_SCHEMA_VERSION);
    return false;
  }
  uint8_t* arena = (uint8_t*)heap_caps_aligned_alloc(
      16, kArenaSize, MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT);
  if (!arena) { ESP_LOGE(TAG, "arena alloc failed"); return false; }

  static tflite::MicroMutableOpResolver<1> resolver;
  if (resolver.AddFullyConnected() != kTfLiteOk) return false;

  static tflite::MicroInterpreter interp(model, resolver, arena, kArenaSize);
  g_interp = &interp;
  if (g_interp->AllocateTensors() != kTfLiteOk) {
    ESP_LOGE(TAG, "AllocateTensors failed: arena too small or missing op");
    return false;
  }
  g_in = g_interp->input(0);
  g_out = g_interp->output(0);
  ESP_LOGI(TAG, "arena used: %u of %u bytes",
           (unsigned)g_interp->arena_used_bytes(), (unsigned)kArenaSize);
  return true;
}

static float infer_error(const float* feat) {
  const float s_in = g_in->params.scale;
  const int   z_in = g_in->params.zero_point;
  for (int i = 0; i < kBins; ++i) {
    int q = (int)lroundf(feat[i] / s_in) + z_in;
    g_in->data.int8[i] = (int8_t)(q < -128 ? -128 : (q > 127 ? 127 : q));
  }
  int64_t t0 = esp_timer_get_time();
  if (g_interp->Invoke() != kTfLiteOk) return -1.0f;
  int64_t dt = esp_timer_get_time() - t0;

  const float s_out = g_out->params.scale;
  const int   z_out = g_out->params.zero_point;
  float acc = 0.f;
  for (int i = 0; i < kBins; ++i) {
    float rec = (g_out->data.int8[i] - z_out) * s_out;
    float d = rec - feat[i];
    acc += d * d;
  }
  ESP_LOGD(TAG, "invoke %lld us", (long long)dt);
  return acc / kBins;
}

extern "C" void app_main(void) {
  if (!model_init()) { vTaskDelay(portMAX_DELAY); }
  static int16_t window[kWindow];
  static float feat[kBins];
  while (true) {
    // fill 'window' from your accelerometer driver or DMA buffer here
    compute_features(window, feat);
    float err = infer_error(feat);
    if (err > kThreshold) ESP_LOGW(TAG, "ANOMALY err=%.3f", err);
    vTaskDelay(pdMS_TO_TICKS(50));
  }
}

Several details deserve attention. The arena_used_bytes() call is available on the interpreter in current upstream code and gives the planned memory use after AllocateTensors(); if your vendored version lacks it, use the recording interpreter. The static resolver and interpreter avoid heap use for the objects themselves. The input conversion divides by the scale and adds the zero point, the exact inverse of the dequantization used in the host validation script. The threshold constant is a placeholder: derive yours from the healthy error percentile you printed, and re-derive it whenever you retrain.

Older tutorials pass a tflite::ErrorReporter* into the interpreter constructor and include micro_error_reporter.h. Google’s current Get Started page still shows that older pattern, while newer upstream releases moved to a leaner constructor and MicroPrintf. Which signature your vendored copy accepts depends on the component version, so match the example code in the esp-tflite-micro release you pin rather than copying blindly from any single page.

Sizing the Arena and Measuring Real Performance

The arena is where most deployments quietly go wrong, so treat it as an engineering quantity with a measurement procedure, not a magic number.

A four-step arena procedure

Start generous. Allocate a large arena on the first boot, for example 64 KB for a small model, so AllocateTensors() succeeds.
Read the planned usage. Log arena_used_bytes() after allocation, as the code above does. This is the allocator’s high-water mark for the plan, including persistent and scratch buffers.
Shrink with margin. Set the arena to the measured value plus a small margin. A margin of 5 to 10 percent, or 512 bytes to 1 KB on tiny models, absorbs alignment differences between toolchain builds. Do not use a 30 percent fudge factor on a 200 KB arena; that wastes 60 KB you may need for TLS.
Re-measure whenever the model, the op set, or the component version changes. Kernel scratch buffers differ between reference and ESP-NN implementations, so toggling ESP-NN can change the arena requirement.

For our 4,744-parameter autoencoder the activation tensors are tiny: the largest live pair is 64 inputs plus 32 outputs, which is 96 bytes of int8 data. The arena will therefore be dominated by bookkeeping, the planner’s metadata, and per-operator data rather than activations, and you should expect low single-digit kilobytes. We have not measured it on your build, so run the procedure and record the number.

Which memory region?

ESP32-S3 boards can attach external PSRAM. It is tempting to put the arena there because it is large. Espressif’s own benchmark note says the person-detection figures use internal memory, and its MobileNetV3 figures use SPIRAM for the arena, which is a reminder that the placement changes timing. External PSRAM accesses go through a cache and are slower than internal SRAM, and heavy PSRAM traffic can contend with flash cache and DMA. The practical rule: keep small, latency-sensitive arenas in internal RAM, and spill only large vision models to PSRAM, then re-benchmark. In ESP-IDF terms, MALLOC_CAP_INTERNAL pins to internal SRAM while MALLOC_CAP_SPIRAM selects external memory.

What to benchmark, and how

Time three things separately, not just Invoke():

Preprocessing time: capture to feature vector. For audio and vibration this is often larger than inference. Edge Impulse documents that on a standard ESP32 its MFCC feature extraction fell from 297 ms to 54 ms with ESP-DSP acceleration, which is a reminder that DSP code can dominate the loop.
Invoke time: use esp_timer_get_time() around Invoke(), as in the code, averaged over hundreds of runs after a warm-up.
End-to-end latency: sample arrival to decision published.

Keep a simple table in your repository and update it whenever the model, ESP-NN setting, or ESP-IDF version changes:

Metric	Reference kernels	ESP-NN	Notes
Invoke time (us)	measure	measure	average of 500 runs
Arena used (bytes)	measure	measure	from arena_used_bytes
Feature time (us)	measure	measure	FFT plus log
Flash for model (bytes)	measure	n/a	file size plus alignment

We leave the cells blank on purpose. Any number we printed would be for a model and board you do not have, and fabricated benchmarks are worse than none. Use Espressif’s published person-detection figures only as a proof that the optimized path is dramatically faster, not as a prediction for your network.

Reading the speedup honestly

A convolution-heavy network benefits most from ESP-NN because convolution and depthwise convolution dominate its compute and are among the optimized kernels. Our autoencoder uses only fully connected layers, which ESP-NN also optimizes, but with 4,608 MACs the absolute saving is small compared with the FFT. This is why the first optimization target in the anomaly node is usually the FFT and the sampling pipeline, not the neural network. Profile before you optimize. A model with a larger first layer, a 1D convolution front end, or a wider autoencoder will shift the balance back toward inference, and then the ESP-NN switch earns its keep.

TensorFlow Lite Micro vs ExecuTorch, LiteRT, and Edge Impulse in 2026

TensorFlow Lite Micro is the safest default on ESP32, but it is no longer the only serious option. The right choice depends on the model’s origin framework, how much of the toolchain you want managed for you, and how far down the memory ladder you must go.

Figure 4: Decision flow for choosing an inference runtime for an ESP32-class device, from model origin to deployment constraints.

What each option is, verified against primary sources

LiteRT (full) versus LiteRT for Microcontrollers. The full LiteRT is the renamed TensorFlow Lite for phones and embedded Linux. Google’s own overview says more powerful devices such as embedded Linux boards may be easier to target with the standard framework. On an ESP32 you use the microcontroller flavor, which is the TFLM code base.

ExecuTorch. PyTorch’s on-device runtime. Its documentation (version 1.5 at the time we checked) lists backends including XNNPACK, Arm Ethos-U, Arm Cortex-M, NXP eIQ Neutron, and Cadence Xtensa, and says it can run on “constrained microcontrollers”. The page does not state a runtime size, and we found no primary-source statement that ExecuTorch has a first-class ESP32 backend, so we cannot recommend it for ESP32 today without your own proof of concept. If your team trains in PyTorch and targets Cortex-M or Ethos-U silicon, it is worth evaluating seriously; for ESP32 specifically, verify current support before committing. Our note on ONNX Runtime 1.30 versus 1.29 edge inference changes covers the adjacent ONNX route, which targets larger devices than an ESP32.

Edge Impulse. A managed platform covering data collection, feature extraction, training, and deployment as C++ libraries, Arduino libraries, or prebuilt firmware. Its ESP32 documentation lists tested boards (ESP-EYE and DFRobot FireBeetle ESP32) and states that other boards need modifications. It also documents ESP-DSP acceleration of feature extraction. The trade is speed of development against control and vendor dependence.

Espressif’s own stack. Beyond esp-tflite-micro, Espressif ships further AI libraries; if your use case is voice or face recognition, check their dedicated frameworks before building on raw TFLM. We did not benchmark them here.

Decision matrix

Situation	Best fit	Why
Keras or TensorFlow model, ESP32 or ESP32-S3, custom firmware	TFLM via esp-tflite-micro	Maintained component, ESP-NN kernels, small footprint
Team without ML infrastructure, wants a data-to-firmware workflow	Edge Impulse	Managed pipeline, DSP blocks, quick iteration
PyTorch model, Cortex-M or Ethos-U silicon	ExecuTorch	First-class backends for Arm targets
Vision or LLM-class workload	Not an ESP32 job	Use Jetson, Hailo, or an NPU microcontroller
Extreme size limits under 16 KB total	Hand-written C inference	A framework may cost more than the model

The model-format question

The community habit of calling everything “TFLite” hides a real compatibility rule: the .tflite flatbuffer schema is shared between LiteRT and TFLM, but the operator set on the microcontroller is smaller. A model that converts and runs on a phone may fail AllocateTensors() on the device because an operator is missing. Design for the intersection from the start: stick to fully connected, convolution, depthwise convolution, pooling, softmax, reshape, add and similar basics, and test op coverage in CI by loading the model in the host-built TFLM interpreter.

Trade-offs, Gotchas, and What Goes Wrong

Edge inference fails in unglamorous ways. These are the failure modes worth designing against, ordered roughly by how often they bite.

Silent fallback to reference kernels. If ESP-NN is disabled, misconfigured, or the target is a chip with only generic optimizations, the code still runs and still gives correct answers, only far slower. Espressif’s own numbers show a 43x gap on the S3 for person detection. Detect it by logging invoke time at boot and asserting it against a budget in your factory test.

Preprocessing drift. The model sees whatever the firmware computes. A different window function, a missing log1p, or a different ADC scaling produces plausible-looking features that break the autoencoder. Fix it with a golden test: save a raw window and its expected feature vector from Python, then run the C++ code on the same window in a unit test and assert agreement within tolerance.

Calibration outliers and operating-mode coverage. Covered above, but it deserves repeating because it is invisible. Compare float and int8 error distributions on a held-out set that includes every normal operating mode. If the int8 p99.9 error shifts by more than a few percent relative to float, change the calibration set, not the threshold.

Threshold brittleness. A fixed reconstruction-error threshold is a one-number model of “normal”. Machines age, sensors get remounted, and seasonal temperatures move spectra. Plan for threshold updates over MQTT, per-device baselines learned in a commissioning phase, and a persistence rule such as requiring N consecutive windows above threshold before alerting. Our MQTT Sparkplug B reference architecture and the comparison of Sparkplug B versus plain MQTT topics cover how to publish scores and receive configuration in a way that integrates with a unified namespace.

Arena exhaustion after an update. An OTA update that changes the model or toggles an optimization can push the arena over its limit. Because the failure occurs in AllocateTensors(), the node may boot-loop. Mitigate with A/B partitions, a self-test before marking an image valid, and an arena margin that you verify in CI by running the new model on a host build of TFLM with a recorded arena report.

Concurrency and memory contention. The interpreter is not thread-safe for concurrent Invoke() calls on the same instance. Run inference in one task. Wi-Fi and TLS traffic compete for internal RAM, so a large internal arena can starve the network stack. Check heap_caps_get_minimum_free_size() under load, not at idle.

Version churn. The rebrand, API changes such as the error reporter removal, and ESP-IDF major versions all mean code copied from older posts may not compile. Pin component and IDF versions, vendor-lock your model conversion environment in a container, and keep a reproducible build.

Security and model integrity. A tampered model changes device behavior. Sign firmware, enable secure boot where your threat model requires it, and treat the model file as code. For a fuller view of securing edge fleets, see our guide on running k3s at the edge in production for the gateway tier.

What int8 cannot do. Quantization loses precision, and on-device training is not supported by the runtime; Google lists lack of on-device training as an explicit limitation. Adaptation must happen off-device, with new models pushed to the node.

Practical Recommendations

Treat the ESP32 node as a sensor with a small, fixed-function classifier, not as a miniature server. Train and quantize in a reproducible container, and keep the host-side validation script in CI so every model change is checked against the float baseline. Choose the S3 if there is any chance of convolutional models, because vector instructions through ESP-NN are what turn vision-class networks from unusable into feasible. Choose the cheaper C3 or the original ESP32 only when your model is genuinely tiny and you have measured it.

Keep the data path honest. Push only decisions and compact scores upstream: a 16-bit anomaly score every second is a few bytes, against streaming raw vibration at 2,560 samples per second. Send periodic raw snapshots on demand for diagnosis. At the plant level, feed those scores into your historian and digital twin so the twin’s health model has edge-derived features; see our guide to ingesting time-series from OPC UA historians into digital twins.

Checklist before you ship:

[ ] Model uses only operators you registered; verified on a host build.
[ ] Representative dataset covers all normal operating modes and has no outliers.
[ ] Float versus int8 error percentiles compared; threshold derived from int8 results.
[ ] Preprocessing golden test passes between Python and C++.
[ ] ESP-NN confirmed active; invoke time asserted in factory test.
[ ] Arena measured with arena_used_bytes(), margin applied, re-measured per release.
[ ] Component version, ESP-IDF version, and TensorFlow version pinned.
[ ] OTA uses A/B partitions with self-test; model integrity protected.
[ ] Threshold remotely configurable; persistence logic prevents alert flapping.

Frequently Asked Questions

Is TensorFlow Lite Micro still supported in 2026?

Yes. Google’s AI Edge documentation now names it LiteRT for Microcontrollers, but the open-source repository is still tensorflow/tflite-micro, and Espressif’s esp-tflite-micro component tracks it, with version 1.4.1 listed as latest on the registry when we checked. Expect the two names to be used interchangeably. For long-lived products, pin versions and monitor the upstream repository and Espressif’s release notes, because component and ESP-IDF support windows move.

What is the difference between TensorFlow Lite Micro and LiteRT?

LiteRT is the 2024 rename of TensorFlow Lite for phones and embedded Linux. LiteRT for Microcontrollers is the documentation name for TensorFlow Lite Micro, a separate, much smaller C++ runtime with no dynamic allocation, no operating system dependency, and a restricted operator set. They share the flatbuffer model format, but a model that runs on LiteRT may not run on the microcontroller runtime if it uses an unsupported operator.

How much RAM does TensorFlow Lite Micro need on ESP32?

It depends entirely on the model. Google states the core runtime fits in 16 KB on a Cortex-M3, but your tensor arena adds activations and scratch space, from a few kilobytes for a tiny dense network to far more for vision models. Measure with arena_used_bytes() after AllocateTensors(), add a modest margin, and re-measure whenever the model or optimization settings change. Weights live in flash, not RAM.

Does ESP-NN work on every ESP32?

ESP-NN supports the ESP32-S3, ESP32-P4, ESP32, and ESP32-C3. The S3 and P4 get assembly kernels using vector or SIMD instructions, while the original ESP32 and C3 get generic optimizations, so speedups differ. Espressif’s person-detection figures show 54 ms on the S3 versus 380 ms on the ESP32, both at 240 MHz. The README notes that P4 optimization is still in progress, so recheck before relying on it.

Can I train a model on the ESP32 itself?

Not with TensorFlow Lite Micro. Google lists on-device training as unsupported. The practical pattern is to train in the cloud or on a workstation, quantize, and deploy the flatbuffer. For adaptation, collect features on the device, send them upstream, retrain periodically, and deliver a new model over the air. Simple statistical baselines, such as running means, can adapt on-device without a neural network.

Should I use Edge Impulse or TensorFlow Lite Micro directly?

Use Edge Impulse when you want a managed pipeline from data collection to a C++ or Arduino library and can accept its supported boards and constraints; its ESP32 documentation lists ESP-EYE and FireBeetle as tested. Use TFLM directly through esp-tflite-micro when you need custom hardware, tight memory control, or reproducible, auditable builds. Many teams prototype on the platform and move to direct integration for production.