ESP32 Data Logging: SD Card, SPIFFS, and Cloud Options

Whether you are monitoring temperature in a greenhouse, tracking vibration on industrial equipment, or recording air quality across a city, data logging is the backbone of any serious IoT project. Raw sensor readings are only useful if you can store them reliably, retrieve them later, and analyze trends over time.

The ESP32 gives you three practical storage paths: SD cards for large, removable storage; SPIFFS/LittleFS for small built-in flash storage; and cloud services for remote access and visualization. This guide walks through all three with working code, wiring diagrams, and honest trade-offs so you can pick the right approach for your project.

Why Data Logging Matters

Before jumping into code, consider why you need logging in the first place:

• Trend analysis: Spot slow-moving changes (a motor bearing warming up over weeks) that real-time monitoring misses.
• Debugging: When something fails at 3 AM, timestamped logs tell you exactly what happened and when.
• Compliance: Industries like food storage, pharmaceuticals, and agriculture require proof that conditions stayed within limits.
• Pattern recognition: Correlate sensor data with external events — did humidity spike every time the door opened?
• Offline operation: Many field deployments have no WiFi. Local storage is the only option.

The ESP32 is ideal for this work. It has enough processing power to handle sensor reads, file I/O, and WiFi uploads simultaneously, and its deep sleep mode lets battery-powered loggers run for months.

Option 1: SD Card Logging

An SD card is the most versatile option. You get gigabytes of removable storage, standard FAT32 formatting that any computer can read, and no dependency on WiFi.

Wiring the MicroSD Card Module

The SD card module communicates over SPI. Here is the standard wiring to an ESP32 DevKit:

SD Module Pin	ESP32 GPIO	Function
VCC	3.3V	Power (some modules accept 5V with onboard regulator)
GND	GND	Ground
MOSI	GPIO 23	Master Out, Slave In
MISO	GPIO 19	Master In, Slave Out
SCK	GPIO 18	Serial Clock
CS	GPIO 5	Chip Select

Note: GPIO 5 is the default CS pin for the SD library on ESP32. You can use any available GPIO, but you will need to pass it to SD.begin(csPin).

Basic SD Card Initialization and CSV Writing

This sketch reads a DHT22 sensor every 30 seconds and appends a CSV row to the SD card:

#include <SD.h>
#include <SPI.h>
#include <DHT.h>

#define SD_CS 5
#define DHT_PIN 4
#define DHT_TYPE DHT22

DHT dht(DHT_PIN, DHT_TYPE);
unsigned long lastLog = 0;
const unsigned long LOG_INTERVAL = 30000; // 30 seconds

void setup() {
  Serial.begin(115200);
  dht.begin();

  if (!SD.begin(SD_CS)) {
    Serial.println("SD card initialization failed!");
    Serial.println("Check: wiring, card inserted, FAT32 format, card < 32GB");
    return;
  }
  Serial.println("SD card ready.");

  // Write CSV header if file is new
  if (!SD.exists("/datalog.csv")) {
    File f = SD.open("/datalog.csv", FILE_WRITE);
    if (f) {
      f.println("millis,temperature_c,humidity_pct");
      f.close();
    }
  }
}

void loop() {
  if (millis() - lastLog >= LOG_INTERVAL) {
    lastLog = millis();

    float temp = dht.readTemperature();
    float hum = dht.readHumidity();

    if (isnan(temp) || isnan(hum)) {
      Serial.println("Sensor read failed, skipping.");
      return;
    }

    File f = SD.open("/datalog.csv", FILE_APPEND);
    if (f) {
      f.printf("%lu,%.1f,%.1f\n", millis(), temp, hum);
      f.flush(); // Force write to card — critical for power-loss safety
      f.close();
      Serial.printf("Logged: %.1f C, %.1f%%\n", temp, hum);
    } else {
      Serial.println("Failed to open file for writing.");
    }
  }
}

Key detail: Always call f.flush() before f.close(). If power is cut between a write and close, flush() ensures data is actually on the card and not stuck in a buffer.

Timestamped Logging with NTP

Millis-based timestamps are useless once you unplug the board. For real timestamps, sync with an NTP server over WiFi at boot:

#include <WiFi.h>
#include <time.h>
#include <SD.h>
#include <SPI.h>

#define SD_CS 5

const char* ssid = "YOUR_WIFI";
const char* password = "YOUR_PASSWORD";

// IST is UTC+5:30
const long gmtOffset_sec = 19800;
const int daylightOffset_sec = 0;

void syncTime() {
  WiFi.begin(ssid, password);
  Serial.print("Connecting to WiFi");

  int attempts = 0;
  while (WiFi.status() != WL_CONNECTED && attempts < 20) {
    delay(500);
    Serial.print(".");
    attempts++;
  }

  if (WiFi.status() == WL_CONNECTED) {
    configTime(gmtOffset_sec, daylightOffset_sec, "pool.ntp.org", "time.nist.gov");

    struct tm timeinfo;
    if (getLocalTime(&timeinfo, 10000)) {
      Serial.println("\nTime synced: ");
      Serial.println(&timeinfo, "%Y-%m-%d %H:%M:%S");
    }

    // Disconnect WiFi to save power — time is now in RTC memory
    WiFi.disconnect(true);
    WiFi.mode(WIFI_OFF);
  } else {
    Serial.println("\nWiFi failed — timestamps will use millis.");
  }
}

String getTimestamp() {
  struct tm timeinfo;
  if (!getLocalTime(&timeinfo)) {
    return String(millis());
  }
  char buf[25];
  strftime(buf, sizeof(buf), "%Y-%m-%d %H:%M:%S", &timeinfo);
  return String(buf);
}

void setup() {
  Serial.begin(115200);
  SD.begin(SD_CS);
  syncTime();
}

void logData(float temp, float humidity) {
  File f = SD.open("/datalog.csv", FILE_APPEND);
  if (f) {
    f.printf("%s,%.1f,%.1f\n", getTimestamp().c_str(), temp, humidity);
    f.flush();
    f.close();
  }
}

Tip: The ESP32's internal RTC keeps time after NTP sync, even with WiFi off. It drifts a few seconds per day, which is fine for most logging. For multi-day offline accuracy, add a DS3231 RTC module (covered later).

Creating a New File Each Day

For long-running loggers, splitting data into daily files keeps things manageable:

String getDailyFilename() {
  struct tm timeinfo;
  if (!getLocalTime(&timeinfo)) {
    return "/fallback.csv";
  }
  char filename[20];
  strftime(filename, sizeof(filename), "/%Y%m%d.csv", &timeinfo);
  return String(filename);
}

void logWithDailyFile(float temp, float humidity) {
  String filename = getDailyFilename();

  // Write header if this is a new day's file
  if (!SD.exists(filename.c_str())) {
    File f = SD.open(filename.c_str(), FILE_WRITE);
    if (f) {
      f.println("timestamp,temperature_c,humidity_pct");
      f.close();
    }
  }

  File f = SD.open(filename.c_str(), FILE_APPEND);
  if (f) {
    f.printf("%s,%.1f,%.1f\n", getTimestamp().c_str(), temp, humidity);
    f.flush();
    f.close();
  }
}

This produces files like 20260110.csv, 20260111.csv, and so on. Pop the card into your laptop and each file is one day of data, ready for Excel or Python analysis.

SD Card Selection Tips

Requirement	Recommendation
File system	FAT32 (not exFAT). ESP32 SD library only supports FAT32.
Maximum size	32 GB. Larger cards default to exFAT and need manual FAT32 formatting.
Speed class	Class 10 or UHS-I. Faster cards do not help much at SPI speeds but they are more reliable.
Brand	Stick with SanDisk, Samsung, or Kingston. Cheap no-brand cards often fail silently.
Wear leveling	Not needed for typical logging. Even at one write per second, a 32 GB card will outlast the project.

Reading Data Back

You can read logged data directly on the ESP32, useful for serving it over a web interface:

void printFile(const char* path) {
  File f = SD.open(path);
  if (!f) {
    Serial.println("Failed to open file.");
    return;
  }
  while (f.available()) {
    Serial.write(f.read());
  }
  f.close();
}

Option 2: SPIFFS / LittleFS (Internal Flash)

The ESP32 has 4 MB of flash memory, and you can partition a portion of it as a filesystem. SPIFFS (Serial Peripheral Interface Flash File System) was the original option; LittleFS is its modern replacement with better reliability and directory support.

What You Get

• No extra hardware — the filesystem lives on the ESP32 itself.
• Typical size: 1.5 MB with the default partition scheme. You can adjust this in the partition table.
• Survives reboots — data persists until you explicitly erase or reflash.

Writing Sensor Data to LittleFS

#include <LittleFS.h>

void setup() {
  Serial.begin(115200);

  if (!LittleFS.begin(true)) { // true = format on first use
    Serial.println("LittleFS mount failed!");
    return;
  }
  Serial.println("LittleFS ready.");

  // Check available space
  size_t total = LittleFS.totalBytes();
  size_t used = LittleFS.usedBytes();
  Serial.printf("Storage: %u / %u bytes used (%.1f%% free)\n",
                used, total, 100.0 * (total - used) / total);
}

void logToFlash(float temp, float humidity) {
  File f = LittleFS.open("/log.csv", FILE_APPEND);
  if (f) {
    f.printf("%lu,%.1f,%.1f\n", millis(), temp, humidity);
    f.close();
  }
}

void readFlashLog() {
  File f = LittleFS.open("/log.csv", FILE_READ);
  if (f) {
    while (f.available()) {
      Serial.write(f.read());
    }
    f.close();
  }
}

Limitations

Factor	LittleFS	SD Card
Storage	~1.5 MB	Up to 32 GB
Write cycles	~10,000-100,000 per sector	Practically unlimited
Speed	Slower (flash erase cycles)	Faster for sequential writes
Removable	No — data stays on chip	Yes — pop card into laptop
Extra hardware	None	SD module + card
Power draw	Negligible	50-100 mA during writes

When to Use LittleFS

• Configuration storage: WiFi credentials, calibration values, device settings.
• Small data buffers: Hold a few hundred readings before uploading to the cloud.
• Temporary cache: Buffer data when the SD card is full or WiFi is down.
• Fallback storage: If the SD card fails, switch to LittleFS as an emergency buffer.

Do not use LittleFS as your primary long-term storage if you are logging more than a few KB per day. The write cycle limit and small capacity make it unsuitable for heavy logging.

Option 3: Cloud Logging

If your ESP32 has WiFi access, you can stream data directly to cloud services for remote monitoring, visualization, and alerting.

ThingSpeak (Free, Quick Setup)

ThingSpeak by MathWorks offers free accounts with up to 4 channels of 8 fields each, with built-in graphing:

#include <WiFi.h>
#include <HTTPClient.h>

const char* ssid = "YOUR_WIFI";
const char* password = "YOUR_PASSWORD";
const char* thingspeakKey = "YOUR_WRITE_API_KEY";

void sendToThingSpeak(float temp, float humidity) {
  if (WiFi.status() != WL_CONNECTED) {
    WiFi.begin(ssid, password);
    int retries = 0;
    while (WiFi.status() != WL_CONNECTED && retries < 20) {
      delay(500);
      retries++;
    }
    if (WiFi.status() != WL_CONNECTED) return;
  }

  HTTPClient http;
  String url = "https://api.thingspeak.com/update?api_key=";
  url += thingspeakKey;
  url += "&field1=" + String(temp, 1);
  url += "&field2=" + String(humidity, 1);

  http.begin(url);
  int httpCode = http.GET();

  if (httpCode == 200) {
    Serial.println("ThingSpeak update successful.");
  } else {
    Serial.printf("ThingSpeak error: %d\n", httpCode);
  }
  http.end();
}

Free tier limit: One update every 15 seconds. Good for environmental monitoring, not for high-frequency data.

Google Sheets via Apps Script

For a free, spreadsheet-based dashboard, deploy a Google Apps Script as a web app:

Step 1 — In Google Sheets, go to Extensions > Apps Script and paste:

function doGet(e) {
  var sheet = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
  var timestamp = new Date().toLocaleString("en-IN", {timeZone: "Asia/Kolkata"});

  sheet.appendRow([
    timestamp,
    parseFloat(e.parameter.temp),
    parseFloat(e.parameter.humidity)
  ]);

  return ContentService.createTextOutput("OK");
}

Deploy as a web app (Execute as: Me, Access: Anyone). Copy the deployment URL.

Step 2 — ESP32 code:

#include <WiFi.h>
#include <HTTPClient.h>

const char* sheetsURL = "https://script.google.com/macros/s/YOUR_DEPLOYMENT_ID/exec";

void sendToGoogleSheets(float temp, float humidity) {
  if (WiFi.status() != WL_CONNECTED) return;

  HTTPClient http;
  String url = String(sheetsURL) + "?temp=" + String(temp, 1)
               + "&humidity=" + String(humidity, 1);

  http.setFollowRedirects(HTTPC_STRICT_FOLLOW_REDIRECTS);
  http.begin(url);
  int httpCode = http.GET();

  Serial.printf("Google Sheets response: %d\n", httpCode);
  http.end();
}

Important: Google Apps Script redirects once before responding. The setFollowRedirects call is essential or you will get a 302 and no data.

InfluxDB + Grafana (Professional Dashboards)

For serious time-series analysis, InfluxDB is purpose-built for sensor data, and Grafana turns it into beautiful dashboards. You can self-host both on a Raspberry Pi or use InfluxDB Cloud's free tier.

#include <WiFi.h>
#include <HTTPClient.h>

const char* influxURL = "http://192.168.1.100:8086/api/v2/write?org=myorg&bucket=sensors&precision=s";
const char* influxToken = "YOUR_API_TOKEN";

void sendToInfluxDB(float temp, float humidity) {
  if (WiFi.status() != WL_CONNECTED) return;

  HTTPClient http;
  http.begin(influxURL);
  http.addHeader("Authorization", "Token " + String(influxToken));
  http.addHeader("Content-Type", "text/plain");

  // InfluxDB line protocol: measurement,tag=value field=value timestamp
  String payload = "environment,device=esp32_01 ";
  payload += "temperature=" + String(temp, 2) + ",";
  payload += "humidity=" + String(humidity, 2);

  int httpCode = http.POST(payload);
  Serial.printf("InfluxDB response: %d\n", httpCode);
  http.end();
}

InfluxDB line protocol is compact and efficient. Grafana connects to InfluxDB as a data source and lets you build dashboards with alerts (e.g., email you if temperature exceeds 50 degrees).

Hybrid Approach: Local + Cloud

The most robust systems combine local and cloud storage. Log everything to SD, upload to the cloud when WiFi is available:

void logAndUpload(float temp, float humidity) {
  // Always log locally first — this never fails (if SD card works)
  logWithDailyFile(temp, humidity);

  // Attempt cloud upload — non-blocking, best-effort
  if (WiFi.status() == WL_CONNECTED) {
    sendToThingSpeak(temp, humidity);
  } else {
    Serial.println("WiFi unavailable — data saved to SD only.");
  }
}

For batch uploading offline data when WiFi reconnects:

void uploadPendingData() {
  File f = SD.open("/pending.csv", FILE_READ);
  if (!f || f.size() == 0) return;

  while (f.available()) {
    String line = f.readStringUntil('\n');
    // Parse CSV line and send to cloud
    // ... your upload logic here
  }
  f.close();

  // Clear the pending file after successful upload
  SD.remove("/pending.csv");
}

Ring Buffer in LittleFS

When SD card storage is full or the card is physically absent, a ring buffer in LittleFS acts as a safety net. It keeps the most recent N readings, overwriting the oldest:

#include <LittleFS.h>
#include <ArduinoJson.h>

#define MAX_ENTRIES 500
#define RING_FILE "/ring.json"

void ringBufferWrite(float temp, float humidity) {
  JsonDocument doc;
  File f = LittleFS.open(RING_FILE, FILE_READ);
  if (f && f.size() > 0) {
    deserializeJson(doc, f);
  }
  if (f) f.close();

  JsonArray arr = doc["data"].is<JsonArray>() ? doc["data"].as<JsonArray>() : doc["data"].to<JsonArray>();

  // Add new entry
  JsonObject entry = arr.add<JsonObject>();
  entry["t"] = millis();
  entry["tc"] = round(temp * 10) / 10.0;
  entry["h"] = round(humidity * 10) / 10.0;

  // Trim to MAX_ENTRIES (remove oldest)
  while (arr.size() > MAX_ENTRIES) {
    arr.remove(0);
  }

  // Write back
  f = LittleFS.open(RING_FILE, FILE_WRITE);
  if (f) {
    serializeJson(doc, f);
    f.close();
  }
}

Warning: This approach rewrites the entire file each time, which is hard on flash. For high-frequency logging, use a fixed-size binary format with head/tail pointers instead of JSON.

Data Format: CSV vs JSON vs Binary

Format	Size per row (example)	Readability	Parse speed	Best for
CSV	~35 bytes	Excellent — open in Excel	Fast	SD card logging, data exchange
JSON	~80 bytes	Good — human readable	Moderate	API payloads, config files
Binary	~12 bytes	None — needs custom parser	Fastest	High-frequency logging, limited storage

Recommendation: Use CSV for SD card logging (universally readable), JSON for cloud API payloads, and binary only when storage or write speed is the bottleneck.

A binary struct approach for comparison:

struct __attribute__((packed)) LogEntry {
  uint32_t timestamp;  // 4 bytes — epoch seconds
  int16_t temp_x10;    // 2 bytes — temperature * 10
  uint16_t hum_x10;    // 2 bytes — humidity * 10
};
// Only 8 bytes per reading vs 35+ for CSV

Power Considerations

The SD card module is one of the most power-hungry peripherals in a battery-powered logger.

State	Current draw
SD card idle	20-30 mA
SD card writing	50-100 mA
SD card off (power cut via MOSFET)	0 mA
ESP32 deep sleep	~10 uA

Batching Writes for Deep Sleep

Instead of writing every reading immediately, buffer in RTC memory and write in batches:

#define BATCH_SIZE 10

RTC_DATA_ATTR int bootCount = 0;
RTC_DATA_ATTR float tempBuffer[BATCH_SIZE];
RTC_DATA_ATTR float humBuffer[BATCH_SIZE];

void setup() {
  Serial.begin(115200);

  // Read sensor
  float temp = readTemperature(); // your sensor read function
  float hum = readHumidity();

  tempBuffer[bootCount % BATCH_SIZE] = temp;
  humBuffer[bootCount % BATCH_SIZE] = hum;
  bootCount++;

  // Write to SD only when buffer is full
  if (bootCount % BATCH_SIZE == 0) {
    SD.begin(5);
    File f = SD.open("/datalog.csv", FILE_APPEND);
    if (f) {
      for (int i = 0; i < BATCH_SIZE; i++) {
        f.printf("%d,%.1f,%.1f\n", bootCount - BATCH_SIZE + i, tempBuffer[i], humBuffer[i]);
      }
      f.flush();
      f.close();
    }
  }

  // Deep sleep for 60 seconds
  esp_sleep_enable_timer_wakeup(60 * 1000000ULL);
  esp_deep_sleep_start();
}

void loop() {} // Never reached

This way the SD card only powers up once every 10 readings, dramatically extending battery life. With 60-second intervals and a batch size of 10, the SD card activates once every 10 minutes.

Timestamps: NTP vs RTC Module

Method	Accuracy	Works offline	Cost	Drift
NTP sync at boot	Excellent initially	Only until reboot	Free	~5 sec/day
DS3231 RTC module	Excellent always	Yes	Around 100 INR	~2 sec/month
millis()	Relative only	Yes	Free	None (but resets on reboot)

For projects that run offline for days or weeks, a DS3231 RTC module is worth the small cost. It connects over I2C (SDA on GPIO 21, SCL on GPIO 22) and keeps sub-second accuracy:

#include <RTClib.h>

RTC_DS3231 rtc;

void setup() {
  Wire.begin();
  rtc.begin();

  // Set time once (comment out after first upload):
  // rtc.adjust(DateTime(2026, 1, 10, 14, 30, 0));
}

String getRTCTimestamp() {
  DateTime now = rtc.now();
  char buf[25];
  sprintf(buf, "%04d-%02d-%02d %02d:%02d:%02d",
          now.year(), now.month(), now.day(),
          now.hour(), now.minute(), now.second());
  return String(buf);
}

Reliability: Surviving Power Loss

Data corruption from unexpected power loss is the number one problem in field data loggers. Follow these rules:

1. Always call flush() after writing. Without it, data sits in a buffer and is lost on power loss.
2. Open, write, flush, close. Do not keep files open between writes. An open file handle is a corruption risk.
3. Use append mode (FILE_APPEND). Never read the entire file, modify it, and rewrite. Append is atomic at the filesystem level.
4. Write complete lines. A partial CSV row is worse than a missing one. Build the entire string first, then write it in one call.
5. Add a watchdog timer to reboot if your code hangs during a write:

#include <esp_task_wdt.h>

void setup() {
  esp_task_wdt_init(30, true); // 30-second watchdog
  esp_task_wdt_add(NULL);
}

void loop() {
  esp_task_wdt_reset(); // Reset watchdog each loop
  // ... your logging code
}

Troubleshooting Common Issues

SD card not detected

• Check wiring, especially CS pin. Try a different GPIO.
• Ensure the card is formatted as FAT32 (not exFAT or NTFS).
• Try a different card. Some cheap cards have incompatible controllers.
• Add a 10 uF capacitor between VCC and GND on the SD module for power stability.

File corruption or garbled data

• Always flush() and close() after writing.
• Never remove the SD card while the ESP32 is powered on.
• Check for insufficient power supply. The ESP32 plus an SD card need a stable 5V / 1A minimum.

Running out of space

• Implement automatic file rotation: delete files older than N days.
• Monitor free space and alert (via serial or LED) when below a threshold.
• Switch to binary format to reduce file sizes by 60-70%.

LittleFS write failures

• Check remaining space with LittleFS.totalBytes() - LittleFS.usedBytes().
• The flash partition may need formatting: LittleFS.format() (destroys all data).
• Reduce write frequency to extend flash life.

WiFi upload failures

• Always check WiFi.status() == WL_CONNECTED before HTTP calls.
• Set reasonable timeouts on HTTPClient: http.setTimeout(10000).
• Queue failed uploads locally and retry later. Never block the main logging loop waiting for WiFi.

Choosing the Right Approach

Scenario	Recommended approach
Remote field sensor, no WiFi	SD card + DS3231 RTC + deep sleep batching
Home weather station with WiFi	Cloud (ThingSpeak or Google Sheets) + LittleFS buffer
Industrial monitoring	SD card + InfluxDB/Grafana (hybrid)
Quick prototype or demo	LittleFS only
Battery-powered, weeks of runtime	SD card + batched writes + MOSFET power cut
Multi-site monitoring dashboard	Cloud (InfluxDB + Grafana) with local SD fallback

Wrapping Up

A reliable data logging system does not need to be complicated. Start with SD card logging in CSV format — it works offline, stores months of data, and any computer can read the files. Add NTP or an RTC module for proper timestamps. If you have WiFi, layer on cloud uploads for remote monitoring. Use LittleFS as a buffer or fallback, not as primary storage.

The key principles are: always flush writes to survive power loss, batch writes when running on battery, use daily file rotation to keep things organized, and build in redundancy so a WiFi outage or full SD card does not mean lost data.

All the components mentioned in this guide — ESP32 dev boards, MicroSD card modules, DS3231 RTC modules, and DHT22 sensors — are available at wavtron.in. Pick up a kit and start building your own data logging system today.