Checksums

A checksum is a small value computed from a block of data that is used to detect errors introduced during transmission or storage. If the data changes (even by a single bit), the checksum changes, revealing corruption. Common algorithms include CRC32, MD5, SHA-256.

Why This Matters

Checksums protect data integrity in distributed systems. When transferring files across networks, replicating databases, or storing data on disk, you need to verify that data has not been corrupted.

The Building Blocks

• Error detection, not correction: Checksums tell you data is corrupted but cannot fix it. You must retransmit or re-read.
• CRC32: Fast, used in TCP, Ethernet, and zip files. Good for detecting transmission errors.
• MD5/SHA: Cryptographic hash functions. Slower but provide stronger guarantees. Used for file integrity verification.
• TCP checksums: Every TCP segment includes a checksum. The receiver verifies it and discards corrupted segments.
• Application-level checksums: S3, GFS, and HDFS compute checksums for stored data and verify on every read.

Under the Hood

The sender computes a checksum of the data and sends both the data and the checksum. The receiver computes the checksum independently and compares it to the received checksum. If they match, the data is (very likely) intact. If they differ, the data was corrupted and must be retransmitted.

How Companies Actually Do This

Amazon S3 uses MD5 checksums for every object stored. You can verify data integrity by comparing the ETag header.

Git uses SHA-1 hashes (moving to SHA-256) to identify every commit, tree, and blob. Any change produces a different hash.

ZFS file system uses checksums on every block of data, detecting and correcting silent data corruption (bit rot).

Common Pitfalls

1. Using checksums for security (they prevent corruption, not tampering — use HMAC for authentication)
2. Not verifying checksums after data transfer

Interview Questions Worth Practicing

1. What is a checksum and how is it used in distributed systems?
2. How does TCP ensure data integrity?
3. What is the difference between CRC and cryptographic hashes?

The Tradeoffs

• Speed vs Security: CRC32 is very fast but not cryptographically secure. SHA-256 is secure but slower.
• Checksum size: Larger checksums catch more errors but use more bandwidth/storage.

Related Topics

• reliability
• Data Replication
• fault-tolerance

Original DosiBridge notes for Checksums

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Checksums is to connect the concept to a production service where growth exposes the first weak assumption in the original design. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• Packet Path: For Checksums, decide the packet path before choosing tools; otherwise the design looks clean but fails when checksums changes.
• Timeout Budget: For Checksums, decide the timeout budget before choosing tools; otherwise the design looks clean but fails when checksums changes.
• TLS Boundary: For Checksums, decide the TLS boundary before choosing tools; otherwise the design looks clean but fails when checksums changes.
• Proxy Behavior: For Checksums, decide the proxy behavior before choosing tools; otherwise the design looks clean but fails when checksums changes.
• Backpressure: For Checksums, decide the backpressure before choosing tools; otherwise the design looks clean but fails when checksums changes.

Production review question

If a team adopts Checksums, what observable signal proves it helped instead of only making the architecture diagram more fashionable?

Practical metric to track

Track checksums health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Checksums, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.