Skip to content

Uploading & deduplication

Not yet on Maven Central

dedup4j 0.1.0 has not been published. These coordinates are the intended release coordinates and cannot be resolved from Maven Central today. The Maven Central namespace is confirmed in release gate G0 and this file is the single place it is edited.

BlobStore is the upload front door. It takes bytes in whatever form your application already has them and returns a BlobReference.

The facade

public interface BlobStore {
    BlobReference store(MultipartFile file);
    BlobReference store(Path path);
    BlobReference store(byte[] content, String filename, String contentType);
    BlobReference store(InputStream content, long sizeBytes, String filename,
                        String contentType, Map<String, String> metadata);
    BatchStoreResult storeAll(MultipartFile[] files);
}

Four store overloads for the four shapes bytes usually arrive in — a web upload, a file on disk, an in-memory array, or a stream you are already holding.

Uploads are capped at 25 MB by default

dedup4j.deduplication.max-upload-size defaults to 25MB and is enforced. Larger uploads are rejected. Raise it deliberately — the content is read into memory to be hashed.

The stream overload needs the size up front

sizeBytes is a parameter because content length is part of content identity and dedup4j will not buffer an entire stream to discover it.

What store returns

public record BlobReference(
    UUID assetContentId,      // your handle — store this
    ContentHash contentHash,  // content identity
    String contentType,
    String storageProvider,
    String bucketOrContainer,
    String objectKey,
    boolean duplicate         // were these bytes already here?
) {}

assetContentId is the only field most applications persist. The rest describe where the bytes landed and are useful for diagnostics and for building your own download URLs via location().

Content identity

public record ContentHash(String algorithm, String hash, long sizeBytes) {}

Identity is the triple, not the hash alone. The digest is SHA-256, which is where collision resistance actually comes from; size is an additional discriminator, so two objects match only if both agree.

The algorithm is fixed

SHA-256 is not configurable. Changing it would orphan every previously stored object, since identity is the hash — see ContentHasher.

What identity is not:

  • not the filename — invoice.pdf and a-copy.pdf with the same bytes are the same content
  • not the content type
  • not the metadata
  • not the upload time or uploader

Only the bytes decide.

Storing the same bytes twice

BlobReference first  = blobStore.store(bytes, "invoice.pdf", "application/pdf");
BlobReference second = blobStore.store(bytes, "copy.pdf",    "application/pdf");

first.duplicate();   // false — new content
second.duplicate();  // true  — recognised

second.assetContentId().equals(first.assetContentId());  // true

On the second call dedup4j hashes the content, finds an existing row, uploads nothing, and returns a reference to the object already stored.

A duplicate store already counts for you

store returning duplicate: true has already incremented the reference count. New content is created at a count of one; a duplicate store retains the existing content.

So the rule is symmetric: every store call yields exactly one reference, and needs exactly one matching release. Do not call retain after a duplicate store — that would count the same record twice, and the content would never be deleted.

retain is for the other case: a new record pointing at content you did not just store, such as copying an existing attachment onto a second document without re-uploading bytes.

Batch uploads

BatchStoreResult result = blobStore.storeAll(files);

result.allSucceeded();
result.successes();   // List<BlobStoreSuccess>
result.failures();    // List<BlobStoreFailure>

Partial success is the normal case

storeAll is not all-or-nothing. Each file succeeds or fails independently, and one bad file does not discard the rest.

The outcome type is a sealed interface, so the compiler can check you handled both:

public sealed interface BatchStoreOutcome
        permits BlobStoreSuccess, BlobStoreFailure {
    int index();
    String filename();
}

for (BatchStoreOutcome outcome : result.outcomes()) {
    switch (outcome) {
        case BlobStoreSuccess s ->
            attach(s.reference().assetContentId());
        case BlobStoreFailure f ->
            report(f.index(), f.filename(), f.failure());
    }
}

index() is the file's position in the array you passed, so a failure maps back to the exact input — necessary when several uploads share a filename or when the filename is absent.

Check failures()

Treating a BatchStoreResult as success because the call returned is the easiest mistake to make here. The call returning means the batch ran, not that every file stored. Use allSucceeded() before assuming.

What dedup4j does not do on upload

  • No virus scanning or content inspection. Bytes are stored as given.
  • No image processing, transcoding, or thumbnails.
  • No access control. Anyone who can call store can store.
  • No atomicity with your own writes across the database and the object store — see Architecture & limitations.

Next