Interpretable document clustering

Organizations accumulate documents over the years, often without a consistent structure and in a variety of formats. While modern AI systems (e.g., agentic assistants, RAGs) provide new, powerful ways to speed up human workflows, their output is often bound by the quality of the input data.

After years of growing the archives, their owners not only lose any organized structure, but often don’t even know what kind of documents they contain. This is a poor starting point for an effective AI transformation: not knowing the data makes planning hard, and the messiness is likely to hurt performance.

This project investigates a lightweight approach to discovering the kinds of data a given corpus contains and to finding structure in a messy document store. This could be useful on its own or as a first step toward automatically (or semi-automatically) organizing such repositories and enriching them with metadata.

As a proof of concept, we keep the scope narrow and:

• focus on how much structure can be recovered from documents in an unsupervised manner
• go beyond text and experiment with representations based on layout and visual form
• explore grouping documents in an interpretable way and making the inferred structure easily readable

Background

Topic modelling

Finding structure in a text corpus is not a new problem, and has historically been solved with methods based on word statistics.

A classic example is LDA (Latent Dirichlet Allocation): each document is treated as a mixture of topics, and each topic as a distribution over words, inferred from which words tend to appear together. It remains a useful tool today, but it has limitations. It only sees text, so anything carried by layout or visual form is invisible to it. It also focuses on word or phrase counts, so it might miss more complex semantic signals that modern language models are built to retrieve.

Clustering asks a different question. Instead of a mixture of topics per document, it groups similar documents based on a representation that is not limited to bag-of-words.

Embedding based clustering

Modern approaches cluster documents in an embedding space (usually a neural embedding, though TF-IDF can also work), following roughly:

embeddings → dimensionality reduction → clustering

A concrete example of this approach applied to text is BERTopic, which uses sentence embeddings → UMAP → HDBSCAN, followed by the extraction of topic representations based on word frequency.

We follow a similar general pipeline for the clustering part, but focus on the choice of embeddings.

Beyond text

Documents carry more information than just the text’s semantics: layout and visual style are valuable signals and may be exactly what one wants to group by. We therefore look at how the choice of embedding shapes the resulting clusters and how it influences which signals surface.

Cluster interpretability

Both LDA and BERTopic describe the resulting clusters (topics) using representative words or phrases. That has its practical uses, but it does not make the clusters easy to interpret or organize. One can read into a word list and guess the theme of a group, but it takes effort, and such representation does not give much to base a structure on. The examples below show what this looks like in practice.

Both examples come from our synthetic corpus (described below under Data).

LDA (3 / 6 topics shown):

BERTopic (3 / 30 topics shown)

So, besides comparing embeddings and clustering approaches, we also explore a more readable way to interpret and structure the resulting groups.

Experiments

We start with the clustering itself, and come back to cluster interpretability afterward.

Data

Custom synthetic dataset: To be able to analyze the results with respect to different signals, we generate a synthetic dataset with three predefined, fully crossed axes:

• topic (5) – Cybersecurity, Human Resources, Legal & Compliance, Research & Development, Sales & Customer Relations
• document_type (5) – Report, Email, Invoice, Meeting Minutes, Presentation
• style (3) – three distinct visual templates (fonts, colors, layout conventions)

Each combination is passed to an LLM that writes the document as HTML, which we then convert to PDF and PNG. The resulting dataset contains 600 samples (5 × 5 × 3, with 8 documents per cell). We would rather use documents from a real corpus, but open datasets with actual documents (not just text) and with labels are hard to find.

RVL-CDIP: To benchmark on real data, we use the RVL-CDIP dataset, which contains historical documents from the tobacco industry. It is a standard dataset for image-based document classification and clustering research. The dataset is labeled by one of 16 document types. The quality of the scans is a bit low, which adds a lot of noise. To limit the computational scope, we used 50 samples for each type, resulting in 800 samples in the dataset.

Data preparation

For RVL, we use Azure Content Understanding for OCR (Optical Character Recognition) to extract text from the images (as .md, but almost no formatting). For the custom dataset, we have PDFs available, so we simply run Docling to extract .md, which might include some layout information.

Which embeddings

We want to explore representations that differ in the kinds of signals they carry, so we pick a few families that sit at various points on the text/visual spectrum. Text embeddings are computed on the extracted markdown, and image embeddings on the page PNG.

• Cohere Embed v4 (text, image) – multimodal document embedding; both modalities land in one shared space, and the image version is trained on documents so it seems to read the text too.
• text-embedding-3-small – general-purpose text embedding, a second opinion on the pure semantic signal.
• CLIP, SigLIP – general vision-language models trained to align images with captions.
• DINOv2, DiT – vision models without language supervision (DiT pretrained on scanned documents).
• TF-IDF, pixel: thumbnail – non-neural baselines: weighted term vectors, and the page as a flattened 32×32 grayscale image (we also tried a couple different pixel-based baselines).

The two Cohere modalities share one space, so we experiment with combining them:

• image − text – a guess worth testing: if both modalities share a space, subtracting the text might cancel the semantics and leave the visual and layout signal.
• image + text – carries both signals in one vector of the same width.

Clustering results

Here we run the specific embedding, reduce dimensionality with PCA (32 components), and use k-means as the clustering algorithm.

For each embedding, we reduce the dimensionality with PCA (32 components) and run the clustering with k-means. This gives a stable setup that lets us explore how well different embedding models work for different axes, without extra noise from the downstream settings. We did try UMAP and HDBSCAN (as in BERTopic), but the results were less stable and not consistently better or worse; we stuck to PCA and k-means to focus the comparison on the representations. HDBSCAN’s ability to set outliers aside makes it an appealing direction (potentially worth further exploration), but it would need more tuning to be reliable.

As baselines, we use BERTopic (on its default MiniLM embeddings) and LDA (both with lightly tuned hyperparameters), and a random partition for reference. We also use TF-IDF and a simple pixel baseline to compare with what a much simpler representation can already do.

The performance is measured against the labels (pre-defined axes in the custom dataset and the label in RVL) in terms of purity, which we define as the fraction that matches the dominant class in each cluster (essentially, how clean the clusters are on average). Purity increases with the number of clusters, so we compare configurations at similar cluster counts, denoted by k in the table (BERTopic is the exception, since HDBSCAN chooses k itself, but it underperforms on its default embeddings regardless).

Not surprisingly, the visual embeddings (CLIP, SigLIP, DINOv2, DiT) show better results on style and document type, while text-aware embeddings work well for clustering by topic. For RVL Cohere v4 (image) worked best, and it also got a decent score on the custom document type and topic. This makes sense as it’s a document-specific embedding and likely can read the text well enough.

Subtracting the Cohere v4 (text) embedding from the Cohere v4 (image) embedding appears to strip away the semantics needed for topical grouping and yields a representation that separates document types almost perfectly on the synthetic dataset. Whether that transfers to other corpora is an open question, but it is a very interesting result nevertheless.

We can also get rid of the semantic signal by downscaling the documents until the text becomes unreadable and then passing to Cohere v4 (image) (the Cohere v4 (image, 128px) row). This does improve clustering by document type and style, but nowhere near as much as the subtraction does. Interesting finding, but probably less useful for practical application.

Using more than one embedding

Each embedding recovered one axis well, rarely more than one. Rather than a single clustering, we could cluster twice with different configurations, each targeting a different signal. We experiment with two-level clustering to see whether it allows us to recover multiple axes simultaneously.

Approach

We considered different ideas of how to combine two levels of clustering:

• cluster independently with each embedding and take the cross-product of the two assignments
• cluster the second embedding separately within each level-1 cluster (we lose themes between clusters, but it might scale well on larger corpora)
• subtract each level-1 centroid before the second clustering, removing the first axis without splitting the data.

The last worked best in our experiments, so we used it for the results, but the margins were not large enough to draw real conclusions. This could be an interesting angle for further improvement.

A simpler alternative is to concatenate two embeddings and cluster the joint space once.

Results

We run 6 clusters at each level and collapse those that end up too small. We compare the 2-level clustering approach against a flat clustering at a matching cluster number to see what we get when just running flat clustering at comparable settings. We use the following notation to mark these configurations in the table below:

• flat: a single clustering on the given embedding.
• concat(A, B): a single clustering on the two embeddings stacked into one wider space.
• 2lvl: A → B: two-level clustering, level 1 on A and level 2 on B.

Two-level clustering preserves near-perfect document-type performance, while topic or style performance stays close to what flat clustering achieved (one at a time). This is two usable signals instead of one, at the same total number of groups.

For RVL, the results get worse, which is not surprising. The labels are varied along one axis only, and there are 16 classes, so in this case, the trade-off of a smaller number of groups at each level was too harsh. In that case, the concatenated approach worked best, so joining two levels of the signal remains beneficial here. This suggests that a fixed number of clusters at each level might generalize poorly (e.g., if the data has a lot of semantic variation but the format is homogenous or vice versa), and exploring methods that can adjust that dynamically (like HDBSCAN or some heuristics) might be crucial to make this approach more universal.

Grouping and interpretability

As we saw earlier, neither LDA nor BERTopic gives us any usable labels or descriptions of the clusters.

Instead, we use an LLM-based approach to label the groups. Passing all documents via an LLM could be quite expensive for a large corpus. We select N samples (we use N=5, but that could be tuned) for each cluster to show to the model for interpretation, which keeps it efficient. For this analysis, we picked those at random, but it could be beneficial to experiment with selecting either the most representative or the widest coverage sample.

We run this in 2 phases:

1. INTERPRET – here we ask the LLM to describe the cluster, give it a few-word label, and decide which characteristics are consistent across all the examples (e.g., domain, document type, etc.)
2. NORMALIZE/GROUP – here we pass all of the resulting interpretations at once and ask the LLM to normalize the wording and group the clusters along shared characteristics

This turns a long list of clusters into a smaller set of named, coherent groups, with meaningful axes to structure along.

Results

We present the labeling and grouping results as a tree: the top level is the grouping the LLM proposed along one characteristic, and each nested level is a further grouping along the next one. Below, we show the first few groups from two clustering configurations: one that separates document type and topic, and one that separates document type and style.

The colored bars show, for each group, the two most frequent ground-truth classes. These labels were added afterward purely for analysis to show how the naming behaves for clean groups versus mixed ones. The clustering, labeling, and grouping were done fully unsupervised.

A. Cohere v4 (image − text) → Cohere v4 (text) – grouped by document type (level 1) and domain (level 2)

B. Cohere v4 (image − text) → CLIP — grouped by document type (level 1) and visual style (level 2)

We can see that where the clustering worked well, the clusters are neatly named and grouped. The labels get more generic when the clusters are messy, and fewer of them can be grouped successfully. A noticeable share of clusters also ends up ungrouped, since the LLM only groups those that genuinely share a characteristic, but the ones it does group are labeled sensibly.

Imperfect as it is at this stage, such a breakdown gives an immediate, high-level insight into what a given corpus contains. If we were able to estimate which clusters are coherent or mark outliers in each group reliably enough, and then pair it with a human-in-the-loop (or even more powerful LLMs) for review and correction, it could make organizing large repositories of messy documents much easier and faster.

Limitations and further directions

Some things did not fit within the scope (as is expected for a proof-of-concept) and should be improved if this project were continued.

• Data – we used single pages for simplicity. The synthetic corpus was useful for isolating different kinds of structure (semantic, visual, layout), but it is not a sound basis for estimating absolute performance. RVL-CDIP is real but low-quality and not representative of a modern corporate archive. A large, real, labeled corpus would be the right benchmark; however, such corpora are hard to obtain.
• Evaluation – the methods are unsupervised, but the ground-truth labels were used during model selection. We tried to avoid reporting configurations that scored well by chance and focused on findings that were consistent across settings, but what worked on this data may not generalize; a stricter protocol (model selection and hyperparameter tuning on held-out data) should be used in future work.
• Methods – there are plenty of directions to explore further, such as tuning the clustering, estimating sample confidence or cluster coherence, or selecting samples for labeling.

Artificial Intelligence

We deliver AI solutions tailored to your business, driving efficiency and boosting productivity within your teams.

AI offering

Conclusions

We showed that the choice of embedding decides which kind of signal the clustering picks up. An embedding is not necessarily better or worse in general; it just answers different questions about the same corpus. We also explored ways to combine representations so that clustering can surface more than one kind of structure at once.

In addition to that, we propose a quick way to label and organize the clusters in an interpretable way, resulting in a named, grouped structure that a human can easily read. The approach is also cost-effective as the expensive part (the LLM) scales with the number of groups rather than the number of documents.

The approach could already be useful as a discovery tool, revealing themes and structure in large document corpora. The findings would need further verification to be sure they generalize, and making the method robust enough for semi-automatic organization is still ahead. Still, the experiments show promise in recovering meaningful structure from unorganized documents and presenting it in a form a person can actually read.

Sources

Topic models and clustering methods

• LDA – Blei, D. M., Ng, A. Y., Jordan, M. I. (2003). Latent Dirichlet Allocation. JMLR 3, 993–1022. https://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf
• BERTopic – Grootendorst, M. (2022). BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv:2203.05794. https://arxiv.org/abs/2203.05794
• Embedding-based clustering – Sia, S., Dalmia, A., Mielke, S. J. (2020). Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!. EMNLP 2020. arXiv:2004.14914. https://arxiv.org/abs/2004.14914
• UMAP – McInnes, L., Healy, J., Melville, J. (2018). UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv:1802.03426. https://arxiv.org/abs/1802.03426
• HDBSCAN – Campello, R. J. G. B., Moulavi, D., Sander, J. (2013). Density-Based Clustering Based on Hierarchical Density Estimates. PAKDD 2013, 160–172. https://link.springer.com/chapter/10.1007/978-3-642-37456-2_14

Embedding models

• Cohere Embed v4 (text / image, shared embedding space) – Cohere. Embed documentation. https://docs.cohere.com/docs/cohere-embed
• text-embedding-3-small – OpenAI. Embeddings guide. https://platform.openai.com/docs/guides/embeddings
• MiniLM / Sentence-BERT (BERTopic’s default embedding) – Reimers, N., Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv:1908.10084. https://arxiv.org/abs/1908.10084
• CLIP – Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020. https://arxiv.org/abs/2103.00020
• SigLIP – Zhai, X. et al. (2023). Sigmoid Loss for Language Image Pre-Training. arXiv:2303.15343. https://arxiv.org/abs/2303.15343
• DiT – Li, J. et al. (2022). DiT: Self-supervised Pre-training for Document Image Transformer. arXiv:2203.02378. https://arxiv.org/abs/2203.02378
• DINOv2 – Oquab, M. et al. (2023). DINOv2: Learning Robust Visual Features without Supervision. arXiv:2304.07193. https://arxiv.org/abs/2304.07193

Data and tooling

• RVL-CDIP – Harley, A. W., Ufkes, A., Derpanis, K. G. (2015). Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval. ICDAR 2015, 991–995. https://arxiv.org/abs/1502.07058
• Docling (PDF → markdown extraction) – Auer, C. et al. (2024). Docling Technical Report. arXiv:2408.09869. https://arxiv.org/abs/2408.09869