This post was co-authored with Max Silfverberg (Data Scientist, AI Solutions Lead), Antti Hallavo (Lead AI Software Engineer), and Pontus Huotari (Lead Data Scientist). We work at Alma Media, a Finnish digital services, marketplaces and media company. One of our focus areas is developing AI/ML solutions for real estate listing services, where understanding image content plays an important role.
services handle hundreds of thousands of listings a year. Most of those come with dozens of photos with no information about what they show. Meanwhile, search, recommendations, and a range of internal use cases all benefit from knowing whether a photo represents a kitchen, floor plan, or garden.
Our solution is to automatically tag photos with room-type and content classes. Our room types include LIVING ROOM, KITCHEN, and BEDROOM. We also tag schematic content like floor plans and site plans. Additionally, we recognize realtor marketing materials, aerial shots, and garden photos. Altogether, there are 23 classes. As Figure 1 shows, this is a classic multi-label classification task; the same space can encompass several room types at once.
On the face of it, this sounds simple, but we need to make some tricky decisions. How should you treat a living room photo that shows a bedroom through a doorway? What if the photo only shows 10% living room and the remaining 90% is dining area? The answers depend on the application.
If we need to find all photos showing kitchens, we also want to identify living room photos that show a kitchen in the background. However, if the user specifically asks for kitchen photos, we only want to show the ones where the kitchen is in focus. To help decide what to return, classification confidence is important. But depending on how you implement your classifier, you might not have access to that information.
Image classifiers can be built in many ways. The modern default approach is to run images through a third-party API which internally uses a vision-language model (VLM) to analyze images and generate tags according to a prompt.
Another option is to train image classifiers on top of open-source ViT foundation models like Google SigLIP and Meta DINO, either freezing the foundation model or fine-tuning it. Each of these designs comes with its own advantages and trade-offs.
There already exists plenty of work comparing the approaches based on numerical performance [1]. This blog post goes further; we ask the commonly overlooked question: How should you build image classifiers in a business context?
Contents
Three questions before you train anything
We built our proprietary classifiers by fine-tuning google/siglip-base-patch16-224. The question is: why do this? Check Figure 2 for the TL;DR. Read on for the full story.
Question 1: Prompt an API or train your own model?
The choice to classify by prompting through an external API or build your own classifier heavily depends on your use case. First, you need to consider whether your classification task can even be prompted. It’s easy to prompt car and kitchen appliance classifiers but how about click-through rate (CTR) for YouTube video thumbnails? Here we need a trainable classifier because we really don’t know what influences the click decision. Conversely, if you need to extract structured JSON files from photos representing building schematics, a simple classifier just won’t cut it.
Our real estate use case sits in the middle. Many classes like KITCHEN and BATHROOM are easily promptable while others, like HALLWAY, LOFT and ALCOVE are fuzzier and harder to verbalize.
If you decide to train your own classifiers as we did, you of course need training data, probably at least a few thousand examples per class. When launching a new product, that is something you might not have. If you at least have access to plain photos without annotations,you can launch with aprompted VLM as your first classifier. Its predictions gradually accumulate into an annotated dataset, which you can later use to train a custom classifier. This may require a cleanup pass, since the dataset inherits the VLM’s mistakes.
Cost is another major question. With a volume in the millions, the different classification approaches result in dramatically divergent cost profiles. Using Google’s Agent Platform and the gemini-3.5-flash model, the July 2026 price is roughly $1.50 per 1,000 images (at 1K resolution), so classifying a million photos costs roughly $1,500.
Using our own classifier on a dedicated AWS EC2 g4dn.xlarge instance with a T4 GPU, we can classify at least 400 images per second. At a July 2026 on-demand hourly rate of $0.53, classifying a million inputs comes out to $0.37 or roughly 1/4000th of the cost for the API solution (inference compute only).
Nevertheless, if you classify a few hundred photos a day, from the cost perspective it really doesn’t matter how you do it. Costs become an issue only at scale.
In addition to labels, classification confidence is often useful. As mentioned above, if we offer kitchen photos to the user, we should probably go with confident matches. It is, however, tricky to derive reliable confidence estimates from a VLM; verbalized confidence estimates are known to be poorly calibrated [2] and token log-likelihoods from an API frequently don’t represent the class-probabilities you are actually interested in.
If you use an API, you might therefore need to rely on granular tags like PROBABLE/POSSIBLE/UNLIKELY [3], and there is no guarantee that those will be reliable either. If you instead train your own classifier, you get usable per-class scores which can be calibrated when needed. Table 1 summarizes how the two approaches compare.
| Prompted VLM (API) | Custom classifier | |
| Training data | None needed | A few 1000 examples per class |
| Setup effort | Write a prompt | annotate, train, deploy |
| Cost per 1M photos | ~$1,500 | ~$0.37 (on GPU) |
| Per-class scores | Unreliable / not exposed | Explicit, thresholdable, calibratable |
| Fuzzy classes | Hard to verbalize in prompt | Learnable from examples |
| Changing the task | Edit the prompt | Retrain the model |
Question 2: Which foundation model to use?
If you decide to train your own classifier, the only reasonable choice for most is to start with a pretrained open-source vision model, typically a vision transformer. For business use, first check that the model’s license permits commercial use.
Beyond that, your business goal should drive the choice, because different pretraining strategies produce different representations:
- SigLIP [4] (Google) is trained on captioned images, so it attends to caption-worthy things: dogs, cars, people. Its representations are highly object-oriented; background and camera angle receive far less emphasis.
- DINO [5, 6] (Meta) is self-supervised with patch-level objectives: every region of the image contributes to the loss, not just the caption-worthy objects. That makes it a strong candidate when background or layout matters [7]. We put this to the test below.
- RADIO / AM-RADIO [8] (NVIDIA) agglomerate representations from several ViT foundation models through distillation.
- I-JEPA[9] Meta) is self-supervised like DINO but based on masked prediction.
Question 3: To fine-tune or not to fine-tune?
The simplest way to start is training a linear classifier on top of frozen ViT representations. There are two major advantages: it is conceptually straightforward and lightning fast. You can train on a laptop in a matter of minutes using a 100k-instance training set. Typically, this leads to very reasonable performance.
If you decide to fine-tune, the best practice is to use low-rank adapters (LoRA), which freeze the actual ViT backbone and inject a few thin trainable parameter layers into the model [10]. After training, these can be merged with the original model to avoid costs at inference time. LoRA keeps training tractable even on a modest GPU setup, while delivering nearly the same performance gain as full fine-tuning.
Since a shallow linear classifier usually performs well, fine-tuning can result in modest gains in terms of raw F1 score. However, under-labeling can be a real problem when you freeze your foundation model as we see below.
Simple has a price tag
The major problem with the frozen model is low classification confidence. At a standard 0.5 operating threshold, a whopping 35% of photos receive no labels from the model. Tuning down the threshold helps, but it comes at the cost of lower precision. Per-class thresholds might help, but downstream applications need scores that mean the same thing across all 23 classes, and class-specific thresholds would drift with every retraining.
In practice, we settled on a compromise of 0.2, which provides reasonable coverage and precision. Figure 3 illustrates what this looks like for a single photo: at t = 0.5 nothing clears the bar, while at t = 0.2 the two correct labels come through.
A secondary problem is poor classification on a few frequent classes like GARDEN and HALLWAY. Edge cases also cause problems: when a dining set is visible in a living room image, we would like to label it both LIVING ROOM and DINING AREA. However, when the dining set is visible only through a doorway, we don’t want the DINING AREA label.
These problems can be addressed by LoRA fine-tuning.
Putting it to the test
We decided to train our own classifier and compared the two custom approaches outlined above: a frozen foundation model combined with a shallow linear classifier, and fine-tuning with LoRA. In both cases, we added 23 independent classification heads on top of the foundation model, one per class.
The input is an image vector generated by SigLIP. We additionally experiment with DINOv2 as a frozen baseline to see how caption-training compares to self-supervised training. LoRA fine-tuning is done exclusively on SigLIP. We used the original SigLIP model rather than SigLIP 2 in these experiments; since we compare a frozen setup against fine-tuning on the same backbone, the conclusions don’t hinge on the model generation.
For evaluation, we use micro averaged F1 score. This emphasizes performance on common classes like KITCHEN and LIVING ROOM, which are most central for our use cases.
Additionally, we evaluate coverage on the test set: how many of the photos get at least one label? While there is a natural residual of inputs that don’t fall into any of the 23 classes, we want to find all the photos that can be labeled.
Training
We train our classifiers on our own proprietary set of 40k manually annotated photos, where each input gets 1-3 class labels. Our validation data has 1.9k examples; we split this into 100 development and 1.8k test examples. Training, development and test photos come from distinct listings, so photos of the same property never appear in more than one split.
For both our frozen baselines, we trained 23 separate sklearn LogisticRegression models.
We trained LoRA using the PEFT library. Following common practice [10], we wrapped the SigLIP ViT self-attention query and value layers in LoRA adapters, leaving the MLP layers untouched, and used BCE loss on top of 23 independent logistic classification heads. This meant training only about 0.6% of the model’s parameters, roughly a 99% reduction compared to full fine-tuning. It is also why the whole sweep fits on a single T4.
We did a random 40-trial hyperparameter sweep [11] over the configurations in Table 2, fixing all other hyperparameters to standard values.
| Hyperparameter | Range | Distribution |
lr |
1e-5 -> 1e-3 |
log-uniform |
batch_size |
{16, 32, 64} |
uniform categorical |
lora_r |
{8, 16, 32} |
uniform categorical (lora_alpha locked to lora_r) |
For fast and numerically safer training, we used mixed precision with fp16 autocast and loss scaling [12]. We trained for 20 epochs and picked the model that delivers the best F1 score on the development set.
All training is done on an AWS EC2 g4dn.xlarge instance with a single NVIDIA T4 having 16 GB VRAM.
Evaluation
In terms of plain micro averaged F1, differences are modest. At 82.6% F1, the fine-tuned model beats both frozen SigLIP’s 78.4% F1 and frozen DINOv2’s 78.3% F1, but the difference is only around 4 points. The frozen SigLIP and DINOv2 classifiers deliver essentially identical performance. Frozen models are reported at their best dev-set thresholds (0.2 for SigLIP, 0.35 for DINOv2); the fine-tuned model at its default threshold of 0.5, which marginally understates its best achievable F1 (83.1%). Table 3 shows the full results.
| Metric | Frozen SigLIP (t = 0.2) | Frozen DINOv2 (t = 0.35) | LoRA SigLIP (t = 0.5) |
| Micro F1 | 78.4 | 78.3 | 82.6 |
| Micro precision | 85.1 | 85.1 | 86.2 |
| Micro recall | 72.8 | 72.4 | 79.3 |
| Unlabeled photos | 9.4% | 10.8% | 3.4% |
The rise in F1 score is basically due to recall, which improves by roughly 7 points from 72.8% (SigLIP) and 72.4% (DINOv2) to 79.3%. At 85.1%, the frozen models’ precision is already very high, and it only improves by about 1 point.
As Figure 4 shows, these results are not an artifact of the operating threshold; the fine-tuned classifier outperforms the frozen SigLIP classifier at every operating threshold, showing that fine-tuning does not merely push confidence up but genuinely improves classification performance. With only 100 development examples, we treat the selected thresholds and stopping epoch as coarse choices rather than highly tuned optima.
The modest gains in micro averaged F1 hide substantial improvements for individual classes, especially for GARDEN (support in test set: 237) with an impressive 26-point rise compared to frozen SigLIP, and DINING AREA (support in test set: 150) with a respectable 15-point improvement.
The GARDEN class is a particularly interesting example, because it is typically all background, something that SigLIP does not do well off the shelf, as discussed above. In such cases, fine-tuning can deliver dramatic improvements.
However, when we look at performance for the GARDEN class using the frozen DINOv2 model, a different pattern emerges: frozen DINOv2 F1 score is 58%, a 15-point improvement over the frozen SigLIP model. Just by choosing a more suitable foundation model, we have gained more than half of the performance gap compared to a fine-tuned SigLIP model. DINING AREA also shows an improvement of 5 points F1 score.
At the same time, DINOv2 underperforms compared to SigLIP on many classes where semantic understanding of the photo seems more important: it never predicts MARKETING (support in test set: 24), and KITCHEN (support in test set: 272) slips 7 points. Interestingly, performance also degrades on HALLWAY (support in test set: 86), a fundamentally architectural category where we would have expected DINOv2 to excel.
The only class where fine-tuning degrades performance is KITCHEN: an F1 drop of 5 points. We consider this minor, but this is naturally case dependent.
The real selling point for LoRA fine-tuning is that it largely solves under-labeling. The frozen SigLIP classifier leaves 9.4% of photos without labels and DINOv2 does even worse at 10.8%. SigLIP’s rate is 2.4x the natural rate (3.9%) of photos that genuinely fall into none of our 23 classes. LoRA ends up at 3.4%, slightly below the natural rate, meaning it instead very occasionally over-labels.
Figure 5 shows that the under-labeling rate of the fine-tuned classifier remains low for reasonable operating thresholds. We can trade a bit of recall for even higher precision. In contrast, the under-labeling rate of the frozen classifier shoots toward the sky if one tries to sharpen precision by raising the operating threshold. As a classifier, it is therefore far less flexible than the fine-tuned one.
So, when should you fine-tune?
There are a few things worth considering. If under-labeling is a problem for you, then fine-tuning can be worth it. The problem essentially disappeared in our case.
For individual classes, we did see large gains, specifically in recall. GARDEN and DINING AREA are now recognized far more often. However, just choosing an appropriate foundation model (DINOv2 rather than SigLIP) recovered more than half of the GARDEN gap without fine-tuning. Nevertheless, we did not observe degradation of precision, so gains are genuine albeit modest in terms of raw F1.
With a training set of 40k examples, the cost for a full hyperparameter sweep turned out to be around $30, which is negligible. Still, if under-labeling is not an issue, you might prefer to use a frozen backbone, especially when periodic retraining is needed. Training 23 classification heads on a CPU takes minutes and needs no GPU; a full LoRA sweep takes days on a dedicated GPU instance, and that cost repeats every time you retrain.
The lowest-effort option would be VLM-based classification, but at high volumes that becomes a significant recurring cost. The difference between $1,500 and $0.37 for a million inputs adds up quickly. However, bear in mind that training your own classifier requires annotated data. We use 40k manually annotated photos. That is not free either.
For us, the investment has already paid off. The fine-tuned classifier now runs in production, and the labeling quality is good enough that Alma has built new functionality on top of it. Because the labels are produced automatically, they are available at scale for downstream applications to build on.
Conclusion
Whichever approach you choose, image labeling pays off across the real estate listing service: search results, recommendations, and a range of internal use cases all improve. If you’re unsure whether it’s worth it, start small and prompt a VLM to classify a subset of your data. From there, a lightweight classification head on top of an existing embedding model will cut your costs, and if you need more accuracy, fine-tuning your own model is the natural final step.
References
[1] N. Kisel, I. Volkov, K. Janouskova and J. Matas, Multimodal large language models as image classifiers (2026), arXiv:2603.06578
[2] M. Xiong, Z. Hu, X. Lu, Y. Li, J. Fu, J. He and B. Hooi, Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs (2024), International Conference on Learning Representations (ICLR)
[3] S. Lin, J. Hilton and O. Evans, Teaching models to express their uncertainty in words (2022), Transactions on Machine Learning Research
[4] X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer, Sigmoid loss for language image pre-training (2023), IEEE/CVF International Conference on Computer Vision (ICCV)
[5] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski and A. Joulin, Emerging properties in self-supervised vision transformers (2021), IEEE/CVF International Conference on Computer Vision (ICCV)
[6] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, et al., DINOv2: Learning robust visual features without supervision (2024), Transactions on Machine Learning Research
[7] M. El Banani, A. Raj, K.-K. Maninis, A. Kar, Y. Li, M. Rubinstein, D. Sun, L. Guibas, J. Johnson and V. Jampani, Probing the 3D awareness of visual foundation models (2024), IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
[8] M. Ranzinger, G. Heinrich, J. Kautz and P. Molchanov, AM-RADIO: Agglomerative vision foundation model reduce all domains into one (2024), IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
[9] M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun and N. Ballas, Self-supervised learning from images with a joint-embedding predictive architecture (2023), IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
[10] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang and W. Chen, LoRA: Low-rank adaptation of large language models (2022), International Conference on Learning Representations (ICLR)
[11] J. Bergstra and Y. Bengio, Random search for hyper-parameter optimization (2012), Journal of Machine Learning Research, 13
[12] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh and H. Wu, Mixed precision training (2018), International Conference on Learning Representations (ICLR)