Lê Chí Đại
← Projects
Deep learning · Model training & adaptation · 2026

Deep Learning — Vision, Language & Multimodal

Training and adapting deep-learning models across three classification tasks: animal images, real-estate descriptions, and paired movie posters and text. The work spans data pipelines, classifier design, full and parameter-efficient fine-tuning, class imbalance, and analysis of model errors and computational cost.

RoleTeam member · ML4U · Team of 4
Year2026
ToolsPyTorch · timm · Transformers · LoRA · Weights & Biases

Project visuals

Figures from project reports, pitch decks, and saved experiment outputs. Select an image to open it at full resolution.

The image-classification report with augmentation filters, a model comparison table, and confusion matrices.
Animals-10: eight training configurations compare CNN and vision-Transformer backbones, with and without augmentation. Source ↗
The text-classification report with preprocessing and setup filters, sortable results, and confusion matrices.
Oregon real-estate classification: thirteen configurations vary preprocessing, back-translation, backbone freezing, and full fine-tuning. Source ↗
The multimodal report showing attention-rollout visualizations for movie posters.
MM-IMDB: attention-rollout analysis of the CLIP vision encoder, comparing mean fusion with max fusion and attention discard. Source ↗

Problem and experimental scope

The project investigates how pretrained architectures and adaptation strategies behave across image, language, and multimodal data. Each task includes exploratory data analysis, train/validation/test preparation, model training, and evaluation. The comparisons go beyond architecture choice to test augmentation, preprocessing, fine-tuning strategy, and the trade-off between predictive quality and model cost.

Vision: CNNs vs. vision Transformers

For Animals-10, the team trained eight configurations: ResNet18, EfficientNet-B0, ViT-B/16, and Swin-T, each with and without augmentation. The pipeline uses PyTorch and timm, backbone-specific normalization, a shared Linear–BatchNorm–ReLU–Dropout–Linear classifier head, and a 70/15/15 data split. Training is tracked in Weights & Biases. Evaluation combines accuracy and macro F1 with parameter counts, FLOPs, model size, confusion matrices, and GradCAM/EigenCAM analysis of difficult, out-of-distribution, and potentially mislabeled inputs.

Language: preprocessing, imbalance, and fine-tuning

The Oregon real-estate task predicts six property types from listing descriptions. Thirteen BiLSTM, BERT, and DistilBERT configurations compare minimal versus aggressive text cleaning, standard versus back-translated training data, and frozen versus fully fine-tuned Transformer backbones. Back-translation augments minority classes; an 80/10/10 split preserves class proportions. Accuracy, macro F1, and weighted F1 are evaluated alongside training time, test time, and model size, with example-level analysis of ambiguous descriptions and suspected label errors.

Multimodal: CLIP fusion and few-shot LoRA

The MM-IMDB task predicts 23 movie genres from paired poster and text inputs. The team compares a zero-shot CLIP baseline with few-shot adaptation at K=16, 32, and 64. The adaptation architecture uses CLIP ViT-B/32, rank-8 LoRA adapters, and a late-fusion classifier: independently L2-normalized 512-dimensional image and text embeddings are concatenated into a 1024-dimensional representation. Weighted BCEWithLogitsLoss addresses the multi-label task and rare genres. Attention rollout compares mean-fused attention with max fusion and attention discard to inspect the vision encoder’s behavior.

Results on the reported test splits

  • Animals-10: Swin-T with augmentation reached 98.32% accuracy. EfficientNet-B0 without augmentation reached 98.27% at 0.39 GFLOPs versus Swin-T’s 4.37 GFLOPs—a concrete accuracy/compute trade-off.
  • Oregon listings: fully fine-tuned BERT with minimal preprocessing and augmented data reached 90.94% accuracy and 80.88% macro F1. The best reported BiLSTM macro F1 was 69.08%.
  • MM-IMDB: the K=64 CLIP-fusion/LoRA configuration reported 48.98% macro F1 and 60.09% weighted F1, compared with 5.63% and 28.42% for the zero-shot baseline.

Team and artifacts

Built for CO3133 by ML4U: Đặng Vũ Tuấn Kiệt, Lê Chí Đại, Nguyễn Quốc Huy, and Lương Minh Thuận. The team publishes the experimental methods, training curves, model comparisons, and error analyses in three technical reports, with linked Gradio inference demos for image, text, and multimodal classification.

Next project

gradphic