convaiinnovations

/

laya

Like 5.89k

Hi! Thanks for releasing Laya. I built LayaMacro, which grafts a frozen image encoder (SigLIP 2-large) onto Laya so it answers typed questions about a photo, and wanted to share it with you and the community.

How it works

• 576 SigLIP 2 patch tokens → a small projector → Laya's input, placed after the question and options, so the [MASK] marker positions never move.
• Every answer is still a Laya typed decision. Nutrition is five score questions over 18 value ranges each (kcal, protein, carbs, fat, grams), giving calibrated 80 % ranges. Food questions are choice: "Does this meal contain rice?", "Which of these foods is on the plate?"
• ModernBERT backbone frozen; I trained the projector and Laya's decision head (28.6M parameters), 10 epochs on 2 GPUs.

Results (test photos)

• Calories: 48.5 kcal mean absolute error on Nutrition5k. A dedicated method, OmniFood8K, is clearly better there (36.8).
• Food questions: 0.89 on "contains X?" and 0.86 on "which of 5 foods?" (chance 0.56 / 0.20). A wording never used in training works as well (0.88). Foods never trained on: only 0.65.
• Caveat: training is seed-sensitive. A second seed did not learn the food questions.

Two notes from integrating (as of laya 0.3.20), in case they're useful

• DecisionModel.forward doesn't take inputs_embeds (the backbone does), so I reserve placeholder slots through build_sequence and overwrite their embeddings. That keeps your marker_pos and masks as the source of truth.
• laya-typed-decisions ships a choice:11+ temperature of ≈0.10, which the loader clamps to 0.5 with a warning.

Code (Apache-2.0): https://github.com/Aquos06/LayaMacro
Weights (CC BY-NC-SA 4.0, non-commercial; a research model, not a medical device): https://huggingface.co/Aquos06/LayaMacro

I'm not affiliated with you. Happy to answer questions, or to hear if I've misused anything.