· Index — Research · notes as we go

Malayalam AI,measured. — Research at Araskova Labs: evaluation benchmark, corpus, document AI and on-premise deployment

Sovereign Malayalam AI: the data, benchmarks and models Kerala runs on its own infrastructure. These are our research notes: our own write-ups of work in progress, not papers, and not published or peer-reviewed anywhere. Essays from the lab are in the journal.

BenchmarkCorpusDocument AIOn-premise
Research notes · our own write-ups
01
Research notes · our own write-ups
Active tracks · Malayalam AI
04
Active tracks · Malayalam AI
FAILURE RATE · 4 MODELS · JUNE 2026
~73%
FAILURE RATE · 4 MODELS · JUNE 2026
TOKENS · AS OF JUNE 2026
47.5M
TOKENS · AS OF JUNE 2026

· 001 — The roadmap

AI that understandsMalayalam.

About 38 million people speak Malayalam, and the work of daily life in Kerala (land records, ration cards, school enrolment, local government) happens in it. Today's leading models get most of it wrong. We're building the corpus and the measurement, then an open-weight Malayalam model to fix that.

Read the benchmarkPartner or fund

  1. [01]Complete

    Benchmark

    200 real administrative queries in Malayalam, put to four leading LLMs and scored. Roughly 73% of answers failed.

  2. [02]Active

    Corpus

    47.5M tokens (as of June 2026) of curated Malayalam and English: administrative, educational, legal and conversational, checked by native speakers.

  3. [03]Planned · Q4 2026

    Malayalam LLM / SLM

    Train a small Malayalam language model and fine-tune open models on the corpus with reinforcement learning from feedback, then measure against the same benchmark, before and after.

  4. [04]Next

    Open weights

    Release the model weights openly, with the benchmark methodology and dataset, so anyone can build on Malayalam.

· 002 — The tracks

Four tracks,one language.

Each track with the figure we've measured and the date we measured it, and its research note where one is written up.

  1. [01]Malayalam Evaluation Benchmark

    LLM Evaluation · Administrative Malayalam

    MAL_EVAL_BENCH_01INTERNAL · COMPLETE
    050100FAILURE RATE · %

    ~73%

    FAILURE RATE · 4 MODELS · JUNE 2026

    200 real-world Malayalam administrative queries, scored against four leading models in June 2026. Failure means code-switching, transliteration errors, semantic drift or invented terminology.

  2. [02]Corpus Curation & Licensing

    Language Corpora · Malayalam + English

    MAL_CORPUS_LIC_02ACTIVE
    0102030405047.5M · ML + ENM TOKENS

    47.5M

    TOKENS · AS OF JUNE 2026

    A curated Malayalam and English corpus with the source and licence of every document recorded, so institutions can use models trained on it. Details are on the corpus datasheet.

  3. [03]Malayalam Document AI

    OCR · Printed, Handwritten & Old Script

    MAL_DOC_OCR_03ACTIVE
    അPRINTEDകHANDWRITTENൿOLD SCRIPT

    OCR

    PRINTED · HANDWRITTEN · OLD SCRIPT

    OCR and extraction for printed, handwritten and old-script (pazhaya lipi) Malayalam: forms, registers and records turned into structured data, on-premise.

  4. [04]On-Premise Deployment

    Quantized Inference · Institution Hardware

    ONPREM_INFER_04ACTIVE
    INSTITUTION NETWORKCLOUDWORKSTATIONSERVER

    On-prem

    BY DESIGN

    Quantized models and document pipelines sized to run on the servers and workstations an institution already owns, with its data kept on its own network.

· 003 — Benchmark

How leading modelsfail on Malayalam.

The benchmark measures whether a model gives accurate, contextually correct Malayalam on the tasks of daily administration in Kerala.

An answer fails if it shows any of

  1. [01]Code-switchingReverts to English mid-sentence for technical or administrative terms.
  2. [02]Transliteration errorsPhonetically incorrect Malayalam rendering.
  3. [03]Semantic driftValid Malayalam that loses a deadline, a form field or an eligibility rule.
  4. [04]Invented terminologyMalayalam equivalents for terms that have no accepted translation.

MAL_EVAL_BENCH_01 · the run

Queries
200 real-world Malayalam administrative queries
Models evaluated
GPT-4o · Gemini 1.5 Pro · Claude 3 Opus · Llama 3 70B
Run
Internally by Araskova Labs, June 2026
Result
~73% of answers failed, across all four models
Dataset
Complete and scored (internal)
Public release
Not yet: methodology and dataset to be published with the corpus
Read the benchmark note

· 004 — Corpus datasheet

What's inthe corpus.

A datasheet for the corpus behind our benchmark and models: where the text comes from, under what licence, and what it is (and isn't) suitable for.

Total tokens
47.5M · as of June 2026
Domains
Administrative, educational, legal and conversational text
Review
Checked by native Malayalam speakers
Malayalam / English split
In preparation
Sources & licences
In preparationEvery document's source and licence is recorded
Deduplication & filtering
In preparation
Intended use & known gaps
In preparation

· 005 — The notes

Research notesnewest first.

Our own write-ups of work in progress. Not papers, and not published or peer-reviewed anywhere.

  1. NOTE-01· Latest[cs.CL] · NLP Research28 Jun 2026 · 2 min readBy Anvin P Shibu

    Malayalam LLM Benchmark: Failure Analysis on Administrative Queries

    An internal benchmark of leading LLMs on real-world Malayalam administrative queries reveals a ~73% failure rate — validating our regional data engine thesis.

    Read the note→Engineering context ↗

· 006 — For research partners and institutions

Build Malayalam AIwith us.

We want to work with universities, research groups and public institutions on Malayalam evaluation, corpora and document AI: replicating the benchmark, contributing or licensing data, or piloting document AI on your own servers.

Write to the labPartner with us
Or join the research fellowship →