NEDOQwen: An Auditable Low-Cost Diagnosis-and-Repair Study for a Turkish-Centric 0.824B Language Model

Author abstract

Abstract

Small language models can make non-English model development more accessible, but internal prompt checks and language-specific design choices may obscure substantial external failures. We present NEDOQwen, a Turkish-centric 824M-parameter decoder-only research model, as an auditable case study in low-cost diagnosis and targeted repair. Clean supervised fine-tuning improves internal Turkish instruction diagnostics, while public Turkish and Turkic multiple-choice slices reveal weak academic knowledge, tokenizer coverage limitations, and severe answer-label bias. A 600-step repair stage increases TurkishMMLU-sub accuracy from 18.11% to 21.00% and TUMLU-mini Turkish accuracy from 21.33% to 32.22% under a common zero-shot answer-label log-probability protocol. An answer-label-only ablation reaches 19.56% and 27.00%, showing that calibration explains part, but not all, of the observed gains. Prediction distributions remain strongly concentrated on C and D, and qualitative failures persist. The contribution is therefore not a state-of-the-art Turkish model, but a version-aware, low-cost workflow for exposing internal–external evaluation mismatch, auditing tokenizer behavior, and reporting partial repair without mistaking benchmark gains for broad competence.

Research question

What does a small model’s internal progress hide?

NEDOQwen asks whether apparently improved Turkish instruction behavior survives external evaluation, and whether a tightly scoped repair stage can correct the failures without overstating general model competence. The study combines benchmark accuracy, prediction-label distributions, tokenizer diagnostics, qualitative checks, and exact artifact provenance.

Results

Key findings

Experimental design

Method at a glance

Model

824,256,000-parameter decoder-only causal language model

Architecture

24 layers, 1536 hidden size, grouped-query attention, RoPE, RMSNorm, and SwiGLU

Tokenizer

65,536-token Turkish-aware typed-surface vocabulary

Context

1,024 tokens

Evaluation

900-item TurkishMMLU-sub and 900-item TUMLU-mini Turkish slices

Protocol

Zero-shot answer-label log-probability scoring under a common evaluation harness

Repair

600 steps over 2,005 targeted repair rows mixed with 2,000 clean instruction rows

Compute

Approximately 0.465 GPU-hours for the targeted repair stage

Artifact provenance

Public releases and evaluated checkpoints are versioned separately

The public Hugging Face base and SFT repositories belong to the stable step-5000 lineage. The report’s principal benchmark results come from a later experimental step-9000 clean-SFT checkpoint, its answer-label-only ablation, and the full repair-v2-600 checkpoint. The public models are useful research artifacts, but they should not be treated as the exact source of the reported repair results.

Version-aware reporting prevents a public checkpoint from being mistaken for a different evaluated model merely because both share the NEDOQwen name.

Open research

Code, models, and datasets

Artifact status: The GitHub repository contains training, encoding, SFT, sampling, and inspection code. Hugging Face hosts the large checkpoints and datasets. The public models are from the step-5000 lineage; exact experimental repair checkpoints are not represented by these two model pages.

Scope

Limitations

• The study does not include blinded human evaluation.
• The reported gains do not establish broad Turkish-language competence or state-of-the-art performance.
• Prediction-label skew remains substantial after repair.
• Tokenizer compactness and unknown-token measurements are diagnostics, not evidence of a causal effect on benchmark accuracy.
• The two current public model repositories do not contain the exact evaluated step-9000 and repair-600 checkpoints.

Compute support

Acknowledgments

The authors acknowledge TÜBİTAK ULAKBİM High Performance and Grid Computing Center (TRUBA) for providing computational resources used in this study. This support was provided in kind, and no grant or award number was issued.

Reuse and citation

Cite this preprint

Öztürk, A. R., Dalar, Y. E., Aksoy, Ö. F., Sezer, N. M., & Salihoğlu, F. A. (2026). NEDOQwen: An Auditable Low-Cost Diagnosis-and-Repair Study for a Turkish-Centric 0.824B Language Model. Research Square. https://doi.org/10.21203/rs.3.rs-10384535/v1

Download the BibTeX citation

@article{ozturk2026nedoqwen,
  title       = {NEDOQwen: An Auditable Low-Cost Diagnosis-and-Repair Study
                 for a Turkish-Centric 0.824B Language Model},
  author      = {Öztürk, Ahmet Rıfat and Dalar, Yağız Ekrem and
                 Aksoy, Ömer Faruk and Sezer, Nedim Mutlu and
                 Salihoğlu, Feyzi Arda},
  year        = {2026},
  doi         = {10.21203/rs.3.rs-10384535/v1},
  url         = {https://doi.org/10.21203/rs.3.rs-10384535/v1},
  note        = {Preprint}
}