ICSETEP RDG sub-project · 2026–2027

প্রজ্ঞা progya — discernment, applied wisdom

A unified AI-powered national competency framework for 21st-century skills

Every year roughly one million higher-secondary graduates sit university admission tests and about 400,000 graduates prepare for the Bangladesh Civil Service preliminary examination, of whom near 3% advance. Progya asks a research question hidden inside that statistic: can the preparation itself — not the exam — be turned into a measured, personalised, nationally observable skills development process?

Principal Investigator Professor Dr. Mohammad Saifur Rahman Department of CSE, BUET

Co-Principal Investigator I Professor Dr. M. Sohel Rahman Department of CSE, BUET

Co-Principal Investigator II Associate Professor Dr. Md. Shariful Islam Bhuyan Department of CSE, BUET

Funded under the ICSETEP Research & Development Grant (RDG) · Improving Computer and Software Engineering Tertiary Education Project, UGC Bangladesh

Full title: Progya (প্রজ্ঞা): Developing a Unified AI-Powered National Competency Framework Integrating Psychometric Modeling, Domain-adapted LLM and Personalized Learning Pathways for 21st-Century Skills Assessment and Predictive Policy Analytics

Project PIN C-57

Duration: 24 monthsJan 2026 – Dec 2027
Learner model: BKTInterpretable by design
Generation: Progya-QGDomain-adapted LLM, HITL
Pilot cohort: 200+CAT and BCS tracks
Question bank: 10,000+Expert-validated items

Assessment at national scale produces enormous effort and almost no measurement

High-stakes public examinations in Bangladesh function as selection mechanisms. The effort poured into preparing for them is largely rote, largely unmeasured, and — for the overwhelming majority who do not advance — largely discarded. Simultaneously, the performance data generated by hundreds of thousands of examinees each cycle is the highest-resolution picture of national human capital that exists, and it is never analysed as such.

Progya treats this as a socio-technical systems problem with three coupled research components: a formal competency ontology, an interpretable learner model that drives personalised practice, and a privacy-preserving method for validating that model against real population outcomes.

The scientific claim is not that any one of these components is new. It is that no one has integrated all three into a single functioning system at national scale, and that doing so in a non-English, resource-constrained, high-stakes setting surfaces problems the existing literature has not had to solve.

Research gap

Learner modelling, AI content generation, and learning analytics for policy are each mature fields. Their synthesis into one cohesive system — validated at population scale, in a developing-nation institutional context — is essentially unstudied.

Progya is designed so that its failure modes are informative. If the calibration loop shows no correlation between platform-estimated mastery and real exam performance, that is a publishable negative result about the transfer validity of adaptive practice systems.

Three literatures, and where this project departs from each

For every strand we adopt the established result and state explicitly where we diverge, and why the divergence is a design decision rather than a shortcut.

Learner modelling and adaptive pathways

Corbett & Anderson 1995 · Piech et al. 2015 · Shin et al. 2021 (SAINT+)

Bayesian Knowledge Tracing models mastery of a skill as a latent probability updated from observed responses. Deep Knowledge Tracing and Transformer variants beat it on predictive accuracy by capturing long-range dependencies across a learner's interaction sequence.

Our departure We deliberately adopt BKT, not DKT. In a public-facing national system, the ability to explain why a learner was shown a particular item outweighs a marginal AUC gain. Fairness and transparency are functional requirements here, not afterthoughts — accuracy traded for accountability is a defensible research position, and we intend to quantify the size of that trade.

AI-driven content generation for assessment

Raffel et al. 2020 (T5) · Amershi et al. 2014 (HITL)

Question generation has moved from templates to fine-tuned pre-trained language models. The consistent finding across the literature is that fully automated generation is not yet reliable enough for high-stakes assessment; the strongest systems keep subject-matter experts in the loop.

Our departure We fine-tune a compact open-weight model (Qwen3-4B class) with PEFT/LoRA on a curated corpus of the Bangladeshi higher-secondary syllabus and civil service competencies. The contribution is a blueprint for localised, high-fidelity assessment models in a non-English high-stakes context — a setting the English-language QG literature barely covers.

Learning analytics for educational policy

Siemens & Gašević 2012 · Williamson 2016

Learning analytics has largely operated at course and institution level. The emerging frontier is educational data mining for governance — using large-scale data to identify systemic inequity and guide curriculum reform — which raises immediate and unresolved privacy questions.

Our departure The System-Level Calibration Loop is our methodological contribution: aggregated, anonymised population outcomes calibrate the model without any individual score ever entering the system. Open-loop for the individual, closed-loop for the system.

An ontology, a generator, and a learner model that share one vocabulary

Every component addresses the same machine-readable skill URIs. That shared vocabulary is what makes the system analysable end to end — from a single response to a national skills map.

Fig. 1 — Hub-and-spoke competency ontology. The hub holds cross-cutting foundational skills; spokes hold examination-specific knowledge. Every terminal skill carries a unique machine-readable URI, giving the generator, the learner model and the analytics layer one shared ground truth.

The ontology is defined by expert panels convened with the relevant education and recruitment bodies, resolved down to Level-4 granularity, and then frozen as an addressable vocabulary. Without this step neither the generator nor the learner model has anything stable to point at.

It is also the layer that makes the system portable. A framework expressed as URIs rather than as a curriculum document can be adopted by a curriculum authority, an accreditation body, or a skills certification agency without renegotiating its structure each time.

Item tagging target: 50–100 validated items per Level-4 skill.

The human-in-the-loop generation pipeline

A closed cycle in which expert judgement is the training signal, not a post-hoc filter.

STAGE 1

Corpus curation

Source syllabus material and recruitment regulations are digitised and semantically structured against the ontology.

STAGE 2

Gold standard set

A general-purpose model drafts instruction–output pairs; expert panels correct and validate to yield ≥1,000 gold examples.

STAGE 3

PEFT / LoRA fine-tune

A compact open-weight model is adapted on the gold set, producing Progya-QG — a precision instrument, not a chatbot.

STAGE 4

Iterative refinement

Generated items return to the moderation interface; each expert decision becomes new training signal. The loop tightens.

Mastery as a tracked probability, calibrated against reality without touching a single score

Fig. 2 — The System-Level Calibration Loop. After each examination cycle, anonymised aggregate outcomes are compared against the platform's predicted mastery distribution. Discrepancies correct item difficulty and BKT parameters. No individual result is ever linked back to a user — the privacy-preserving property is architectural, not procedural.

Four sequential phases, each gated on a verifiable artefact

Phases are strictly ordered: the ontology must be frozen before generation can be tagged, generation must produce a bank before the adaptive engine has anything to select from, and the platform must run a pilot before any calibration data exists.

01Months 1–6

Formalise the foundational framework

Data-sharing protocols and partnership terms formally agreed with education and recruitment bodies
Expert panels convened with defined Terms of Reference
Hub of cross-cutting skills defined and validated by both panels
CAT and BCS spoke frameworks completed to Level 4, each skill assigned a unique URI

02Months 7–12

AI development and content generation

Gold-standard dataset of ≥1,000 expert-validated instruction–output pairs completed
Progya-QG v1.0 trained, meeting a target quality score on a held-out test set
Human-in-the-loop moderation interface deployed and in expert use
Validated question bank reaches 5,000+ items, each tagged to a granular skill

03Months 13–18

Platform development and pilot launch

MVP deployed with authentication, dashboards and the assessment interface
BKT adaptive engine integrated and driving item selection
Learning Record Store operational; xAPI statements verified for all key interactions
Pilot launched with a minimum of 200 recruited volunteers across both tracks

04Months 19–24

Data analysis and dissemination

First anonymised aggregate summative dataset ingested and processed
First System-Level Calibration Report completed and shared with partners
At least two papers submitted to peer-reviewed venues
Project Completion Report submitted

What exists at the end, and how we will know whether it worked

Artefact
Validated competency framework

A machine-readable hub-and-spoke ontology covering tertiary admission and public service competencies, validated by expert panels.
Model
Progya-QG

A domain-adapted language model specialised for assessment item generation in Bengali and English against the national curriculum.
Dataset
National question bank

10,000+ expert-validated items, each tagged to a specific Level-4 competency URI.
System
Adaptive learning platform

A pilot-tested web platform with the BKT engine, personalised practice, and an xAPI Learning Record Store.
Prototype
Predictive policy analytics

A dashboard visualising anonymised macro-level skill distributions, trends and regional disparity.
Publication
Peer-reviewed output

Papers on the calibration methodology and on localised assessment generation, plus workshop dissemination.

Impact indicators — targets set at proposal stage and evaluated against pilot data
Indicator	Target	What it tests
Average mastery gain, users completing ≥200 questions	> 30%	Whether the adaptive engine produces measurable learning, not just engagement
Reduction in gap between a user's strongest and weakest decile of skills	> 25%	Whether personalisation is actually targeting weakness rather than reinforcing strength
Correlation, platform mastery vs. real summative exam performance	Statistically significant	Transfer validity — the central scientific claim of the calibration loop
Active weekly users during the six-month core pilot	150+ avg.	Whether the system holds a cohort long enough to generate usable trajectories
Three-month retention	> 40%	Sustained use, a precondition for longitudinal analysis
Post-graduate researchers funded and engaged	5 MS · 1 PhD	Research capacity built, not just software delivered

Principal risks and how they are handled

Cold start with no specialised training data

No domain corpus exists for this syllabus and these competencies; the gold standard set must be built by hand before fine-tuning yields anything useful.

Phase 2 front-loads expert time and treats the gold set itself as a citable deliverable, so the effort produces a research asset regardless of model outcome.

Dependence on subject-matter expert availability

Both the ontology and the moderation loop are gated on sustained access to experts from partner institutions.

Terms of Reference are formalised in Phase 1; independent consultants provide external validation of the framework and its psychometric soundness.

Examination policy change mid-project

A structural change to syllabus or exam format would invalidate parts of the spoke frameworks and the tagged bank.

Hub skills are deliberately policy-independent; spokes are versioned separately so a change re-tags rather than rebuilds.

Rapid model obsolescence

Base models and adaptation techniques chosen today may be superseded within the project lifecycle.

PEFT/LoRA on a compact open-weight base keeps re-adaptation cheap; the gold dataset, not the checkpoint, is the durable asset.

Foundational literature

Corbett, A. T., & Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253–278.
Piech, C., Bassen, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L., & Sohl-Dickstein, J. (2015). Deep Knowledge Tracing. Advances in Neural Information Processing Systems, 28.
Shin, D., Shim, Y., Yu, H., Lee, S., Kim, B., & Choi, Y. (2021). SAINT+: Integrating temporal features for EdNet correctness prediction. LAK21, 490–496.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21, 1–67.
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014). Power to the People: The Role of Humans in Interactive Machine Learning. AI Magazine, 35(4), 105–120.
Siemens, G., & Gašević, D. (2012). Guest editorial — Learning and knowledge analytics. Educational Technology & Society, 15(3), 1–2.
Williamson, B. (2016). Digital education governance: An introduction. European Educational Research Journal, 15(1), 3–13.