Enhancing RAG agent performance through structured knowledge bases from NLP-parsed data warehouses
Loading...
Supplementary material
Other Title
Authors
Nguyen, Duc Phu
Author ORCID Profiles (clickable)
Degree
Master of Applied Technologies
Grantor
Unitec
Date
2026
Supervisors
Song, Lei
Sharifzadeh, Hamid
Sharifzadeh, Hamid
Type
Masters Thesis
Ngā Upoko Tukutuku (Māori subject headings)
Keyword
intelligent agents (computer software)
SQL (computer program language)
natural language processing (computer science)
PostgreSQL
query languages (computer science)
data warehousing
procurement
AI in business
SQL (computer program language)
natural language processing (computer science)
PostgreSQL
query languages (computer science)
data warehousing
procurement
AI in business
Citation
Nguyen, D.P. (2026) Enhancing RAG agent performance through structured knowledge bases from NLP-parsed data warehouses (Unpublished document submitted in partial fulfilment of the requirements for the degree of Master of Applied Technologies (Computing)). Unitec.
https://hdl.handle.net/10652/7151
Abstract
Enterprise teams need correct, explainable answers from governed data warehouses (DWs), but Natural-Language-to-SQL (NL2SQL) systems often fail when schemas evolve, con straints are complex, and policies must be enforced consistently. Current RAG-only approaches and unconstrained NL2SQL do not reliably enumerate eligible options or enforce procurement policies under schema change. In ten controlled IT procurement user cases, schema-aware execution recovered the full budget-compliant device pools and enforced auto-approval rules with 0% violations, while a RAG-only baseline omitted eligible devices and occasionally exceeded budget at rank 1. This thesis investigates how a schema-aware hybrid pipeline can provide grounded, auditable answers for budget constrained and policy-constrained IT procurement decisions over a realistic warehouse slice. The proposed design scopes each interaction to a safe, read-only query surface, retrieves schema and policy context, and generates a single validated SELECT statement. It combines dense vector retrieval over documentation and metadata with a Qdrant index, a lightweight knowledge graph for entity and relationship hints, and a per-session PostgreSQL warehouse slice that encodes request budgets and departmental auto-approval rules.
To assess effectiveness, three pipelines are implemented and compared: a RAG-only pipeline that answers directly from retrieved text; a SQL-only pipeline that queries the session warehouse through a fixed, validated SELECT template; and the proposed Hybrid pipeline that couples semantic retrieval with constrained, warehouse-backed execution. Using ten controlled user cases (A–J) spanning departments, roles, quarters, and new versus replacement requests, the evaluation measures (Q1) correctness of the returned budget-fitting device pool and (Q2) correctness and efficiency of standard auto-approvable shortlists. Against a deterministic SQL baseline, the SQL-only and Hybrid pipelines recovered the exact Q1 device sets for all cases (65 devices total), while the RAG-only pipeline returned 49 devices and matched the expected set in 2 of 10 cases (mean absolute count error 1.6). Top-1 budget compliance was 10 of 10 cases for the SQL-only and Hybrid pipelines and 8 of 10 for the RAG-only pipeline. For Q2, only the SQL-backed pipelines (Experiments 2 and 3) were evaluated because auto-approval is represented as governed warehouse attributes (for example, auto approve ok) that Experiment 1 cannot access or enforce reliably from text alone; both SQL-only and Hybrid matched the baseline auto-approved pools in all cases and returned 0% of devices that were outside budget or not auto-approved. Although they return identical Q2 decisions, compared with SQL-only, the Hybrid pipeline reduced median end-to-end latency from 9.45s to 5.77s and median total tokens from 7,457 to 3,656 (about 51% lower).
These results show that constraining generation to a schema-scoped and policy-scoped context can deliver accurate, constraint-respecting answers at lower operational cost, without sacrificing auditability. The thesis contributes: (i) a governance-first NL2SQL pattern with static name, join, type, and policy checks plus evidence tracing; (ii) an empirical comparison of RAG-only, SQL-only, and Hybrid pipelines on a realistic IT procurement workload; and (iii) reusable per-session design patterns for organisations seeking to deploy schema-aware RAG in regulated, data-intensive environments.
Publisher
Permanent link
Link to ePress publication
DOI
Copyright holder
Author
Copyright notice
All rights reserved