Transformer Neural Networks for Natural Language Processing

Abstract

This thesis outlines a retrieval-augmented generation (RAG) pipeline tailored specifically for domain-specific documentation, with particular focus on telecoms config guides. The pipeline outlined here integrates structural pre-processing, dense semantic indexing, and controlled prompt composition to improve factual coherence in LLM responses. The primary contribution is the design of a preprocessing method, based on which semi-structured guides are converted into hierarchically divided JSON files, to enable context-aware retrieval over a vector database. The system is evaluated on 45 queries comprising question answering, summarization, and CLI code generation, on the Arista EOS 4.17.0F guide used as a test corpus. Empirical performance shows retrieval drastically reduces hallucination and increases semantic relevance, with GPT-4 + RAG producing 0.82 accuracy and 0.02 hallucination rate. For contrast, TinyLlama returned nonsensical output for over one-third of inputs. These findings demonstrate the effectiveness of grounded retrieval in aligning model output with technical reference material and illustrate the benefit of structured input preparation for high-fidelity generation in documentation-heavy applications.

This thesis outlines a retrieval-augmented generation (RAG) pipeline tailored specifically for domain-specific documentation, with particular focus on telecoms config guides. The pipeline outlined here integrates structural pre-processing, dense semantic indexing, and controlled prompt composition to improve factual coherence in LLM responses. The primary contribution is the design of a preprocessing method, based on which semi-structured guides are converted into hierarchically divided JSON files, to enable context-aware retrieval over a vector database. The system is evaluated on 45 queries comprising question answering, summarization, and CLI code generation, on the Arista EOS 4.17.0F guide used as a test corpus. Empirical performance shows retrieval drastically reduces hallucination and increases semantic relevance, with GPT-4 + RAG producing 0.82 accuracy and 0.02 hallucination rate. For contrast, TinyLlama returned nonsensical output for over one-third of inputs. These findings demonstrate the effectiveness of grounded retrieval in aligning model output with technical reference material and illustrate the benefit of structured input preparation for high-fidelity generation in documentation-heavy applications.