MALASHCHUK, V. Chatbot založený na hlubokých neuronových sítích [online]. Brno: Vysoké učení technické v Brně. Fakulta informačních technologií. 2025.
The student worked diligently and independently, regularly consulted with me, and presented results. Overall, the student did a great job.
| Kritérium | Známka | Body | Slovní hodnocení |
|---|---|---|---|
| Informace k zadání | The thesis covers an advanced topic that goes beyond the knowledge acquired in a bachelor's degree program. The student had to select a suitable dataset, design and implement non-trivial versions of a chatbot, and conduct experiments. In my opinion, the assignment was fulfilled and even extended. Two types of chatbots were implemented, and systematic experiments were carried out, both quantitative and qualitative with users. | ||
| Práce s literaturou | The student was independent in searching for study materials and, in my opinion, achieved a good understanding of the field. | ||
| Aktivita během řešení, konzultace, komunikace | The student worked independently and proactively, consulting on specific problems and progress. Apart from a break at the beginning of the second semester, the student was active from the very start, and the first prototypes were developed very quickly. The student fulfilled what we agreed on during consultations and sometimes even more. | ||
| Aktivita při dokončování | The student completed the work ahead of schedule. The full text of the thesis was presented to me in advance and my comments were satisfactorily incorporated. My only criticism concerns some of the experiments. Their results could have been discussed with me more thoroughly, and I did not have the opportunity to test the later versions of the chatbots myself. | ||
| Publikační činnost, ocenění |
Mr. Malashchuk carefully studied the problem for the assignment and created two variants of the chatbot solution, considering different scenarios based on selected datasets. He implemented the proposed solution using current relevant architectures and tools. He evaluated his models both objectively, using the relevant BLEU metrics, and subjectively, eliminating the limitations of the objective evaluation. The solved topic is technically challenging, and the author demonstrated a good understanding of the subject matter in both the theoretical and experimental parts of the thesis, which is well beyond the scope of common undergraduate knowledge. The whole thesis is of well above average technical and experimental quality.
| Kritérium | Známka | Body | Slovní hodnocení |
|---|---|---|---|
| Náročnost zadání | The difficulty of the assignment depends on the technologies chosen. The author chooses advanced technologies from the field of NLP, which makes the assignment more difficult. | ||
| Rozsah splnění požadavků zadání | In addition, the solution of the assignment includes the implementation of the chatbot by two different approaches, including relevant comparisons, which can be evaluated as a substantial extension of the assignment. | ||
| Rozsah technické zprávy | |||
| Prezentační úroveň technické zprávy | 95 | The theoretical part in chapters 2 and 3, although briefly and without deeper details, and somewhat beyond the specific scope of the assignment, discusses the concepts of AI, ML, NN and Deep learning, NLP, LLM and chatbots. It builds on a considerable list of relevant professional study literature. It is striking how many scientific articles and books the author cites and draws from in the theoretical part. The text of the theoretical part is of such high quality, expertly prepared and clearly presented, including absolutely perfect English, that any expert in the field would not be ashamed of it. However, the author demonstrates throughout the work a good understanding of the subject matter and consistency of professional presentation of both the design and execution of the experiments, such that his understanding and presentation of the subject matter are credible. Already in the introduction, and then in both the theory and the design sections, the author mentions the key BLEU metric, which is explained and referenced only in Section 4.4. It would have been useful to introduce this metric earlier. The design section, which should be more methodological, also contains implementation details, such as file names, which should be specified only when the design is implemented (in the implementation chapter). | |
| Formální úprava technické zprávy | 80 | The typographical and linguistic level is excellent, except for a few minor mistakes. The text contains only a few errors in punctuation and in the division of words at the ends of lines. Some of the figures are in raster format with a rather low resolution, although they are diagrams that should be in vector format. This makes the image harder to read and understand (e.g. Figure 4.3). Some figures are not referenced from the text, and some figures (e.g. Figure 3.1) mention parameters that are not explained properly anywhere. Furthermore, it can be recommended that the explanation of the problem and the parameters should not be in the figure caption (e.g. Figure 3.2) but in the text, and the figure should only be a schematic supplement to better understand the architecture. | |
| Práce s literaturou | 100 | The author draws from an extensive list of study literature, both scientific publications and books, and it is evident from the entire solution and text that the author understands and discusses the topic in a professional and substantive manner. | |
| Realizační výstup | 95 | The technical solution uses current and relevant practices, technologies, and tools. The source files have a clear logical structure, and the readme is excellent. Although key parts of the source code contain good comments, it would be useful to have comments everywhere. However, it is not at all clear from the text (neither in the design chapter 4 nor in the experimental chapter 5) what specific models the author uses, both for the Retrieval-Based and the Transformer-Based variants. For the Retrieval-Based Model experiment, Table 5.5, the author could have better explained attributes such as Val (the reader will somehow deduce the others), and above all, draw conclusions on what hyperparameters are best to use. The author does a good job of evaluating the evolution of the model learning on the second dataset (DoctorAI) and adjusts the hyperparameters to better reflect the characteristics of the problem. Unfortunately, it is not entirely clear what specific model parameters are used for the author-defined categories of small, medium and large. For Figure 5.5, it would be useful to indicate the meaning of the rating, i.e. if 1 is the minimum or maximum. The author extended the experiment to include the subjective perception of the user's discussion with the chatbot. This can be seen as a very reasonable approach, where the results of the dialogue are not only evaluated objectively (which can be misleading, as the author rightly points out) but also subjectively by the user. The author could have stated the number of test subjects earlier than at the end of the paper. Overall, however, the output is extensive and of an excellent professional standard. | |
| Využitelnost výsledků | The developed work can serve as a good theoretical basis as well as an evaluation of two different chatbot implementations with respect to the available training sets and architectures. |
eVSKP id 164616