JAROLÍM, A. Vyhledávání v dokumentech s jemnou granularitou určování relevance [online]. Brno: Vysoké učení technické v Brně. Fakulta informačních technologií. 2025.
Študent pravidelne komunikoval, vyjadril snahu prísť s novými nápadmi, a aj pochopiť existujúcej práci. Svoju prácu dokončil z ambicióznymi výsledkami, a už teraz ju úspešne prezentoval naprieč výskumnými aktivitami fakulty. Preto navrhujem hodnotenie A.
| Kritérium | Známka | Body | Slovní hodnocení |
|---|---|---|---|
| Informace k zadání | Obtiažnejšie zadanie. Bolo treba podrobne porozumieť metódam trénovania a indexovania multi-vektorových modelov, a navrhnúť novú metódu, ktorá by umožňovala vo vyhľadávaní označiť dôležité časti dokumentu. | ||
| Práce s literaturou | Pristupoval aktívne, na schôdzky prinášal nové práce a nápady, a bol ochnotý im porozumieť do detailu. | ||
| Aktivita během řešení, konzultace, komunikace | Pravidelná aktivita, termíny dodržané. | ||
| Aktivita při dokončování | Práca nebola dokončená v predstihu, niektoré detailné experimenty a analýzy boli dokončované ešte pár dní pred odovzdaním. Jej obsah, a aj slabé časti v preliminárnych verziách práce ale boli konzultované dostatočne. | ||
| Publikační činnost, ocenění | Práca bola prezentovaná v rámci študentskej konferencie EXCEL@FIT , a bola ohodnotená dvomi ocneneniami. Práca má tiež výborný potencial stať sa odbornou publikáciou. Študent podal prihlášku na doktorské štúdium, ktorého počiatkom bude dokončenie práce. Študent sa tiež zapojil do aktivíť riešenia dvoch výskumných projektov, konkrétne EU projektu ELOQUENCE, a aj bilaterálneho TAČR projektu FactDeMice. |
The overall work done is very good, and the results are useful for the community. The only reason for not giving an "excellent" rating is due the writing part of the thesis. As explained in my earlier comments, there are parts in the thesis that are not explained clearly.
| Kritérium | Známka | Body | Slovní hodnocení |
|---|---|---|---|
| Náročnost zadání | I rate the assignment as very difficult since it aims to study and propose a solution for a non-trivial problem - extracting interpretable cues for neural information retrieval systems. | ||
| Rozsah splnění požadavků zadání | The thesis presents literature study, proposed methodology and results that fulfull the assignmetnt. | ||
| Rozsah technické zprávy | The report is sufficiently long. | ||
| Prezentační úroveň technické zprávy | 80 | I feel that thesis could have be organized slightly better. Some parts are written very nicely, whereas other sections are little disorganized. For example, the until chapter 4 the reader has no idea how the task is formulated or evaluated. The descriptions and examples keep hovering around multiple datasets, and models, and doesn't say anything concretely. The KL divergence loss appears wrong as it is missing the normaliser. The thesis should have provided the actual derivation instead of saying things in vague words "log-space". Eq 4.5 In the results section, some comparisons are made using only F1-scores Fig 6.5, Table 6.3. I think precision and recall are more interpretable metrics than F1 since one can get obtain a given F1 for multiple combinations of precision and recall. For interpretability it is important to identify which metric is more important - preicision or recall. The final chapter, results and the analysis is very well explained. | |
| Formální úprava technické zprávy | 90 | The formatting of the thesis looks mostly fine, except for the following two issues Figure 4.4 could have been more detailed or bigger. Its hard to differentiate "blue" arrows from the black ones in the printed version of the thesis. | |
| Práce s literaturou | 85 | Relevant literature was described. Although some additional commentary or explanation on Attention rollout, AttCAT, GradSAM would be more helpful. The descriptions of these in the thesis appear too short. For example in 3.1.2 AttCAT, CAT is not defined or explained. Similarly in the chapter 2, the explanation of Transformer, Biencoder, cross-encoder seems incomplete. For example in 2.2.2 the input to Biencoder is explained in detailed, but there was no comment on how the model is trained - which I think is important. The description of transformer encoder appear very cramped, without actually explaining the philosophy behind the architecuture. | |
| Realizační výstup | 90 | Since the work is more research oriented, I do not expect a software package, rather a requirements and detailed Readme which are provided.The python scripts themselves are not documented, but are divided into logical files. | |
| Využitelnost výsledků | The main outcomes of the work are its methodology and findings, which I think are very useful for the scientific community. |
eVSKP id 164140