The _ALLOWED_SCHEMES set includes "file", which contradicts the docstring for validate_url and the security model described in SECURITY.md. Allowing the file scheme in a tool that fetches arbitrary URLs can lead to Server-Side Request Forgery (SSRF) or local file disclosure if a user-provided URL or a malicious redirect points to a local file. This should be restricted to http and https only.

The regex chr(10) + chr(13) matches the literal sequence \n\r. This will not match standard Unix (\n) or Windows (\r\n) line endings. To correctly sanitize the question for YAML frontmatter and prevent it from breaking the format, you should replace all carriage returns and newlines with spaces.

The vault_root is hardcoded to /tmp/unsloth_wiki, which is not persistent. Additionally, studio/backend/core/inference/tools.py uses an environment variable UNSLOTH_WIKI_VAULT for the same purpose. This should be updated to respect the environment variable for consistency and persistence.

datetime.datetime.now() creates a naive datetime object using the local system time. It is a best practice to use timezone-aware datetimes (e.g., UTC) to ensure consistency across different server environments and avoid ambiguity during daylight saving time transitions.

            import datetime\n            timestamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%d %H:%M:%S")

path.read_text() reads the entire file into memory at once. If the corpus contains very large files (e.g., large documentation or source files), this could lead to excessive memory usage or a MemoryError. It is safer to read the file line by line or in chunks when counting words.

        with path.open("r", errors="ignore") as f:\n            return sum(len(line.split()) for line in f)

nx.betweenness_centrality(G) is a computationally expensive operation ($O(VE)$ for unweighted graphs). While acceptable for small graphs, it can become a significant performance bottleneck as the knowledge graph grows. Consider using faster centrality measures (like degree centrality) or limiting this analysis to a subgraph if performance becomes an issue.

Copy messages before injecting RAG system context

When system_prompt is empty, template_messages aliases the original messages list, so the later insert(0, context_message) mutates the caller’s conversation in place. In this flow, _save_chat_history_to_wiki(messages) then persists the injected RAG block as if it were user conversation, which can recursively pollute future retrieval context and inflate prompts across turns. Build template_messages as a copy before insertion to keep prompt-only context out of persisted chat history.

Useful? React with 👍 / 👎.

Use configured wiki vault path in backend worker

This hardcodes the backend worker to /tmp/unsloth_wiki while the route layer and watcher use UNSLOTH_WIKI_VAULT. If operators set a custom vault path, non-GGUF inference reads/writes a different wiki than /wiki/* endpoints and startup watcher maintenance, causing split state and missing RAG context on one path. The worker should resolve the same env-configured vault root as the rest of the wiki stack.

Useful? React with 👍 / 👎.

Avoid double-ingesting chat history files

After writing a chat-history batch to raw/, this code ingests the file immediately, but main.py enables WikiIngestionWatcher by default on the same raw/ directory. In that default setup, each flushed chat-history file is ingested twice (once here, once by the watcher), causing redundant index rebuilds/log churn and unnecessary maintenance work. Choose a single ingestion path (watcher or direct ingest) or mark these files as already handled.

Useful? React with 👍 / 👎.

Restore get_current_model on InferenceBackend

check_vision_model_compatibility() still calls self.get_current_model(), but this commit removed that helper when this section was replaced with wiki/RAG methods. Any caller that hits the vision compatibility helper now gets AttributeError instead of a boolean compatibility result, which breaks the vision-guard flow at runtime.

Useful? React with 👍 / 👎.

Use collision-proof filenames for chat history flushes

This flush path uses a filename with only second-level precision and then writes with write_text, so two flushes in the same second target the same file and the later flush overwrites earlier buffered snapshots. That is reachable under concurrent traffic or when UNSLOTH_WIKI_CHAT_HISTORY_FLUSH_SECONDS=0, causing silent history loss and weaker downstream RAG context.

Useful? React with 👍 / 👎.

Handle missing watchdog dependency for default watcher mode

The watcher module imports watchdog at module scope, so environments without that extra cannot import core.wiki.watcher at all. Because watcher startup is enabled by default, this leads to startup falling back to the exception path and silently disabling automatic raw-file ingestion/analysis unless watchdog is preinstalled (it is not listed in the backend default requirements file).

Useful? React with 👍 / 👎.

Skip RAG/history hooks for internal wiki LLM calls

generate_chat_response now always runs _get_rag_context(...) and _save_chat_history_to_wiki(messages), but this method is also used by _route_wiki_llm_stub for wiki ingestion/enrichment/retry flows (not just user chat). In that path, internal maintenance prompts are persisted as chat history and re-ingested into the wiki, which pollutes retrieval with non-conversational prompt scaffolding and can recursively bloat the RAG corpus over time. Add an opt-out flag (or a dedicated internal generation path) so only user-facing chat requests write history / inject RAG.

Useful? React with 👍 / 👎.

Normalize message content types before buffering chat history

This assumes every message content is a string and calls .strip(). The legacy /generate/stream route forwards GenerateRequest.messages as raw OpenAI-format dicts, so content can be non-string (e.g., multimodal arrays/tool payloads); in those cases this raises AttributeError, the outer except swallows it, and the entire snapshot is dropped. Coerce/serialize non-string content before trimming so history persistence remains reliable across valid message shapes.

Useful? React with 👍 / 👎.

Remove unsupported wiki-history flag from orchestrator call

_route_wiki_llm_stub passes enable_wiki_rag_history=False to backend.generate_chat_response, but this route gets its backend from get_inference_backend() (the orchestrator), whose generate_chat_response does not accept that keyword. The resulting TypeError is swallowed by the local except, so transformer-backed wiki calls silently fall back to returning the prompt text instead of model output, degrading /wiki/query, enrichment, and fallback-retry behavior whenever GGUF is not active.

Useful? React with 👍 / 👎.

Record watcher hashes only after successful ingest

The watcher marks a file as processed (_processed_mtime_ns/_processed_hash) before ingest_file succeeds. If ingestion fails transiently (e.g., file still being finalized), later events with unchanged content are skipped as duplicates, so that raw file may never be ingested unless it is edited again. This breaks reliable at-least-once ingestion for newly dropped files.

Useful? React with 👍 / 👎.

Ignore hidden subdirectories when scanning raw ingest candidates

ingest_pending_raw_files() recursively scans raw/ and relies on should_skip_local_file(), but this predicate only checks the basename and does not exclude files under hidden directories (for example raw/.archive/...). After /wiki/archive/stale moves raw files into .archive, those files become eligible for re-ingestion on the next pending-ingest run, which can silently resurrect archived content and churn duplicate wiki pages.

Useful? React with 👍 / 👎.

Honor analysis-page exclusion during empty ranking fallback

When include_analysis_pages_in_query is disabled, query() first filters out analysis/* pages, but if that leaves no candidates it immediately reruns _rank_pages(question) without reapplying the filter. In that case analysis pages are reintroduced despite the config flag, so deployments that disable analysis-page retrieval can still get self-referential analysis context.

Useful? React with 👍 / 👎.

Use collision-resistant titles for local wiki ingest

Returning only file_path.stem here makes distinct files with the same basename (for example raw/repoA/README.md and raw/repoB/README.md) share one source_title; downstream LLMWikiEngine.ingest_source slugs that title to the same sources/<slug>.md path, so later ingests silently overwrite earlier content. This causes real data loss and missing context whenever users ingest directory trees that contain common filenames.

Useful? React with 👍 / 👎.

Detect history intent with explicit phrases only

Including the bare keyword "token" in history-intent detection makes many non-history queries (for example tokenization, API token usage, token limits) flip into the history branch. In that branch, retrieval is biased toward or replaced by chat-history-* pages, so prompt context can become unrelated to the user’s actual topic and degrade answer quality. History mode should require explicit conversation-memory phrases instead of this broad match.

Useful? React with 👍 / 👎.

Ingest flushed chat batches when watcher is unavailable

This flush path writes buffered chat history to raw/ and immediately clears the in-memory buffer, but it never invokes wiki_ingestor to index the file. In transformer-backed mode, if UNSLOTH_WIKI_WATCHER=false (or watcher startup fails), these snapshots remain un-ingested, so _get_rag_context cannot retrieve recent history even though persistence appears successful. Add a direct-ingest fallback for this path when automatic watching is not active.

Useful? React with 👍 / 👎.

Parse wiki env integers defensively

These module-level assignments call int(os.getenv(...)) directly, so any non-numeric value (including empty-string env overrides) raises ValueError during import and prevents the inference routes from loading at startup. Since these are operational knobs, a single misconfigured deployment variable can take the whole API down before request handling begins; use the same guarded parsing pattern already used elsewhere in this codebase.

Useful? React with 👍 / 👎.

Restrict URL validation to http/https schemes

validate_url() is documented and messaged as allowing only HTTP(S), but _ALLOWED_SCHEMES currently includes "file", which means file://... inputs are accepted and then read by safe_fetch(). In any flow that calls graphify ingest with user-provided URLs, this allows local file exfiltration (for example file:///etc/passwd) into generated artifacts.

Useful? React with 👍 / 👎.