Spyra-20B — Domain-Specific LLM for Architectural Design Reasoning

Spyra-20B is a domain-specific Large Language Model for the AEC industry (Architecture, Engineering, Construction). It combines Tree-of-Thought (ToT) and Chain-of-Thought (CoT) reasoning to decompose complex architectural design problems — similar to how experienced architects think, explore alternatives, and make decisions.

Model Details

Model Description

Spyra-20B was developed to bridge the gap between general-purpose LLMs and the multidimensional reasoning required in architectural design. The model can simultaneously balance creativity, building code compliance, material sustainability, structural constraints, and functional zoning within a coherent design strategy.

The model uses a two-channel architecture: an Analysis Channel for structured reasoning (ToT/CoT) and a Final Channel for the user-facing answer. The reasoning effort adapts dynamically (low / medium / huge) based on task complexity.

Developed by: [Nik Ansre & Yara Hirsekorn / Jade Hochschule]
Model type: Causal Language Model (fine-tuned with QLoRA)
Language(s): German (primary), English
License: [Apache license 2.0]
Base model: unsloth/gpt-oss-20b-BF16
Architecture: Mixture-of-Experts (MoE), 20B parameters, 32 experts
Fine-tuning method: QLoRA (Quantized Low-Rank Adaptation)

Intended Use

Primary Use Cases

Spyra-20B is designed as a professional assistant for architects, urban planners, and AEC professionals. Intended tasks include:

Draft logic: Parametric design in Rhino/Grasshopper, spatial planning, zoning strategies
Post-compaction: Evaluating densification strategies (e.g. Extension, addition, courtyard development) with trade-off analysis
Approval processes: Building permit logic, regulatory compliance, Bauordnung interpretation
Building renovation & historic preservation: Renovation planning, handling archaeological findings, heritage constraints
Structural planning: Structural considerations, material selection, load-bearing analysis
Process logic: Construction process management, stakeholder coordination, phased planning

Out-of-Scope Use

This model is not intended for:

Safety-critical autonomous decisions: The model is an assistant tool — all design and construction decisions must be verified by qualified professionals.
Legal advice: Outputs regarding building codes or regulations are informational and do not constitute legal counsel.
Structural calculations: The model can reason about structural concepts but must not replace certified structural engineering software or professional engineers.
Medical, financial, or other non-AEC domains: The model is specialized for architecture and construction.
Generating content for deception or manipulation.

Bias, Risks, and Limitations

Known Limitations

Training data bias: The model was trained on expert-curated German and English AEC data. It may underperform on regional building codes outside the DACH region (Germany, Austria, Switzerland).
No visual understanding: The model is text-only and cannot process images, plans, BIM models, or drawings.
Knowledge cutoff: The model's knowledge is limited to its training data and does not include regulations or standards published after the training date.
Hallucination risk: Like all LLMs, Spyra-20B may generate plausible but incorrect information, particularly for edge cases not well represented in the training data.
No real-time data: The model cannot access current project data, weather, material prices, or live regulatory databases.

Recommendations

Always verify model outputs with qualified AEC professionals before making design or construction decisions.
Use the model as a reasoning assistant, not as a replacement for professional judgment.
Cross-check regulatory and code-related outputs against current official sources.
Consider RAG (Retrieval-Augmented Generation) integration for project-specific or up-to-date regulatory information.

Training Details

Training Data

The training dataset is an expert-curated corpus of AEC-specific conversations covering architectural design, structural engineering, building permits, urban densification, heritage preservation, and parametric design. Each example contains:

A system prompt defining the expert role and context
A user query with a complex architectural task
A thinking field (reasoning chain) with the step-by-step thought process
A final answer with the structured response
Metadata: domain, language, reasoning effort level (low / medium / huge)

For tasks that require deep scenario simulation (Tree-of-Thought), the following system prompt is used during training:

Du bist ein Senior Architekt. Nutze tiefe Simulationen für komplexe Probleme.

This prompt activates the model's ToT reasoning mode, in which it simulates multiple solution paths in parallel, evaluates their consequences across dimensions (e.g. structural feasibility, regulatory compliance, cost, stakeholder impact), and selects the best option with explicit justification — mirroring how experienced architects approach complex design decisions.

The data was curated by domain experts and is not publicly available.

Copyright & Training Data Summary (EU AI Act, Art. 53(1)(d)): The training data consists of original, expert-authored content created specifically for this project. No copyrighted third-party works (books, articles, standards documents) were reproduced in the training data.

Training Procedure

Architecture

The training uses a native two-channel format aligned with the base model's architecture:

<|start|>assistant<|channel|>analysis<|message|><reason:medium>
[Reasoning: ToT/CoT thought process]<|end|>
<|start|>assistant<|channel|>final<|message|>
[Structured answer]<|return|>

Completion-Only Training: The loss is computed exclusively on the Final Channel output. The Analysis Channel serves as a structural aid but does not contribute to the loss function.

Training Hyperparameters

Parameter	Value
Base model	unsloth/gpt-oss-20b-BF16
Method	QLoRA (4-bit quantization + LoRA adapters)
Max steps	3,500
Precision	BF16 mixed precision
Training regime	bf16 mixed precision

Hardware

GPU: 2× NVIDIA RTX A6000 (48 GB VRAM each), training on single GPU with CPU offloading
Training time: ~4 hours
Framework: Unsloth + Hugging Face TRL/PEFT + PyTorch 2.9.0

Environmental Impact

Factor	Value
Hardware	2× NVIDIA RTX A6000
Training time	~4 hours
Estimated energy consumption	~2.4 kWh (600W × 4h)
Estimated CO₂ emissions	~1.0 kg CO₂eq (EU average grid intensity)

Carbon emissions estimated using the ML Impact Calculator (Lacoste et al., 2019).

EU AI Act Compliance Notes

This section documents compliance-relevant information per EU AI Act (Regulation (EU) 2024/1689).

Risk Classification

Spyra-20B is a domain-specific fine-tuned model based on a general-purpose AI model. It is designed as an advisory tool for AEC professionals and does not make autonomous decisions. It is not classified as a high-risk AI system under Annex III of the AI Act, as it does not operate in safety-critical domains (e.g. critical infrastructure control, medical devices) without human oversight.

Transparency (Art. 50 / Art. 53)

AI-generated content: Outputs are generated by an AI model and should be clearly labeled as such in any deployment context.
Model card: This document serves as the model card and transparency documentation.
Technical documentation: Available upon request to competent authorities.

Training Data Summary (Art. 53(1)(d))

The training data is an original, expert-curated dataset. No web-scraped, copyrighted, or personal data was used. A detailed training data summary is available upon request.

Copyright Policy (Art. 53(1)(c))

The training data was authored specifically for this project. No text-and-data-mining (TDM) of copyrighted works was performed. The model respects EU copyright law (Directive (EU) 2019/790).

Human Oversight

This model is designed to be used under human oversight. All outputs must be reviewed by qualified professionals before being used in design, construction, or regulatory decisions.

How to Use

With Ollama

ollama create spyra-20b -f Modelfile
ollama run spyra-20b

Modelfile

FROM ./Spyra-20B-f16.gguf

TEMPLATE """<|start|>system<|message|>
{{- if .System }}
{{ .System }}

{{ end -}}
You are Spyra, a professional AI assistant for the AEC industry.
RULES:
1. First, think in the 'analysis' channel.
2. THEN, YOU MUST WRITE THE ANSWER in the 'final' channel.
3. Do not repeat sentences.
4. Answer in the user's language.
5. Keep answers short and correct.
<|end|>
{{- range .Messages }}
{{- if eq .Role "user" }}
<|start|>user<|message|>{{ .Content }}<|end|>
{{- else if eq .Role "assistant" }}
<|start|>assistant<|channel|>final<|message|>{{ .Content }}<|end|>
{{- end }}
{{- end }}
<|start|>assistant"""

PARAMETER num_ctx 8192
PARAMETER temperature 0.4
PARAMETER stop "<|return|>"
PARAMETER stop "<|endoftext|>"
PARAMETER stop "<|call|>"
PARAMETER stop "<|start|>user"