Author background: Backend development, self-taught LLM fine-tuning in spare time. This article records the complete process of a personal hands-on project (RoboThink) from model selection, training, quantization, to local deployment, focusing on the pitfalls I encountered, issues that only became clear in hindsight, and unverified assumptions. The project results are not flashy, but the lessons learned along the way are very valuable to me, so I organized them into this write-up.

0. Conclusion First

1. Fine-tuning a 7B model using LoRA on a single-GPU A100 has a much lower engineering barrier than expected: 1,296 samples, 162 steps, about 5 minutes. Combined with GGUF quantization, it can run offline on a Mac with 16GB of memory.
2. However, an entire data engineering gap lies between "making it run" and "making it work well". My first version of the model showed obvious failures on Japanese test questions: the thinking process was output in Chinese, and it repeatedly made assumptions inside <think> without ever converging, until it hit the length truncation limit.
3. Post-mortem analysis suggests the issue was likely not caused by a single factor, but rather a combination of single-language training data, inconsistent training/inference formats, and too few evaluation samples. Among these, the "inconsistent format" was only discovered while writing this article and cross-checking the files one by one.
4. The biggest takeaway from this project is not the LoRA weights, but a more concrete physical intuition of "what fine-tuning is actually changing in the model."

1. Project Goals and Constraints

My goals were very specific:

• The model should provide answers with reasoning processes and clear conclusions for warehouse/AGV scheduling problems (lane avoidance, charging priority, simple operations research calculations);
• Support queries in three languages: Chinese, Japanese, and English;
• Complete within a personal budget: rent cloud GPUs by the hour for training, and use only my personal Apple Silicon Mac with 16GB unified memory for inference.

These three constraints basically determined all subsequent technical choices.

The overall pipeline is as follows:

  [General R1 Distilled Data]                      [Colab A100 80GB]
   Filter <think> complete samples ─────────►  Unsloth + TRL SFTTrainer
   Assemble into ChatML text                   LoRA r=32, bf16, 1 epoch
                                              │
                                              ▼
                                  robothink_7b_lora (approx. 323MB adapter)
                                              │
                           ┌──────────────────┴──────────────────┐
                           ▼                                     ▼
              Merge Weights → GGUF q4_k_m                Direct PEFT Mount (Fallback)
                           │                                     │
                           ▼                                     ▼
              llama-cpp-python + Metal             Transformers + MPS
                           └──────────────┬──────────────────────┘
                                          ▼
                                  chat.py CLI Interaction

2. Selection: Using Arithmetic Instead of Intuition

2.1 Why 7B?

For personal hardware, model size is first and foremost a memory arithmetic problem.

• With 7.6B parameters stored in bf16 (2 bytes per parameter), the weights alone are about 15.4 GB, which already exceeds the actual available space on a 16GB memory machine;
• After q4_k_m quantization, it averages about 4.8 bits/parameter, resulting in weights of about 4.5–5 GB. Adding KV Cache and system overhead, a 16GB machine can run it comfortably.

Therefore, "7B + 4-bit quantization" is almost the sweet spot for this type of hardware. Going any higher (14B) still requires 8–9GB after quantization, which gets very tight during concurrency or long contexts.

The reason for choosing DeepSeek-R1-Distill-Qwen-7B as the base model: it already possesses the behavior of thinking first via <think> and then answering, and the Qwen base model has decent multi-language capabilities in Chinese, Japanese, and English. What I needed to do was "domain adaptation," not teach it reasoning from scratch.

2.2 How the LoRA Parameter Count is Derived

The training logs showed trainable parameters of 80,740,352 / 7,696,356,864 (1.05%). I manually calculated this against the model architecture to confirm my understanding of the number:

Key dimensions of Qwen2-7B: hidden layer 3584, MLP intermediate layer 18944, GQA K/V projection dimension 512, across 28 layers. LoRA adds r × (in + out) parameters to an in → out linear layer. With r = 32, each layer has:

2,883,584 × 28 layers = 80,740,352, which matches the log precisely.

The value of this exercise is that I clearly know how parameters and VRAM change when selecting rank and target_modules, rather than treating r=32 as a magic number.

3. Training Pipeline Details

3.1 Data: General Open-Source Distillation Set + Simple Filtering

For the first version, I did not generate custom data. Instead, I directly took the first 3,000 samples from Congliu/Chinese-DeepSeek-R1-Distill-data-110k-SFT, kept samples containing complete <think>...</think> tags with lengths between 80 and 1,800 characters, ultimately retaining 1,296 samples.

def is_valid_cot(sample):
    out = sample.get("output", "")
    return "<think>" in out and "</think>" in out and 80 <= len(out) <= 1800

There was a point here that I didn't think through deeply at the time: this dataset is in Chinese. It has no direct relation to the warehousing scenario; essentially, it just keeps the model "maintaining the R1-style output." As we will see later, this was a foreshadowing issue.

3.2 Format: Hand-Written ChatML

I manually assembled each sample into a ChatML-formatted text field:

text = (f"<|im_start|>system\n{SYSTEM_PROMPT}<|im_end|>\n"
        f"<|im_start|>user\n{query}<|im_end|>\n"
        f"<|im_start|>assistant\n{out}<|im_end|>")

The thought at the time was, "The Qwen series uses ChatML, so this should be fine." This assumption proved problematic later; see Section 6.

3.3 Hyperparameters

Loss dropped from 1.71 at step 10 to between 1.2 and 1.3, plateauing in the second half.

This loss curve needs to be interpreted with caution: it only shows that the model is fitting the output style of this batch of data, and does not indicate that it has become smarter regarding warehouse scheduling problems. There is no necessary correlation between a decrease in training loss and "improvement in the task I care about," which was directly reflected in the evaluation below.

3.4 A Minor Engineering Lesson: Config Files Not Actually Effective

There was a file named configs/training_config.yaml in the project, specifying num_samples: 1500 and logging_steps: 5. However, the training code in the Notebook was hardcoded (logging_steps = 10, actual samples 1,296) and never read this configuration file at all.

This is a classic case of "documentation and code drift": the configuration file looks standardized, but nobody actually uses it, which ends up misleading your future self. Moving forward, I plan to make the training script truly load parameters from YAML or simply delete the configuration file.

4. Quantization Export and Local Deployment

4.1 Merge + GGUF Quantization

Training yielded a LoRA adapter of about 323MB. To get good inference performance on a Mac, I used Unsloth to merge the adapter into the base model and then converted it into a q4_k_m GGUF:

model.save_pretrained_gguf("robothink_7b_q4", tokenizer, quantization_method="q4_k_m")

The logs show the process: first merge into 16-bit weights → convert to BF16 GGUF → then quantize to Q4_K_M. The entire process takes about 16 minutes in the cloud (official estimate; I didn't time it precisely).

4.2 Two Design Choices in chat.py

(1) Dual-Engine Auto-Selection

Upon startup, it searches for .gguf files in models/, the project root, and ~/Downloads:

• If found: Uses llama-cpp-python with n_gpu_layers=-1 to offload all layers to the Metal GPU;
• If not found: Falls back to PyTorch (mps → cuda → cpu) to load the original base model, and mounts LoRA via PEFT.

This way, even without exporting GGUF, the project can still run, albeit slower and consuming more memory.

(2) Stripping the Reasoning Process from History

Every response from a reasoning model comes with a large <think> block. Putting all of them back into the multi-layer conversation context wastes KV Cache and causes subsequent turns' attention to be distracted by old reasoning processes. Therefore, history storage only keeps the conclusion:

clean_reply = re.sub(r"<think>.*?</think>", "", full_reply, flags=re.DOTALL)
clean_reply = clean_reply.replace("<|im_end|>", "").strip()
if clean_reply:
    history.append({"role": "assistant", "content": clean_reply})

Later, when checking DeepSeek's official chat template, I found that the template itself does this as well (extracting the portion after </think> for assistant history content). This counts as reaching the same destination by different paths, giving me more confidence in this design.

5. Evaluation: A Failed Experiment

After training, I only ran one test question in the Notebook (Japanese AGV charging priority):

AGV-1 battery level 82%, distance to charging station 8m; AGV-2 battery level 11%, distance to charging station 45m; there is only one idle charging pile, who should the FMS give priority to?

An expected answer is clear: AGV-2. A battery level of 11% is already approaching the risk line of "getting stranded and blocking the path," taking higher priority than distance.

The model's actual performance:

• The question was in Japanese, but the <think> block muttered to itself entirely in Chinese;
• It oscillated repeatedly between "prioritizing lower battery" and "prioritizing higher battery";
• It arbitrarily assumed "charging speed is 10% per hour" and "5% per hour," coming up with a bunch of numbers nowhere to be found in the prompt;
• It was truncated at the limit of 1,024 new tokens without providing any final conclusion.

This is a classic case of overthinking: the model had no converging "exit point."

It must be emphasized: this was only a single sample. Based on this, I should not conclude that "the model is universally bad," nor should I believe any claims of "100% pass." In fact, the original "100% Pass" claims in the project README were not supported by automated evaluation, and I have realized such claims should not remain in the documentation.

6. Review: Potential Causes of Failure

Below is my analysis of this failure. These are evidence-backed hypotheses rather than conclusions verified by ablation experiments.

6.1 Data was Single-Language (Chinese)

Training data came entirely from Chinese distillation data. LoRA fine-tuning pulls the model's "thinking language" toward the distribution of the training data. Japanese questions + Chinese thinking is a very natural result under this hypothesis: what the model learned is "R1-style thinking = Chinese."

Verifiable approach: Track the primary language of <think> under English, Chinese, and Japanese queries respectively, to see if it behaves normally with Chinese queries and suffers language drift with Japanese/English queries.

6.2 Inconsistent Training and Inference Formats (The Most Important Discovery)

This is what I discovered when cross-checking files one by one.

• During training: I hand-wrote ChatML (<|im_start|>...<|im_end|>);
• Exported chat template (chat_template.jinja): DeepSeek style — user turns are <｜User｜>, assistant turns are <｜Assistant｜>, generation prompt ends with <｜Assistant｜><think>\n, and the stop token is <｜end▁of▁sentence｜>.

In other words, the model saw one "dialect" during training, but encountered another "dialect" during inference in chat.py (whether using the GGUF built-in template or apply_chat_template). I manually used ChatML in the Notebook evaluation, so evaluation and training aligned, but the actual production path did not.

Such inconsistencies may not completely break the model, but they weaken the fine-tuning effect and make phenomena like "looks fine during training, behaves unstably after deployment" very hard to troubleshoot.

A safer approach: Render training data uniformly using the template provided by the tokenizer, so that training, evaluation, and inference share the exact same template:

def format_with_template(batch):
    texts = []
    for inst, inp, out in zip(batch["instruction"], batch["input"], batch["output"]):
        user = f"{inst}\n{inp}".strip() if inp else inst
        messages = [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user},
            {"role": "assistant", "content": out},
        ]
        texts.append(tokenizer.apply_chat_template(messages, tokenize=False))
    return {"text": texts}

A detail to keep in mind: this template discards content before </think> when rendering assistant history content. If the assistant section of the training sample needs to retain the reasoning process, you cannot render it directly with this template; you must either stitch it manually or modify the template. This needs to be considered during implementation.

6.3 Inconsistent System Prompts

During training, the System Prompt only contained the half-sentence: "Please reason step by step inside <think>"; whereas the System Prompt in chat.py added: "If your preliminary derivation is mathematically and logically rigorous, converge directly to the final conclusion."

My original intention in adding this sentence was to mitigate overthinking, but the model never saw this sentence during training, and expecting it to change its behavior during inference based on a single prompt sentence is unrealistic. If this constraint is expected to work, it should appear in the training data, and the corresponding samples must actually demonstrate the behavior of "converging directly."

6.4 Loss Computed Across the Entire Sequence

Currently, training computes loss across the entire text (including system, user, and assistant), and the model is also "learning to restate the prompt." A more common and reasonable practice is to compute loss only on the assistant's response part (response-only training). Unsloth provides a corresponding tool:

from unsloth.chat_templates import train_on_responses_only

trainer = train_on_responses_only(
    trainer,
    instruction_part="<｜User｜>",
    response_part="<｜Assistant｜>",
)

The separation tokens must strictly match the actual template used, so this step should be done after fixing 6.2.

6.5 Evaluation was Too Thin

Only one question, no automated evaluation, no fixed random seed, and no reference group compared against the base model. This means I cannot answer the most basic question: compared to the unfine-tuned base model, did fine-tuning actually improve anything?

7. Next Steps: Constructing Domain Datasets

7.1 Why Domain Data is Mandatory

General distillation data can maintain the model's "expression style," but it cannot teach it "under warehouse scenarios, what conditions dictate prioritizing what." These criteria come from business rules, such as:

• If battery level falls below a certain safety threshold, it indicates a risk of mid-way breakdown and blocking the channel, giving it higher priority than distance factors;
• When meeting head-on in a single-lane corridor, priority to back up should be given to the side with the lowest cost of yielding (closer to an avoidance point, lighter load, lower task priority);
• Urgent tasks take higher priority than regular tasks, but their impact on global congestion must be weighed.

Such rules are best written into data in a deterministic way: standard answers are calculated by code based on rules and then packaged into natural language Q&A, rather than letting the model "freestyle" and then manually checking it.

7.2 Convergent Thinking Templates

To address the "never-ending thinking" issue, I designed a fixed four-step structure as the format for <think> in the training data:

1. State Extraction: List facts given in the problem (vehicle positions, battery levels, task levels, constraints);
2. Rule Matching: Point out applicable scheduling rules or safety red lines;
3. Contrastive Elimination: Quantitatively compare options and eliminate those failing conditions;
4. Explicit Conclusion: State instructions directly without expanding into new assumptions.

The key point is to prohibit introducing hypothetical values not present in the problem (e.g., "assuming charging speed is 10% per hour"). This habit is precisely the root cause of the failure case in Section 5.

7.3 Trilingual Consistency

Ensure every sample has "query language = thinking language = response language," and validate via code:

import re

CJK  = re.compile(r"[\u4e00-\u9fff]")
KANA = re.compile(r"[\u3040-\u30ff]")

def language_consistent(lang: str, think: str) -> bool:
    if lang == "en":
        return not CJK.search(think)          # English thinking should not contain CJK characters
    if lang == "ja":
        return bool(KANA.search(think))       # Japanese thinking must contain kana
    if lang == "zh":
        return bool(CJK.search(think)) and not KANA.search(think)
    return False

This check is crude (e.g., Japanese also uses many kanji), but sufficient to stop the most severe errors like "Japanese questions with entirely Chinese thinking."

7.4 Quality Gating

Filter uniformly before writing to the dataset:

• <think> and </think> each appear exactly once in correct order;
• Thinking body length is restricted to a shorter range (I currently set 30–1000 characters), helping the model form the habit of "stopping when logic is sufficient";
• Must have a non-empty conclusion following </think>;
• Pass the language consistency check above.

7.5 Principles Regarding Proprietary Data

If internal company materials are used as data sources in the future, I set several bottom lines for myself:

• Raw materials never leave local storage and are not uploaded to any external services; cleaning and transcription are done completely offline;
• De-identification is performed before transcription (company names, client names, intranet addresses replaced with generic words), and the raw material directory is added to .gitignore to prevent accidental commits;
• For unstructured documents, use a local model for distillation without relying on cloud APIs.

I wrote an offline script for this purpose, which is currently just a framework and hasn't been fully run with real data, let alone verified for effectiveness. This needs to be stated honestly.

8. Evaluation System: Prior to the Second Round of Training

Before conducting the second round of training, I plan to set up evaluations first; otherwise, I will fall back into judging quality by "gut feeling." The plan is as follows:

1. Fixed Evaluation Set: Covers three languages and several scenarios (avoidance, charging, operations research calculations, troubleshooting), with multiple items per category. Questions and standard answers are generated via rule-based code to ensure objective answers;
2. Automatic Metrics:
   • Conclusion accuracy (whether the answer matches the standard answer);
   • <think> closure rate;
   • Average thinking length and ratio of truncation triggers;
   • Language consistency rate;
3. Control Group: The same batch of questions run across the unfine-tuned base model, first-version LoRA, and second-version LoRA using identical decoding parameters;
4. Fixed Decoding Settings: Low temperature or greedy decoding, fixed random seed, ensuring reproducible results.

With this setup, every hypothesis in Section 6 can be verified one by one via single-variable ablation (modifying only language, only template, or only loss mask) rather than changing many things at once and guessing which one worked.

9. Thoughts on "What LLMs Can Do in Business"

During this project, I often wondered: assuming hundreds or thousands of robots are running in the same warehouse, where should a 7B model sit in the system?

My current view is: LLMs should not be placed in the real-time control loop.

• Path planning, laser obstacle avoidance, and normal intersection passage all have mature deterministic algorithms with millisecond latency, predictable, verifiable, and replayable behaviors. LLMs are slow, unstable, and offer no advantages here;
• LLMs are better suited for handling low-frequency, ambiguous scenarios requiring synthesized context: for example, helping sort out troubleshooting steps when multiple alerts occur simultaneously, querying system status in natural language, or providing "suggestions" for complex conflict zones.

If integration with a control system is truly necessary, I believe several layers of protection are required:

1. Event Trigger + Regional Aggregation: Do not let each vehicle request the model individually. Package the states of multiple vehicles within a conflict zone into a snapshot for a one-time decision, avoiding localized uncoordinated actions and drastically reducing request volume;
2. Structured Output + Validation: The model outputs only constrained structured commands (like JSON schema), and deterministic code performs legality checks and safety boundary validations. Rejection or fallback to traditional scheduling happens if it fails;
3. Human-in-the-Loop: Irreversible operations like forced stops or inventory modifications must require manual confirmation;
4. Full Logging: Every model input, output, and final execution result is logged for post-mortem review and continuous dataset improvement.

This also echoes the philosophy of Section 7: "bad cases" occurring in the field should be fed back into the next round of training and evaluation data, forming a closed loop.

10. Summary and Next Steps

Looking back at this project, my biggest takeaways are threefold:

1. The toolchain is extremely mature. From LoRA fine-tuning, quantization, to local inference, a single person can complete the entire pipeline in their spare time at very low cost;
2. Difficulties have shifted to data and evaluation. Training took only 5 minutes, while figuring out "what data to feed and how to judge quality" took up the real bulk of time;
3. Consistency is the source of many hidden problems. Templates, System Prompts, and language distributions between training and inference—if even one of them misaligns, the symptom manifests as "the model mysteriously getting dumber," making troubleshooting extremely difficult.

Next plans, prioritized:

• Unify chat templates for training/evaluation/inference, fixing format inconsistencies;
• Establish a trilingual evaluation set with standard answers and automated evaluation scripts;
• Run a baseline on the base model to quantify the gap "before and after fine-tuning";
• Construct a rule-driven trilingual domain dataset, adding language consistency and length quality gating;
• Try response-only loss and verify the actual contribution of each modification via single-variable ablation;
• Make the training script truly read configuration files and clean up unverified numbers in documentation.

The project is still rough. Writing it down is partly to organize my thoughts, and partly hoping to help friends transitioning from backend development to LLMs: get data and evaluation consistency right first, before pursuing larger models and more compute.

If you disagree with any judgments in this article, or notice any misunderstandings on my part, corrections are most welcome.