HD-RL: Human Decision RL Policy (LoRA Adapter)

This repository contains a LoRA adapter trained using reinforcement learning framework for optimizing LLM explanations for human decision making.

The model is designed to analyze a given text and generate explanation hints/bullets outlining linguistic markers of AI or human authorship.

---

Model Details

• Base Model: Qwen/Qwen3.6-27B
• Adapter Type: LoRA (Low-Rank Adaptation)
• Task: Generating strict, flat-list linguistic explanation hints based on specific formatting templates.

---

Output Constraints & Format

When prompted correctly, the policy strictly follows these constraints:

1. Starts the response immediately with a bulleted list (2 to 6 bullet points).
2. Does not include an introduction, final verdict, or summary.
3. Keeps it a flat list (no sub-headings).
4. Follows precise templates:
   • - "Look for [linguistic category or type of phrasing]."
   • - "Look for [linguistic category or type of phrasing] like '[short example]'."
5. Avoids bolding, italics, or other special Markdown formatting.

---

How to Run

To use this model for inference, you can load the base model via transformers and apply the LoRA weights using peft.

Prerequisites

Make sure you have the required libraries installed:

pip install torch transformers peft accelerate

import os

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# ============================================================================
# Configuration
# ============================================================================
POLICY_MODEL_NAME = "Qwen/Qwen3.6-27B"
POLICY_CACHE_DIR = "..."
LORA_ADAPTER_PATH = "Tizzzzy/HD-RL"

# Generation parameters
MAX_NEW_TOKENS = 512
TEMPERATURE = 1.0
TOP_P = 1.0
REPETITION_PENALTY = 1.1
DO_SAMPLE = True

# Policy prompt template (same as used in training)
POLICY_INSTRUCTION = """Task: Analyze the provided text for linguistic markers of AI or human authorship.

Constraints:
1. Start the response IMMEDIATELY with the bulleted list.
2. Do NOT provide an introduction, a final verdict, or a summary.
3. Do NOT group the bullet points into sub-headings (keep it a flat list).
4. Every bullet point MUST follow one of these two exact templates:
    - "Look for [linguistic category or type of phrasing]."
    - "Look for [linguistic category or type of phrasing] like '[short example]'."
5. When providing examples, you may list multiple words separated by commas (e.g., like "word 1", "word 2").
6. Do NOT use bolding, italics, or any special Markdown formatting.
7. Limit the output to exactly 2 to 6 bullet points.

Text:

{text}

Can you provide an explanation that tells the reader what to focus on?

Explanation:
"""


# ============================================================================
# Functions
# ============================================================================
def load_policy_model_and_tokenizer():
    """
    Load the trained policy model (Qwen3.6-27B + LoRA adapter from the HF Hub).

    Returns:
        (policy_model, policy_tokenizer)
    """
    print(f"\n[Model] Loading tokenizer from {POLICY_MODEL_NAME}")
    policy_tokenizer = AutoTokenizer.from_pretrained(
        POLICY_MODEL_NAME,
        cache_dir=POLICY_CACHE_DIR,
        trust_remote_code=True,
    )
    if policy_tokenizer.pad_token is None:
        policy_tokenizer.pad_token = policy_tokenizer.eos_token
    print("[Model] Tokenizer loaded")

    print(f"[Model] Loading base model {POLICY_MODEL_NAME}")
    policy_model = AutoModelForCausalLM.from_pretrained(
        POLICY_MODEL_NAME,
        torch_dtype=torch.bfloat16,
        device_map="auto",
        cache_dir=POLICY_CACHE_DIR,
        trust_remote_code=True,
    )
    print(f"[Model] Base model loaded: {policy_model.num_parameters():,} parameters")

    print(f"[Model] Loading LoRA adapter from {LORA_ADAPTER_PATH}")
    policy_model = PeftModel.from_pretrained(policy_model, LORA_ADAPTER_PATH)
    print("[Model] LoRA adapter loaded and merged")

    policy_model.eval()
    for param in policy_model.parameters():
        param.requires_grad = False

    return policy_model, policy_tokenizer


def build_policy_prompt(text: str, tokenizer) -> str:
    """Build the policy prompt (chat-templated) and disable thinking."""
    messages = [{"role": "user", "content": POLICY_INSTRUCTION.format(text=text)}]

    try:
        return tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=True,
            enable_thinking=False,
        )
    except TypeError:
        # Fallback for some transformer versions
        return tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=True,
            chat_template_kwargs={"enable_thinking": False},
        )


@torch.no_grad()
def generate_explanation(
    policy_model,
    policy_tokenizer,
    text: str,
    device: str = "cuda",
) -> str:
    """
    Generate a new explanation for the given text using the trained policy.

    Args:
        policy_model: The trained policy model
        policy_tokenizer: The tokenizer
        text: The input text to analyze
        device: Device to run on

    Returns:
        Generated explanation string (stripped of whitespace)
    """
    # Build the prompt
    prompt = build_policy_prompt(text, policy_tokenizer)

    # Tokenize
    inputs = policy_tokenizer(
        prompt,
        return_tensors="pt",
        truncation=True,
        max_length=5096,
    ).to(device)

    # Generate
    output_ids = policy_model.generate(
        **inputs,
        max_new_tokens=MAX_NEW_TOKENS,
        temperature=TEMPERATURE,
        top_p=TOP_P,
        repetition_penalty=REPETITION_PENALTY,
        do_sample=DO_SAMPLE,
        pad_token_id=policy_tokenizer.eos_token_id,
    )

    # Decode: extract only the generated part (remove prompt)
    prompt_len = inputs["input_ids"].shape[1]
    generated_ids = output_ids[0, prompt_len:]
    explanation = policy_tokenizer.decode(generated_ids, skip_special_tokens=True).strip()

    return explanation


def main():
    AI_sample_text = """I’m kind of stuck on my final project and could really use some guidance. The assignment is to build a CPU from scratch in Quartus, but honestly our professor never really taught us the tools or the workflow needed to do it. I’m not looking for someone to do the project for me, just help me understand what I’m supposed to be doing so I can actually learn it and finish it myself. This class matters a lot since my scholarship depends on my grades, and I’ve done well in everything else. I’m just trying to get through this one without falling apart."""
    HUMAN_sample_text = """I'm a recent MBA grad and I've been looking for a job for the past ~6 months. I've been getting interviews from time to time, but unfortunately nothing has stuck yet. I'm at the point where I'm just so sick of having nothing to work on besides job applications and I feel like I should be using this time to improve myself and/or increase my employability in some way. I've looked into business related certifications (PMP, SCMP, Lean Six Sigma etc. ) but they all seem to require a minimum level of hours within that field (which I don't have) or that you are currently working in that field (which I am not). I realize the answer to the question is going to depend on what direction I'd like to take with my career, but broadly speaking, and there any business related certifications/courses I should be looking into that would not only boost my resume, but also give me a positive explanation for what I have been doing in lieu of working for the past several months? I've already been working on polishing my Excel skills but I'm just looking for a bit more of a tangible accreditation that is recognized by employers which would help distinguish me from other applicants."""
    policy_model, policy_tokenizer = load_policy_model_and_tokenizer()
    device = next(policy_model.parameters()).device

    explanation = generate_explanation(policy_model, policy_tokenizer, sample_text, device=str(device))

    print("\n--- Generated Explanation ---")
    print(explanation)


if __name__ == "__main__":
    main()