PySparkQwen: Qwen/Qwen2.5-Coder-1.5B-Instruct Fine-Tuning with QLoRA

QLoRA fine-tuning of Qwen/Qwen2.5-Coder-1.5B-Instruct for PySpark code generation.

PySparkQwen is a PySpark coding model, that is used for data engineering tasks, including DataFrame transformations, joins, Spark SQL, structured streaming, Delta Lake, performance optimization, MLlib, and distributed processing.

Given:

  • PySpark task (prompt)
  • DataFrame or table schemas, column names (and/or relevant runtime constraints)

Training note: The base model was loaded with 4-bit NF4 quantization and adapted locally using parameter-efficient LoRA weights. This repository uses a PEFT adapter, not a standalone copy of the base model.

Example

DataFrame context:

DataFrame: orders
Columns: customer_id (string), order_id (string), amount (double)

Prompt:

Calculate total revenue per customer and return the five customers with the highest revenue.

Expected Model Output:

from pyspark.sql import functions as F

result = (
    orders.groupBy("customer_id")
    .agg(F.sum("amount").alias("total_revenue"))
    .orderBy(F.col("total_revenue").desc())
    .limit(5)
)

Dataset: Synthetic PySpark Code Tasks

Dataset statistic Count
Total examples 900
Training examples 600
Validation examples 150
Held-out test examples 150
Task categories 15
Algorithm families 180

Dataset is balanced across 15 task categories, with 60 examples per category (40 training, 10 validation, and 10 test). It covers:

  1. DataFrame creation and transformations
  2. Data ingestion and file formats
  3. Data cleaning and preprocessing
  4. Filtering, sorting, and aggregation
  5. Joins and window functions
  6. Spark SQL
  7. UDFs and Pandas UDFs
  8. Structured Streaming
  9. Performance optimization and caching
  10. Partitioning and repartitioning
  11. Machine learning with MLlib
  12. Nested and complex data types
  13. Error handling and debugging
  14. Delta Lake operations
  15. Distributed computing and cluster execution

Note: All 900 reference snippets passed Python ast.parse syntax checks.

Model Evaluation

The original Qwen2.5-Coder model and the fine-tuned adapter were evaluated on the same 150 held-out synthetic tasks, using the same prompt construction and greedy decoding (do_sample=False) with an output limit of 384 new tokens.

Greedy decoding selects the highest-probability next token at each step, making this comparison deterministic under the same runtime conditions. It does not search for the globally best full program.

Held-out Test Set (150 Examples)

Metric Base Qwen2.5-Coder-1.5B-Instruct Fine-tuned PySparkQwen-Coder
Evaluated examples 150 150
Python AST-valid outputs 90.00% (135/150) 98.67% (148/150)
Mean character-sequence similarity to reference 0.1168 0.6621

Model Limitations

  • Missing context: In rare cases (without context): The model may invent columns, tables, file paths, or dependencies if the task or schema is incomplete.
  • Review before use: Generated code should be reviewed and tested in a controlled Spark environment before running against production data.

Example Import (PEFT)

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# Model repositories
base_model = "Qwen/Qwen2.5-Coder-1.5B-Instruct"
adapter_model = "AaronTekle/PySparkQwen"

# Load the original Qwen2.5-Coder tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# Configure 4-bit quantization for lower GPU memory usage
compute_dtype = (
    torch.bfloat16
    if torch.cuda.is_bf16_supported()
    else torch.float16
)

quantization = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=compute_dtype,
)

# Load the base Qwen2.5-Coder model
base_model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=quantization,
    device_map="auto",
)

# Load the fine-tuned PySparkQwen LoRA adapter
model = PeftModel.from_pretrained(
    base_model,
    adapter_model,
)
model.eval()

# Example PySpark coding task
context = (
    "DataFrame orders: customer_id string, "
    "order_id string, amount double"
)

question = (
    "Calculate total revenue per customer "
    "and show the top five customers."
)

# Define coding assistant behavior
system_prompt = (
    "You are a PySpark data engineering coding assistant. "
    "Return only executable Python/PySpark code with necessary imports. "
    "Prefer Spark DataFrame APIs over pandas and RDDs. "
    "Use the provided schemas and names without inventing tables or columns. "
)

messages = [
    {"role": "system", "content": system_prompt},
    {
        "role": "user",
        "content": f"{question}\n\nContext / schema:\n{context}",
    },
]

# Apply Qwen2.5-Coder's native instruct chat template
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

# Tokenize the prompt and move tensors to the model's device
inputs = tokenizer(
    prompt,
    return_tensors="pt",
).to(model.get_input_embeddings().weight.device)

# Generate PySpark code using greedy decoding
with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=384,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
    )

# Decode only newly generated tokens
new_tokens = output_ids[0][inputs["input_ids"].shape[-1]:]

generated_code = tokenizer.decode(
    new_tokens,
    skip_special_tokens=True,
).strip()

print("Generated PySpark Code:")
print(generated_code)

References

Downloads last month
6
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for AaronTekle/PySparkQwen

Adapter
(237)
this model

Papers for AaronTekle/PySparkQwen