add the card and the run config and metrics

README.md

@@ -0,0 +1,66 @@
+---
+license: apache-2.0
+language:
+  - en
+base_model: google/flan-t5-large
+tags:
+  - text2text-generation
+  - fact-updating
+  - wikipedia
+library_name: transformers
+pipeline_tag: text2text-generation
+datasets:
+  - gamemaker1/wiki-wrong
+---
+
+# DiffusEdit editor baseline
+
+`google/flan-t5-large` (780M parameters) fine-tuned end to end to rewrite an
+outdated article from evidence, in the style of EDIT5 (Iv et al., 2022).
+It is the fine-tuned upper bound that DiffusEdit is compared against, and
+every parameter was trained, unlike the DiffusEdit classifiers. The code is at
+https://github.com/gamemaker1/diffusedit under
+`code/pipeline/experiments/editor.py`.
+
+## Input and output
+
+```
+input    update: [0] <sentence 0> [1] <sentence 1> ... evidence: (0) <title>: <text> (1) ...
+output   [0] #1 <rewritten sentence 1> [2] ...
+```
+
+The output names each source sentence in order: `[j]` copies sentence j,
+`#j text` replaces it, and a missing j deletes it. The input is cut at 1,024
+tokens from the end and the output at 384 tokens.
+
+## Training
+
+Fit on the 12,180 train articles of `gamemaker1/wiki-wrong`, with
+the validation loss after each epoch for early stopping (patience 1). Learning
+rate 1e-4 with linear decay and 5% warm-up, 32 examples per optimiser step
+(micro-batches of 4, 8 accumulated), gradient clipping at 1.0, bf16 autocast,
+seed 42. The best epoch was the third of four, with validation loss 0.0439,
+after 64 minutes on one A100 40GB.
+
+## Test scores
+
+On the 1,670 supported EDIT sentences of the test split, against the gold
+rewrite: ROUGE-L 0.973, exact match 76.7%, new-value recall 88.7%, old-value
+kept 10.0%. Copying the source scores ROUGE-L 0.875. Greedy decoding writes
+an article in 0.35 seconds.
+
+## Use
+
+```python
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+tok = AutoTokenizer.from_pretrained("gamemaker1/t5edit")
+model = AutoModelForSeq2SeqLM.from_pretrained("gamemaker1/t5edit")
+```
+
+`experiments.editor.input_text` builds the input from a dataset record and
+`experiments.editor.parse` maps the output back to one sentence per source
+sentence.
+
+## Licence
+
+Apache 2.0, as the base model.

run_config.json

@@ -0,0 +1,23 @@
+{
+  "train": "/home/diffusedit/data/split/train.jsonl",
+  "val": "/home/diffusedit/data/split/val.jsonl",
+  "test": "/home/diffusedit/data/split/test.jsonl",
+  "out": "/home/diffusedit/full/runs/editor",
+  "model": "google/flan-t5-large",
+  "predict_only": false,
+  "max_input": 1024,
+  "max_new_tokens": 384,
+  "lr": 0.0001,
+  "batch": 4,
+  "accum": 8,
+  "epochs": 5,
+  "patience": 1,
+  "predict_batch": 8,
+  "gradient_checkpointing": true,
+  "log_every": 50,
+  "limit": null,
+  "seed": 42,
+  "n_train": 12180,
+  "n_val": 1564,
+  "n_test": 1602
+}

run_metrics.json

@@ -0,0 +1,66 @@
+{
+  "train_lengths": {
+    "input": {
+      "p50": 193.0,
+      "p90": 338.0,
+      "p95": 389.0,
+      "p99": 507.2099999999991,
+      "mean": 213.64573070607554,
+      "max": 1520,
+      "over": 7
+    },
+    "target": {
+      "p50": 27.0,
+      "p90": 90.0,
+      "p95": 111.0,
+      "p99": 157.0,
+      "mean": 39.33522167487685,
+      "max": 970,
+      "over": 6
+    },
+    "evidence_cut": 4,
+    "article_cut": 3
+  },
+  "val_history": [
+    {
+      "epoch": 1,
+      "train_loss": 0.20339470065752346,
+      "val_loss": 0.059932903280978694
+    },
+    {
+      "epoch": 2,
+      "train_loss": 0.060603164612924705,
+      "val_loss": 0.04503743698826844
+    },
+    {
+      "epoch": 3,
+      "train_loss": 0.0403840439127009,
+      "val_loss": 0.04387998025125058
+    },
+    {
+      "epoch": 4,
+      "train_loss": 0.029914886026241098,
+      "val_loss": 0.044346317642450714
+    }
+  ],
+  "best_val_loss": 0.04387998025125058,
+  "epochs_run": 4,
+  "best_epoch": 3,
+  "train_seconds": 3814.4,
+  "articles": 1602,
+  "fallbacks": 3,
+  "fallback_reasons": {
+    "copy_text": 3
+  },
+  "articles_with_fallback": 3,
+  "restored_words": 109,
+  "sentences": {
+    "copied": 3565,
+    "rewritten": 1630,
+    "deleted": 417
+  },
+  "mean_output_words": 61.51622971285892,
+  "mean_output_sentences": 3.2428214731585516,
+  "mean_raw_chars": 136.54681647940075,
+  "seconds_per_article": 0.3500464130727837
+}