
%pip install transformers[torch,ja]==4.33.3 datasets==2.14.5 sentencepiece matplotlib seaborn evaluate absl-py bert_score pandas tokenizers==0.13.3

Collecting transformers[ja,torch]==4.33.3
  Downloading transformers-4.33.3-py3-none-any.whl (7.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/7.6 MB 51.3 MB/s eta 0:00:00
Collecting datasets==2.14.5
  Downloading datasets-2.14.5-py3-none-any.whl (519 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 519.6/519.6 kB 52.3 MB/s eta 0:00:00
Collecting sentencepiece
  Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 62.1 MB/s eta 0:00:00
Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (3.7.1)
Requirement already satisfied: seaborn in /usr/local/lib/python3.10/dist-packages (0.12.2)
Collecting evaluate
  Downloading evaluate-0.4.1-py3-none-any.whl (84 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.1/84.1 kB 11.4 MB/s eta 0:00:00
Requirement already satisfied: absl-py in /usr/local/lib/python3.10/dist-packages (1.4.0)
Collecting bert_score
  Downloading bert_score-0.3.13-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.1/61.1 kB 7.8 MB/s eta 0:00:00
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (1.5.3)
Collecting tokenizers==0.13.3
  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 100.9 MB/s eta 0:00:00
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers[ja,torch]==4.33.3) (3.12.4)
Collecting huggingface-hub<1.0,>=0.15.1 (from transformers[ja,torch]==4.33.3)
  Downloading huggingface_hub-0.18.0-py3-none-any.whl (301 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 302.0/302.0 kB 32.5 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers[ja,torch]==4.33.3) (1.23.5)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers[ja,torch]==4.33.3) (23.2)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers[ja,torch]==4.33.3) (6.0.1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers[ja,torch]==4.33.3) (2023.6.3)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers[ja,torch]==4.33.3) (2.31.0)
Collecting safetensors>=0.3.1 (from transformers[ja,torch]==4.33.3)
  Downloading safetensors-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 78.6 MB/s eta 0:00:00
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers[ja,torch]==4.33.3) (4.66.1)
Requirement already satisfied: torch!=1.12.0,>=1.10 in /usr/local/lib/python3.10/dist-packages (from transformers[ja,torch]==4.33.3) (2.1.0+cu118)
Collecting accelerate>=0.20.3 (from transformers[ja,torch]==4.33.3)
  Downloading accelerate-0.24.1-py3-none-any.whl (261 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 261.4/261.4 kB 30.3 MB/s eta 0:00:00
Collecting fugashi>=1.0 (from transformers[ja,torch]==4.33.3)
  Downloading fugashi-1.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (600 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 600.9/600.9 kB 53.2 MB/s eta 0:00:00
Collecting ipadic<2.0,>=1.0.0 (from transformers[ja,torch]==4.33.3)
  Downloading ipadic-1.0.0.tar.gz (13.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.4/13.4 MB 77.9 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting unidic-lite>=1.0.7 (from transformers[ja,torch]==4.33.3)
  Downloading unidic-lite-1.0.8.tar.gz (47.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.4/47.4 MB 16.9 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting unidic>=1.0.2 (from transformers[ja,torch]==4.33.3)
  Downloading unidic-1.1.0.tar.gz (7.7 kB)
  Preparing metadata (setup.py) ... done
Collecting sudachipy>=0.6.6 (from transformers[ja,torch]==4.33.3)
  Downloading SudachiPy-0.6.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 24.0 MB/s eta 0:00:00
Collecting sudachidict-core>=20220729 (from transformers[ja,torch]==4.33.3)
  Downloading SudachiDict_core-20230927-py3-none-any.whl (71.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.7/71.7 MB 8.8 MB/s eta 0:00:00
Collecting rhoknp<1.3.1,>=1.1.0 (from transformers[ja,torch]==4.33.3)
  Downloading rhoknp-1.3.0-py3-none-any.whl (86 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.8/86.8 kB 11.5 MB/s eta 0:00:00
Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets==2.14.5) (9.0.0)
Collecting dill<0.3.8,>=0.3.0 (from datasets==2.14.5)
  Downloading dill-0.3.7-py3-none-any.whl (115 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.3/115.3 kB 14.0 MB/s eta 0:00:00
Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets==2.14.5) (3.4.1)
Collecting multiprocess (from datasets==2.14.5)
  Downloading multiprocess-0.70.15-py310-none-any.whl (134 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 15.5 MB/s eta 0:00:00
Requirement already satisfied: fsspec[http]<2023.9.0,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from datasets==2.14.5) (2023.6.0)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets==2.14.5) (3.8.6)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.1.1)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (4.43.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.4.5)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (9.4.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (3.1.1)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (2.8.2)
Collecting responses<0.19 (from evaluate)
  Downloading responses-0.18.0-py3-none-any.whl (38 kB)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2023.3.post1)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.20.3->transformers[ja,torch]==4.33.3) (5.9.5)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets==2.14.5) (23.1.0)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets==2.14.5) (3.3.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets==2.14.5) (6.0.4)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets==2.14.5) (4.0.3)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets==2.14.5) (1.9.2)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets==2.14.5) (1.4.0)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets==2.14.5) (1.3.1)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.15.1->transformers[ja,torch]==4.33.3) (4.5.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers[ja,torch]==4.33.3) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers[ja,torch]==4.33.3) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers[ja,torch]==4.33.3) (2023.7.22)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch!=1.12.0,>=1.10->transformers[ja,torch]==4.33.3) (1.12)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch!=1.12.0,>=1.10->transformers[ja,torch]==4.33.3) (3.2)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch!=1.12.0,>=1.10->transformers[ja,torch]==4.33.3) (3.1.2)
Requirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.10/dist-packages (from torch!=1.12.0,>=1.10->transformers[ja,torch]==4.33.3) (2.1.0)
Collecting wasabi<1.0.0,>=0.6.0 (from unidic>=1.0.2->transformers[ja,torch]==4.33.3)
  Downloading wasabi-0.10.1-py3-none-any.whl (26 kB)
Collecting plac<2.0.0,>=1.1.3 (from unidic>=1.0.2->transformers[ja,torch]==4.33.3)
  Downloading plac-1.4.1-py2.py3-none-any.whl (22 kB)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch!=1.12.0,>=1.10->transformers[ja,torch]==4.33.3) (2.1.3)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch!=1.12.0,>=1.10->transformers[ja,torch]==4.33.3) (1.3.0)
Building wheels for collected packages: ipadic, unidic, unidic-lite
  Building wheel for ipadic (setup.py) ... done
  Created wheel for ipadic: filename=ipadic-1.0.0-py3-none-any.whl size=13556703 sha256=c734594f6ebc044bf9e2ba4e0351f66bffea4a62d1c1c56f2c5445abd1449190
  Stored in directory: /root/.cache/pip/wheels/5b/ea/e3/2f6e0860a327daba3b030853fce4483ed37468bbf1101c59c3
  Building wheel for unidic (setup.py) ... done
  Created wheel for unidic: filename=unidic-1.1.0-py3-none-any.whl size=7406 sha256=21c1c4b149b24a152647c2b7122b508f03f674ab6c7275fbd942ecf8992bfd0b
  Stored in directory: /root/.cache/pip/wheels/7a/72/72/1f3d654c345ea69d5d51b531c90daf7ba14cc555eaf2c64ab0
  Building wheel for unidic-lite (setup.py) ... done
  Created wheel for unidic-lite: filename=unidic_lite-1.0.8-py3-none-any.whl size=47658816 sha256=bbdeafc6a0e71520c94eb7be23a273d26c3e3b6792b08dcaba6da4902f71935e
  Stored in directory: /root/.cache/pip/wheels/89/e8/68/f9ac36b8cc6c8b3c96888cd57434abed96595d444f42243853
Successfully built ipadic unidic unidic-lite
Installing collected packages: wasabi, unidic-lite, tokenizers, sudachipy, sentencepiece, plac, ipadic, sudachidict-core, safetensors, rhoknp, fugashi, dill, unidic, responses, multiprocess, huggingface-hub, transformers, accelerate, datasets, bert_score, evaluate
  Attempting uninstall: wasabi
    Found existing installation: wasabi 1.1.2
    Uninstalling wasabi-1.1.2:
      Successfully uninstalled wasabi-1.1.2
Successfully installed accelerate-0.24.1 bert_score-0.3.13 datasets-2.14.5 dill-0.3.7 evaluate-0.4.1 fugashi-1.3.0 huggingface-hub-0.18.0 ipadic-1.0.0 multiprocess-0.70.15 plac-1.4.1 responses-0.18.0 rhoknp-1.3.0 safetensors-0.4.0 sentencepiece-0.1.99 sudachidict-core-20230927 sudachipy-0.6.7 tokenizers-0.13.3 transformers-4.33.3 unidic-1.1.0 unidic-lite-1.0.8 wasabi-0.10.1


import torch

if torch.cuda.is_available():
    status = "GPU is enabled."
    device_count = torch.cuda.device_count()
    current_device = torch.cuda.current_device()
    print(f"{status}\ndevice count: {device_count}, current device: {current_device}")
else:
    print("GPU is disabled.")

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"device: {device}")

GPU is enabled.
device count: 1, current device: 0
device: cuda

GPU is disabled.
Device: cpu


import random
import numpy as np
import torch

import warnings
warnings.filterwarnings('ignore')
def seed_everything(seed_value):
    random.seed(seed_value) # Python
    np.random.seed(seed_value) # Numpy
    torch.manual_seed(seed_value) # CPU
    if torch.cuda.is_available():
        torch.cuda.manual_seed(seed_value)
        torch.cuda.manual_seed_all(seed_value) # GPU if available
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False

seed_value = 42
seed_everything(seed_value)


from datasets import load_dataset

dataset = load_dataset(
    "llm-book/livedoor-news-corpus",
    train_ratio = 0.8,
    validation_ratio = 0.1,
    seed=42,
    shuffle=False,

)

Downloading builder script:   0%|          | 0.00/3.88k [00:00<?, ?B/s]

Downloading readme:   0%|          | 0.00/823 [00:00<?, ?B/s]

Downloading data:   0%|          | 0.00/8.86M [00:00<?, ?B/s]

Generating train split: 0 examples [00:00, ? examples/s]

Generating validation split: 0 examples [00:00, ? examples/s]

Generating test split: 0 examples [00:00, ? examples/s]


dataset

DatasetDict({
    train: Dataset({
        features: ['url', 'date', 'title', 'content', 'category'],
        num_rows: 5893
    })
    validation: Dataset({
        features: ['url', 'date', 'title', 'content', 'category'],
        num_rows: 736
    })
    test: Dataset({
        features: ['url', 'date', 'title', 'content', 'category'],
        num_rows: 738
    })
})


import pandas as pd
test_df = pd.DataFrame(dataset['test'])
test_df.head()


from transformers import AutoTokenizer

MODEL_NAME = "google/mt5-small"

mt5_tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
mt5_tokenizer.deprecation_warnings["Asking-to-pad-a-fast-tokenizer"] = True

(…)small/resolve/main/tokenizer_config.json:   0%|          | 0.00/82.0 [00:00<?, ?B/s]

(…)oogle/mt5-small/resolve/main/config.json:   0%|          | 0.00/553 [00:00<?, ?B/s]

(…)ogle/mt5-small/resolve/main/spiece.model:   0%|          | 0.00/4.31M [00:00<?, ?B/s]

(…)all/resolve/main/special_tokens_map.json:   0%|          | 0.00/99.0 [00:00<?, ?B/s]

You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565


text = "本日はAIトレーニングセッションへようこそ！"

encoded_text = mt5_tokenizer(text)
print("Encoded text: ", encoded_text)

tokenized_text = mt5_tokenizer.tokenize(text)
print("Tokenized text: ", tokenized_text)

decoded_text = mt5_tokenizer.decode(encoded_text["input_ids"], skip_special_tokens=True)
print("Decoded Text: ", decoded_text)

Encoded text:  {'input_ids': [259, 212152, 15428, 96992, 191286, 6031, 15578, 68875, 309, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
Tokenized text:  ['▁', '本日は', 'AI', 'トレーニング', 'セッション', 'へ', 'よう', 'こそ', '!']
Decoded Text:  本日はAIトレーニングセッションへようこそ!


import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
def dist_info(text, max_bin_size=1024):
    token_counts = [len(mt5_tokenizer.tokenize(content)) for content in text]

    sns.histplot(token_counts, bins=100, binrange=(0, max_bin_size))
    plt.title("Tokenized Text Length Distribution")
    plt.xlabel("Tokenized Text Length")
    plt.ylabel("Count")
    plt.show()

    percentiles = [25, 50, 75, 90, 95, 99]
    for p in percentiles:
        print(f"{p}% of the dataset has token count below: {np.percentile(token_counts, p)}")


dist_info(dataset["train"]["content"])

25% of the dataset has token count below: 398.0
50% of the dataset has token count below: 589.0
75% of the dataset has token count below: 815.0
90% of the dataset has token count below: 1066.0
95% of the dataset has token count below: 1302.0
99% of the dataset has token count below: 1969.08


dist_info(dataset["train"]["title"], max_bin_size=128)

25% of the dataset has token count below: 16.0
50% of the dataset has token count below: 21.0
75% of the dataset has token count below: 25.0
90% of the dataset has token count below: 29.800000000000182
95% of the dataset has token count below: 32.0
99% of the dataset has token count below: 40.0


SOURCE_MAX_LEN = 512
TARGET_MAX_LEN = 64


NEWLINE_CHAR = "\n"
SPACE_CHAR = "\u3000"
TAB_CHAR = "\t"
CARRIAGE_RETURN_CHAR = "\r"

def text_clean_preprocess(text, newline_char=NEWLINE_CHAR, space_char=SPACE_CHAR, tab_char=TAB_CHAR, carriage_return_char=CARRIAGE_RETURN_CHAR):
    text = text.replace(newline_char, "")
    text = text.replace(space_char, " ")
    text = text.replace(tab_char, "")
    text = text.replace(carriage_return_char, "")
    text = text.lower()

    return text


def tokenize_data(data):

    input_text = [text_clean_preprocess(content) for content in data["content"]]

    target_text = data["title"]

    inputs = mt5_tokenizer(input_text, max_length=SOURCE_MAX_LEN, truncation=True, padding="max_length")
    targets = mt5_tokenizer(target_text, max_length=TARGET_MAX_LEN, truncation=True, padding="max_length")
    label_attention_mask = [1] * len(targets["input_ids"])

    return {
        "input_ids": inputs.input_ids,
        "attention_mask": inputs.attention_mask,
        "decoder_input_ids": targets["input_ids"],
        "labels": targets.input_ids,
        "label_attention_mask": label_attention_mask

    }


tokenized_dataset = dataset.map(
    tokenize_data,
    batched=True,
    remove_columns=["content", "title", "url", "date", "category"]
)

Map:   0%|          | 0/5893 [00:00<?, ? examples/s]

Map:   0%|          | 0/736 [00:00<?, ? examples/s]

Map:   0%|          | 0/738 [00:00<?, ? examples/s]


from transformers import AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained(
    MODEL_NAME,
    trust_remote_code=True,
)

pytorch_model.bin:   0%|          | 0.00/1.20G [00:00<?, ?B/s]

(…)mall/resolve/main/generation_config.json:   0%|          | 0.00/147 [00:00<?, ?B/s]


from transformers import DataCollatorForSeq2Seq

data_collator = DataCollatorForSeq2Seq(
    tokenizer=mt5_tokenizer,
    model=model,
)


import evaluate

def compute_bert_score_demo(preds, labels):
    bert_score_metric = evaluate.load("bertscore")
    bert_score_metric.add_batch(predictions=preds, references=labels)
    result = bert_score_metric.compute(lang="ja")
    avg_scores = {k: sum(v) / len(v) for k, v in result.items() if k != "hashcode"}

    return avg_scores

original = "リンゴが好きです。"
candidate_1 ="リンゴが大好きです。"
candidate_2 = "赤い車を買うつもりです。"

bs_results = {
    "候補1": compute_bert_score_demo([candidate_1], [original]),
    "候補2": compute_bert_score_demo([candidate_2], [original]),
}

bs_df = pd.DataFrame(bs_results).T
bs_df

Downloading builder script:   0%|          | 0.00/7.95k [00:00<?, ?B/s]

(…)cased/resolve/main/tokenizer_config.json:   0%|          | 0.00/29.0 [00:00<?, ?B/s]

(…)tilingual-cased/resolve/main/config.json:   0%|          | 0.00/625 [00:00<?, ?B/s]

(…)ultilingual-cased/resolve/main/vocab.txt:   0%|          | 0.00/996k [00:00<?, ?B/s]

model.safetensors:   0%|          | 0.00/714M [00:00<?, ?B/s]


def compute_bert_score(eval_preds):
    bert_score_metric = evaluate.load("bertscore")
    predictions, labels = eval_preds

    decoded_preds = mt5_tokenizer.batch_decode(predictions, skip_special_tokens=True)
    decoded_labels = mt5_tokenizer.batch_decode(labels, skip_special_tokens=True)

    result = bert_score_metric.compute(
        predictions=decoded_preds, references=decoded_labels, lang="ja"
    )

    return {
        "bertscore_precision": sum(result["precision"]) / len(result["precision"]),
        "bertscore_recall": sum(result["recall"]) / len(result["recall"]),
        "bertscore_f1": sum(result["f1"]) / len(result["f1"])
    }


from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments

NUM_EPOCHS = 5
LEARNING_RATE = 5e-4
WARMUP_RATIO = 0.1
PER_DEVICE_TRAIN_BATCH_SIZE = 8
PER_DEVICE_EVAL_BATCH_SIZE = 8
GRADIENT_ACCUMULATION_STEPS = 4
WEIGHT_DECAY = 0.01
LOGGING_DIR = "./logs"
OUTPUT_DIR = "./results"

training_args = Seq2SeqTrainingArguments(
    per_device_train_batch_size=PER_DEVICE_TRAIN_BATCH_SIZE,
    per_device_eval_batch_size=PER_DEVICE_EVAL_BATCH_SIZE,

    learning_rate=LEARNING_RATE,
    # lr_scheduler_type="linear", #comment on colab
    # warmup_ratio=0.1, #comment on colab

    num_train_epochs=NUM_EPOCHS,

    evaluation_strategy="epoch",
    save_strategy="epoch",
    logging_strategy="epoch",
    logging_steps=100,

    logging_dir=LOGGING_DIR,
    do_train=True,
    do_eval=True,
    output_dir=OUTPUT_DIR,

    save_total_limit=2,
    load_best_model_at_end=True,

    push_to_hub=False,
    predict_with_generate=True,

    gradient_accumulation_steps=GRADIENT_ACCUMULATION_STEPS,
)


trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["validation"],
    data_collator=data_collator,
    tokenizer=mt5_tokenizer,
    compute_metrics=compute_bert_score,
)


trainer.train()

TrainOutput(global_step=920, training_loss=2.273634487649669, metrics={'train_runtime': 2265.4763, 'train_samples_per_second': 13.006, 'train_steps_per_second': 0.406, 'total_flos': 1.556004590321664e+16, 'train_loss': 2.273634487649669, 'epoch': 4.99})


results = trainer.evaluate()

results_df = pd.DataFrame([results])
results_df


def generate_title(content):

    # inputs = [text_clean_preprocess(content)]
    inputs = [f"summarize: " + text_clean_preprocess(content)]

    batch = mt5_tokenizer.batch_encode_plus(
        inputs, max_length=512, truncation=True,
        padding="longest", return_tensors="pt")

    input_ids = batch['input_ids']
    input_mask = batch['attention_mask']


    model.eval()

    outputs = model.generate(
        input_ids=input_ids.cuda(),
        attention_mask=input_mask.cuda(),
        max_length=64,
        temperature=1.1, #
        num_beams=6, #24
        diversity_penalty=3.0, #1.8
        num_beam_groups=3,
        num_return_sequences=3,
        repetition_penalty=9.0,
        # early_stopping=True, #false
        # max_new_tokens=64,
        # do_sample = True
    )

    generated_titles = [mt5_tokenizer.decode(ids, skip_special_tokens=True,
                            clean_up_tokenization_spaces=False)
                        for ids in outputs]

    return generated_titles


print("Total titles in dataset:", len(dataset["test"]["title"]))

selected_index = [75, 140, 286]

for index in selected_index:
    print("original: ", dataset["test"]["title"][index])
    titles = generate_title(dataset["test"]["content"][index])
    for i, title in enumerate(titles):
        print(f"Generated title {i+1}: {title}")
    print()

Total titles in dataset: 738
original:  いいとも!で紹介された「ヒドすぎる」名前が話題に
Generated title 1: 『ザザザの斬新!赤ちゃんネーム』で紹介された「キラキラネーム」がありえない
Generated title 2: 『ザザザの斬新!赤ちゃんネーム』が「ありえない」とネットニュースで話題
Generated title 3: ネットスラングで「キラキラネーム」がありえないとネットで話題【話題】

original:  日本の引きこもりに海外から相次ぐ心配の声
Generated title 1: 日本の引きこもりが海外で話題に
Generated title 2: 日本の引きこもりが海外で話題
Generated title 3: 「日本=出る釘は打たれる」など、日本の引きこもりが海外で話題に

original:  甲子園出場する石巻工「約5000万円が必要」呼びかけに物議
Generated title 1: 【Sports Watch】石巻工業高校が総額約5000万円の協賛金を募っている
Generated title 2: 【Sports Watch】石巻工業高校が総額約5000万円の協賛金を募っている理由
Generated title 3: 石巻工業高校が、総額約5000万円の協賛金を募っている【話題】


# https://news.yahoo.co.jp/pickup/6476740

yahoo_news_1_original_title = """【速報】女川原発2号機「再稼働目標 3か月延期へ」来年5月に＜東北電力＞"""

yahoo_news_1 = """東北電力は来年2月を目標としていた「女川原発2号機の再稼働」について、来年5月に延期することを明らかにした。今年11月としていた安全対策工事の完了時期が来年2月に延びるため。

東北電力によると、工事が3か月延びるのは、発電所内の設備などにつながるケーブルが火災などで損傷しないようにする「火災防護対策」を追加したことが主な要因。この対策を巡っては、他の電力会社が原子力規制員会から指摘を受けた事例を踏まえ、東北電力では去年10月から追加で工事をすることを準備していた。
その工程を精査した結果、3か月ほど完了時期が延びることが判明したもので、それに伴って女川原発2号機の再稼働目標も3か月延期し、来年5月頃となった。
"""

print("Original yahoo title: ", yahoo_news_1_original_title)
for i, title in enumerate(generate_title(yahoo_news_1)):
    print(f"Generated title {i+1}: {title}")

Original yahoo title:  【速報】女川原発2号機「再稼働目標 3か月延期へ」来年5月に＜東北電力＞
Generated title 1: 東北電力、来年2月を目標としていた「女川原発2号機再稼働」を発表
Generated title 2: 東北電力、来年2月を目標としていた「女川原発2号機再稼働」について3か月延期
Generated title 3: 【ニュース】東北電力、来年2月を目標としていた「女川原発2号機再稼働」について3か月延期


sample_email = """
おはようございます。
アナハイム・エレクトロニクス事務局です。

本日は、まもなく締め切りとなります、10/18（水）開催の対面イベント「モルゲンレーテ社製品無料体験会＆相談会 ＠アナハイム・エレクトロニクスカリフォルニア本社」のご案内です！

実際に触れてみないとわからないとお困りの方、ぜひこの機会に体験してみてください。

モルゲンレーテ社製品無料体験会は特に以下のような方にお勧めです。

・電子・電気機器の新規導入でご検討中の方
　※小規模から対応できます！規模は問いません。
・電子機器の更改でクラウド移行を検討している方
・顧客管理システムも含めセキュアに電子機器を構築したい方。

モルゲンレーテ社製品を検討はしているけど、実際に触れてみないとわからない、導入を検討しているけど、何から始めればよいか分からないといった課題、お困りごとのある方は、ぜひこの機会に体験してみてください。

また、体験会の後は、弊社エンジニアの個別相談会も予定しております。

皆さまのお悩みなどお気軽にお話いただければと思います。

以下、無料体験会の詳細、申し込み方法をご確認のうえ、ぜひお気軽にご参加ください！

イベントの参加は無料！　参加お申込みは10月16日（月）となります。
みなさまのご参加お待ちしています！
"""

for i, title in enumerate(generate_title(sample_email)):
    print(f"Generated title {i+1}: {title}")

Generated title 1: アナハイム・エレクトロニクスカリフォルニア本社にて開催の対面イベント「モルゲンレーテ社製品無料体験会&相談会」を開催
Generated title 2: アナハイム・エレクトロニクスカリフォルニア本社にて開催の対面イベント「モルゲンレーテ社製品無料体験会&相談会】
Generated title 3: 「モルゲンレーテ社製品無料体験会&相談会 アナハイム・エレクトロニクスカリフォルニア本社」

	url	date	title	content	category
0	http://news.livedoor.com/article/detail/5936102/	2011-10-14T09:11:00+0900	ゼンショー「事実無根」と反論	10月13日の夜、ゼンショーの広報室長がTwitterで読売新聞の報道に「事実無根」と反論し...	topic-news
1	http://news.livedoor.com/article/detail/5936557/	2011-10-14T11:16:00+0900	「報ステ」OP曲演奏のジャズミュージシャンに“売名行為”と批判相次ぐ	先日、福島県が行っている新米の放射性物質本検査が全て終了した。規制値を超える放射性セシウムは...	topic-news
2	http://news.livedoor.com/article/detail/5936721/	2011-10-14T11:46:00+0900	「何のための“予約”なんですか」孫社長に批判殺到	ソフトバンクは、今朝から発売が始まった“iPhone4S”をはじめとする、ソフトバンク全ての...	topic-news
3	http://news.livedoor.com/article/detail/5937177/	2011-10-15T10:00:00+0900	あまりにも多すぎる「会いたくて」への皮肉か!? 「西野カナゲーム」が流行	いま巷で「西野カナゲーム」なるものが流行してるという。簡単に説明すると、1人目、2人目は...	topic-news
4	http://news.livedoor.com/article/detail/5937649/	2011-10-14T16:03:00+0900	憶測呼ぶ紳助さんの“天敵”引退	10月13日発売の東スポに「紳助の天敵が引退」との見出しが躍った。その天敵とは、警察庁の安藤...	topic-news

	precision	recall	f1
候補1	0.966469	0.986506	0.976384
候補2	0.680357	0.723368	0.701203

Epoch	Training Loss	Validation Loss	Bertscore Precision	Bertscore Recall	Bertscore F1
0	7.068200	1.034466	0.706013	0.664744	0.684008
1	1.268600	0.894248	0.712798	0.681310	0.696195
2	1.079000	0.864152	0.712160	0.684890	0.697850
4	0.996100	0.854636	0.713781	0.689665	0.701086
4	0.956100	0.847545	0.715506	0.692384	0.703341

AIによるメールの件名生成¶

1. お知らせ¶

1.2. Google ColabでGPUを有効にするには？¶

1.3. 参考情報について¶

2. 必要なライブラリのインストール¶

3. GPUが利用できるか確認¶

4. シードの設定 (乱数の制御)¶

5. Data¶

5.1 データの読み込み¶

5.2 Simple Data Exploration¶

¶

6. Model and Tokenizer¶

mt5とは?¶

Text to text トランスフォーマーとは?¶

6.1. Set Tokenizer¶

トークナイザとは?¶

トークナイザ詳細¶

トークナイザの使い方:¶

6.2 ニュース記事やタイトルのトークン数の分布¶

6.2.1 記事のトークンの長さの分布¶

6.2.2 タイトルのトークン数の分布¶

7. 前処理とNormalization¶

8. モデルの読み込み¶

9. 評価の指標¶

9.1 Bert Score¶

10. Training¶

10.1 トレーニング引数¶

10.2 Trainer¶

10.3 Model Training¶

11. モデルの評価¶

12. モデルのテスト¶

12.1. Generate Predictions (予測の生成)¶

12.2. Yahoo! Newsで試してみる¶

12.3. メールで試してみる (Just for fun)¶

13. モデルを改良する方法¶

13.1. データ¶

13.2. モデルとパラメータ¶

13.3. 学習戦略¶

13.4. カリキュラム学習¶

13.5. 後処理とルール¶

13.6. デコーディング戦略を試す：¶

14. まとめ¶