訓練資料不足不用愁！Google 開源先進自然語言處理預訓練技術BERT
訓練資料不足不用愁！Google開源先進自然語言處理預訓練技術BERT
News from: iThome & Google AI blog.
BERT在史丹佛問答資料集SQuAD v1.1獲得新高分，準確度達93.2%，超過之前的最高分91.6％和人類91.2％。
Google對外開源了用於自然語言預訓練的新技術BERT（Bidirectional Encoder Representations from Transformers），讓使用者可以在雲端TPU或是GPU上，快速地訓練問答系統以及其他類型的模型。這個BERT版本建構在TensorFlow之上，專案除了有原始碼之外，也包含了許多預先訓練的語言表示模型。
自然語言處理面臨的挑戰之一便是缺乏訓練資料，Google提到，由於自然語言訓練是一個綜合許多不同任務的多樣化領域，而大多數任務特定資料集就僅包含數千或是數十萬標籤訓練範例，但是基於現代深度學習技術的自然語言，資料集的規模需要達數百萬或是數十億的標籤訓練資料範例，才能獲得真正良好的結果。
但事實上，要取得這麼大量的訓練資料並不容易，因此才發展出了預訓練技術，來縮小需求與實際資料量的空缺差距。預訓練使用網路上大量未標記的文字來訓練通用語言表示模型，而預訓練的模型可以接著使用情感分析或是問答等小型特定資料集，進行預訓練模型微調，這樣的方法比一開始就使用這些特定領域的資料集進行訓練，可以明顯提升模型準確度。預訓練分為上下文（Contextual）或無上下文，其中上下文又可進一步分為單向或雙向的。像是word2vec或GloVe等無上下文模型，為每個字彙產生單詞嵌入表示，Google舉例，「銀行」這個詞對「帳戶」和「河岸」，具有相同的無上下文表示，而上下文模型所產生的單詞表示將會基於句子中的其他詞，以「我存取銀行帳戶」句子來說，單向的上下文模型，以基於「我存取過」而非「帳戶」來表示「銀行」。
BERT採用預訓練上下文表示方法，其中也包括使用了半監督式序列學習、生成式預訓練、ELMo和ULMFit技術，不像過去其他模型，BERT是第一個深度雙向、無監督的語言表示，僅使用純文字資料資料庫進行預訓練。以上面提到的例子來說，BERT雙向上下文模型，則會以句子中單詞前後的文字，包括「我存取過」和「帳戶」來表示「銀行」。Google從深度神經網路的最底層開始，使其具有雙向特性。
雙向上下文模型非常強大，但之前之所以沒有系統使用雙向模型，是因為其訓練並不簡單，單向模型只要預測單次在句子中的前一個單詞，就能進行有效的訓練，但光只是每個單詞的前一個或是下一個單詞這樣的簡易條件，是無法訓練出雙向模型的，因為這將讓該單詞在多層模型中，間接的把自己列入參考。Google採用直覺的方法，以遮罩蓋住一些輸入的詞，並對每個詞雙向地預測遮蓋的詞。雖然這個方法已經存在很久了，但是BERT是第一個成功將其實作於訓練深度神經網路的技術。
Google為了測試BERT的效能，與其他先進的自然語言處理系統相比。BERT幾乎不需要針對特定任務進行神經網路基礎架構調整，就能實現所有的結果。在史丹佛問答資料集SQuAD v1.1，BERT獲得新高分，精準度達93.2%，超過之前最高分的91.6％和人類分數91.2％。
而Google開源的BERT的預訓練新技術，使用者可以在雲端TPU以約30分鐘，或是使用單個GPU花費數小時，就能訓練出先進的問答系統。Google提到，由於雲端TPU的技術，讓他們可以更快速的進行實驗以及調整模型，而這對於超越現有的預訓練技術至關重要。
Google這次開源的專案，包含了建構在TensorFlow上的程式碼以及其他的語言表示模型，除了讓自然語言處理研究人員不需要重頭開始撰寫程式碼外，其中包含的模型，可以讓使用者以數小時或是更短的時間內，對各種自然語言任務進行微調。不過，目前發布的BERT模型僅限英語，在將來還會發布多種語言預訓練模型。
-----------------------------------------------------------------------------------
One of the biggest challenges in natural language processing (NLP) is the shortage of training data. Because NLP is a diversified field with many distinct tasks, most task-specific datasets contain only a few thousand or a few hundred thousand human-labeled training examples. However, modern deep learning-based NLP models see benefits from much larger amounts of data, improving when trained on millions, or billions, of annotated training examples. To help close this gap in data, researchers have developed a variety of techniques for training general purpose language representation models using the enormous amount of unannotated text on the web (known as pre-training). The pre-trained model can then be fine-tuned on small-data NLP tasks like question answering and sentiment analysis, resulting in substantial accuracy improvements compared to training on these datasets from scratch.
This week, we open sourced a new technique for NLP pre-training called Bidirectional Encoder Representations from Transformers, or BERT. With this release, anyone in the world can train their own state-of-the-art question answering system (or a variety of other models) in about 30 minutes on a single Cloud TPU, or in a few hours using a single GPU. The release includes source code built on top of TensorFlow and a number of pre-trained language representation models. In our associated paper, we demonstrate state-of-the-art results on 11 NLP tasks, including the very competitive Stanford Question Answering Dataset (SQuAD v1.1).
What Makes BERT Different?
BERT builds upon recent work in pre-training contextual representations — including Semi-supervised Sequence Learning, Generative Pre-Training, ELMo, and ULMFit. However, unlike these previous models, BERT is the first deeply bidirectional, unsupervised language representation, pre-trained using only a plain text corpus (in this case, Wikipedia).
Why does this matter? Pre-trained representations can either be context-free or contextual, and contextual representations can further be unidirectional or bidirectional. Context-free models such as word2vec or GloVe generate a single word embedding representation for each word in the vocabulary. For example, the word “bank” would have the same context-free representation in “bank account” and “bank of the river.” Contextual models instead generate a representation of each word that is based on the other words in the sentence. For example, in the sentence “I accessed the bank account,” a unidirectional contextual model would represent “bank” based on “I accessed the” but not “account.” However, BERT represents “bank” using both its previous and next context — “I accessed the ... account” — starting from the very bottom of a deep neural network, making it deeply bidirectional.
A visualization of BERT’s neural network architecture compared to previous state-of-the-art contextual pre-training methods is shown below. The arrows indicate the information flow from one layer to the next. The green boxes at the top indicate the final contextualized representation of each input word:
The Strength of Bidirectionality
If bidirectionality is so powerful, why hasn’t it been done before? To understand why, consider that unidirectional models are efficiently trained by predicting each word conditioned on the previous words in the sentence. However, it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that’s being predicted to indirectly “see itself” in a multi-layer model.
To solve this problem, we use the straightforward technique of masking out some of the words in the input and then condition each word bidirectionally to predict the masked words. For example:
While this idea has been around for a very long time, BERT is the first time it was successfully used to pre-train a deep neural network.
BERT also learns to model relationships between sentences by pre-training on a very simple task that can be generated from any text corpus: Given two sentences A and B, is B the actual next sentence that comes after A in the corpus, or just a random sentence? For example:
Training with Cloud TPUs
Everything that we’ve described so far might seem fairly straightforward, so what’s the missing piece that made it work so well? Cloud TPUs. Cloud TPUs gave us the freedom to quickly experiment, debug, and tweak our models, which was critical in allowing us to move beyond existing pre-training techniques. The Transformer model architecture, developed by researchers at Google in 2017, also gave us the foundation we needed to make BERT successful. The Transformer is implemented in our open source release, as well as the tensor2tensor library.
Results with BERT
To evaluate performance, we compared BERT to other state-of-the-art NLP systems. Importantly, BERT achieved all of its results with almost no task-specific changes to the neural network architecture. On SQuAD v1.1, BERT achieves 93.2% F1 score (a measure of accuracy), surpassing the previous state-of-the-art score of 91.6% and human-level score of 91.2%:
BERT also improves the state-of-the-art by 7.6% absolute on the very challenging GLUE benchmark, a set of 9 diverse Natural Language Understanding (NLU) tasks. The amount of human-labeled training data in these tasks ranges from 2,500 examples to 400,000 examples, and BERT substantially improves upon the state-of-the-art accuracy on all of them:
Making BERT Work for You
The models that we are releasing can be fine-tuned on a wide variety of NLP tasks in a few hours or less. The open source release also includes code to run pre-training, although we believe the majority of NLP researchers who use BERT will never need to pre-train their own models from scratch. The BERT models that we are releasing today are English-only, but we hope to release models which have been pre-trained on a variety of languages in the near future.
The open source TensorFlow implementation and pointers to pre-trained BERT models can be found at http://goo.gl/language/bert. Alternatively, you can get started using BERT through Colabwith the notebook “BERT FineTuning with Cloud TPUs.”
You can also read our paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" for more details.
但事實上，要取得這麼大量的訓練資料並不容易，因此才發展出了預訓練技術，來縮小需求與實際資料量的空缺差距。預訓練使用網路上大量未標記的文字來訓練通用語言表示模型，而預訓練的模型可以接著使用情感分析或是問答等小型特定資料集，進行預訓練模型微調，這樣的方法比一開始就使用這些特定領域的資料集進行訓練，可以明顯提升模型準確度。預訓練分為上下文（Contextual）或無上下文，其中上下文又可進一步分為單向或雙向的。像是word2vec或GloVe等無上下文模型，為每個字彙產生單詞嵌入表示，Google舉例，「銀行」這個詞對「帳戶」和「河岸」，具有相同的無上下文表示，而上下文模型所產生的單詞表示將會基於句子中的其他詞，以「我存取銀行帳戶」句子來說，單向的上下文模型，以基於「我存取過」而非「帳戶」來表示「銀行」。
BERT採用預訓練上下文表示方法，其中也包括使用了半監督式序列學習、生成式預訓練、ELMo和ULMFit技術，不像過去其他模型，BERT是第一個深度雙向、無監督的語言表示，僅使用純文字資料資料庫進行預訓練。以上面提到的例子來說，BERT雙向上下文模型，則會以句子中單詞前後的文字，包括「我存取過」和「帳戶」來表示「銀行」。Google從深度神經網路的最底層開始，使其具有雙向特性。
雙向上下文模型非常強大，但之前之所以沒有系統使用雙向模型，是因為其訓練並不簡單，單向模型只要預測單次在句子中的前一個單詞，就能進行有效的訓練，但光只是每個單詞的前一個或是下一個單詞這樣的簡易條件，是無法訓練出雙向模型的，因為這將讓該單詞在多層模型中，間接的把自己列入參考。Google採用直覺的方法，以遮罩蓋住一些輸入的詞，並對每個詞雙向地預測遮蓋的詞。雖然這個方法已經存在很久了，但是BERT是第一個成功將其實作於訓練深度神經網路的技術。
Google為了測試BERT的效能，與其他先進的自然語言處理系統相比。BERT幾乎不需要針對特定任務進行神經網路基礎架構調整，就能實現所有的結果。在史丹佛問答資料集SQuAD v1.1，BERT獲得新高分，精準度達93.2%，超過之前最高分的91.6％和人類分數91.2％。
而Google開源的BERT的預訓練新技術，使用者可以在雲端TPU以約30分鐘，或是使用單個GPU花費數小時，就能訓練出先進的問答系統。Google提到，由於雲端TPU的技術，讓他們可以更快速的進行實驗以及調整模型，而這對於超越現有的預訓練技術至關重要。
Google這次開源的專案，包含了建構在TensorFlow上的程式碼以及其他的語言表示模型，除了讓自然語言處理研究人員不需要重頭開始撰寫程式碼外，其中包含的模型，可以讓使用者以數小時或是更短的時間內，對各種自然語言任務進行微調。不過，目前發布的BERT模型僅限英語，在將來還會發布多種語言預訓練模型。
-----------------------------------------------------------------------------------
One of the biggest challenges in natural language processing (NLP) is the shortage of training data. Because NLP is a diversified field with many distinct tasks, most task-specific datasets contain only a few thousand or a few hundred thousand human-labeled training examples. However, modern deep learning-based NLP models see benefits from much larger amounts of data, improving when trained on millions, or billions, of annotated training examples. To help close this gap in data, researchers have developed a variety of techniques for training general purpose language representation models using the enormous amount of unannotated text on the web (known as pre-training). The pre-trained model can then be fine-tuned on small-data NLP tasks like question answering and sentiment analysis, resulting in substantial accuracy improvements compared to training on these datasets from scratch.
This week, we open sourced a new technique for NLP pre-training called Bidirectional Encoder Representations from Transformers, or BERT. With this release, anyone in the world can train their own state-of-the-art question answering system (or a variety of other models) in about 30 minutes on a single Cloud TPU, or in a few hours using a single GPU. The release includes source code built on top of TensorFlow and a number of pre-trained language representation models. In our associated paper, we demonstrate state-of-the-art results on 11 NLP tasks, including the very competitive Stanford Question Answering Dataset (SQuAD v1.1).
What Makes BERT Different?
BERT builds upon recent work in pre-training contextual representations — including Semi-supervised Sequence Learning, Generative Pre-Training, ELMo, and ULMFit. However, unlike these previous models, BERT is the first deeply bidirectional, unsupervised language representation, pre-trained using only a plain text corpus (in this case, Wikipedia).
Why does this matter? Pre-trained representations can either be context-free or contextual, and contextual representations can further be unidirectional or bidirectional. Context-free models such as word2vec or GloVe generate a single word embedding representation for each word in the vocabulary. For example, the word “bank” would have the same context-free representation in “bank account” and “bank of the river.” Contextual models instead generate a representation of each word that is based on the other words in the sentence. For example, in the sentence “I accessed the bank account,” a unidirectional contextual model would represent “bank” based on “I accessed the” but not “account.” However, BERT represents “bank” using both its previous and next context — “I accessed the ... account” — starting from the very bottom of a deep neural network, making it deeply bidirectional.
A visualization of BERT’s neural network architecture compared to previous state-of-the-art contextual pre-training methods is shown below. The arrows indicate the information flow from one layer to the next. The green boxes at the top indicate the final contextualized representation of each input word:
The Strength of Bidirectionality
If bidirectionality is so powerful, why hasn’t it been done before? To understand why, consider that unidirectional models are efficiently trained by predicting each word conditioned on the previous words in the sentence. However, it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that’s being predicted to indirectly “see itself” in a multi-layer model.
To solve this problem, we use the straightforward technique of masking out some of the words in the input and then condition each word bidirectionally to predict the masked words. For example:
While this idea has been around for a very long time, BERT is the first time it was successfully used to pre-train a deep neural network.
BERT also learns to model relationships between sentences by pre-training on a very simple task that can be generated from any text corpus: Given two sentences A and B, is B the actual next sentence that comes after A in the corpus, or just a random sentence? For example:
Training with Cloud TPUs
Everything that we’ve described so far might seem fairly straightforward, so what’s the missing piece that made it work so well? Cloud TPUs. Cloud TPUs gave us the freedom to quickly experiment, debug, and tweak our models, which was critical in allowing us to move beyond existing pre-training techniques. The Transformer model architecture, developed by researchers at Google in 2017, also gave us the foundation we needed to make BERT successful. The Transformer is implemented in our open source release, as well as the tensor2tensor library.
Results with BERT
To evaluate performance, we compared BERT to other state-of-the-art NLP systems. Importantly, BERT achieved all of its results with almost no task-specific changes to the neural network architecture. On SQuAD v1.1, BERT achieves 93.2% F1 score (a measure of accuracy), surpassing the previous state-of-the-art score of 91.6% and human-level score of 91.2%:
BERT also improves the state-of-the-art by 7.6% absolute on the very challenging GLUE benchmark, a set of 9 diverse Natural Language Understanding (NLU) tasks. The amount of human-labeled training data in these tasks ranges from 2,500 examples to 400,000 examples, and BERT substantially improves upon the state-of-the-art accuracy on all of them:
Making BERT Work for You
The models that we are releasing can be fine-tuned on a wide variety of NLP tasks in a few hours or less. The open source release also includes code to run pre-training, although we believe the majority of NLP researchers who use BERT will never need to pre-train their own models from scratch. The BERT models that we are releasing today are English-only, but we hope to release models which have been pre-trained on a variety of languages in the near future.
The open source TensorFlow implementation and pointers to pre-trained BERT models can be found at http://goo.gl/language/bert. Alternatively, you can get started using BERT through Colabwith the notebook “BERT FineTuning with Cloud TPUs.”
You can also read our paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" for more details.
留言
張貼留言