如何将HuggingFace的transfomers库与fastai合并的示例
 

在本教程中,我们将看到如何使用fastai库从模型中微调预训练的变压器模型。 变压器库 通过HuggingFace。我们将使用中级API来收集数据。即使本教程自成一体,它也可能有助于检查 imagenette教程 对计算机视觉中的中级API(使用高级API进行了简要介绍)进行了重新研究。

导入变压器预训练模型

首先,我们需要安装转换器库。如果尚未完成,请安装该库:

!pip install -Uq transformers

Then let's import what will need: we will fine-tune the GPT2 pretrained model and fine-tune on wikitext-2 here. For this, we need the GPT2LMHeadModel (since we want a language model) and the GPT2Tokenizer 至 prepare the data.

from transformers import GPT2LMHeadModel, GPT2TokenizerFast

我们可以使用此GPT2模型的多个版本, 变压器文档 for more details. Here we will use the basic version (that already takes a lot of space in memory!) You can change the model used by changing the content of pretrained_weights (if it's not a GPT2 model, you'll need 至 change the classes used for the model and the 至 kenizer of course).

pretrained_weights = 'gpt2'
至 kenizer = GPT2TokenizerFast.from_pretrained(pretrained_weights)
model = GPT2LMHeadModel.from_pretrained(pretrained_weights)

Before we move on 至 the fine-tuning part, let's have a look at this 至 kenizer and this model. The 至 kenizers in HuggingFace usually do the 至 kenization and the numericalization in one step (we ignore the padding warning for now):

ids = 至 kenizer.encode('This is an example of 文本, and')
ids
[1212, 318, 281, 1672, 286, 2420, 11, 290]

像法泰 Transforms, the 至 kenizer has a decode method 至 give you back a 文本 from ids:

至 kenizer.decode(ids)
'This is an example of 文本, and'

The model can be used 至 generate predictions (it is pretrained). It has a generate method that expects a batch of prompt, so we feed it our ids and add one batch dimension (there is a padding warning we can ignore as well):

import 至 rch
t = 至 rch.LongTensor(ids)[None]
preds = model.generate(t)
Setting `pad_token_id` 至  `eos_token_id`:50256 for open-end generation.

默认情况下,预测的长度为20:

preds.shape,preds[0]
(torch.Size([1, 20]),
 tensor([1212,  318,  281, 1672,  286, 2420,   11,  290,  340,  338,  407,  257,
          922,  530,   13,  198,  198,  464,  717, 1517]))

我们可以使用解码方法(相对于张量,首选numpy数组):

至 kenizer.decode(preds[0].numpy())
"This is an example of 文本, and it's not a good one.\n\nThe first thing"

与Fastai弥合鸿沟

现在,让我们看看如何使用fastai通过所有训练工具(学习率查找器,1cycle策略等)在wikitext-2上微调此模型。首先,我们导入所有文本实用程序:

from 法泰.text.all import *

准备数据

然后我们下载数据集(如果不存在),它是两个csv文件:

path = untar_data(URLs.WIKITEXT_TINY)
path.ls()
(#2) [Path('/home/jhoward/.fastai/data/wikitext-2/test.csv'),Path('/home/jhoward/.fastai/data/wikitext-2/train.csv')]

让我们看一下这些csv文件的外观:

df_train = pd.read_csv(path/'train.csv', header=None)
df_valid = pd.read_csv(path/'test.csv', header=None)
df_train.head()
0
0 \ n = 2013 –约克市14季= \ n \ n 2013 – 14季是<unk>竞争协会足球季,约克城市足球俱乐部参加的足球联赛第77季。他们的17岁@[电子邮件 protected] 2012年排在第13位,这意味着这是他们连续第二个联赛第二个赛季。季节从2013年7月1日至2014年6月30日。 \ n奈杰尔•沃辛顿(Nigel Worthington)在约克担任经理后的第一个完整赛季开始,连续八次夏季签约。到了年底,约克只不过是保级级别的...
1 \ n =大男孩(歌曲)= \ n \ n“大男孩”<unk>“我现在是个大男孩”是杰克逊5唱片公司录制的第一张单曲,该唱片在1968年1月由Steeltown Records发行。乐队在许多Steeltown的乐曲中演奏乐器,包括“ Big Boy”。这首歌既没有取得成功也没有取得成功,但杰克逊一家仍然对结果感到满意。 \ n在移居Motown Records之前,Jackson 5将与Steeltown Records发行第二张单曲。人们认为该乐队在Steeltown Records的唱片丢失了,但它们却重新...
2 \ n = The Remix(Lady Gaga专辑)= \ n \ n The Remix是美国唱片艺术家Lady Gaga创作的混音专辑。 2010年3月3日在日本发行,收录了她的第一张工作室专辑The Fame(2008)和她的第三张扩展专辑The Fame Monster(2009)的混音。修订后的曲目清单已准备好在2010年5月3日从墨西哥开始的其他市场中发布。许多唱片艺术家制作了这些歌曲,包括“宠物小子”,“激情坑”和“箭的声音”。重新混音的版本同时具有快节奏和<unk> composit...
3 \ n =除夕(Up All Night)= \ n \ n“除夕”是美国喜剧电视剧“ Up All Night”第一个季的第十二集。此集最初于2012年1月12日在美国NBC播出。这是埃里卡(Erica)写的<unk>由贝丝·麦卡锡(Beth McCarthy)执导@ [电子邮件 protected] 米勒。情节还以贾森李(Jason Lee)作为克里斯和里根的邻居以及艾娃(Ava)的男友凯文(Kevin)的客座演出为特色。 \ n在里根(克里斯蒂娜·阿普尔盖特)和克里斯的(威尔<unk>)在第一个除夕游戏之夜,里根的竞争力突显出来,导致了里根...
4 \ n = Geopyxis carbonaria = \ n \ n Geopyxis carbonaria是Geopyxis属的一种真菌<unk>。该物种于1805年首次被科学所描述,并在1889年改名,俗称木炭爱的精灵@[电子邮件 protected] 矮人杯<unk> cup , <unk> <unk>杯子或精灵杯。小,<unk> @[电子邮件 protected] 真菌的形果体微红@[电子邮件 protected] 棕色,带有白色条纹,长达2厘米(0 @ [电子邮件 protected] 跨8英寸)。它们的茎短而细。通常在最近烧过刷子的土壤中发现果体,有时数量很大。

我们将所有文本收集在一个numpy数组中(因为在fa​​stai中使用这种方式会更容易):

all_texts = np.concatenate([df_train[0].values, df_valid[0].values])

要处理这些数据以训练模型,我们需要构建一个 Transform 将会被懒惰地应用。在这种情况下,我们可以一劳永逸地进行预处理,并且仅使用变换进行解码(我们将看到解码后的样子),但是HuggingFace的快速分词器如其名称所示是快速的,因此它不是这样做确实会影响性能。

在快餐店 Transform 您可以定义:

  • an encodes method that is applied when you call the transform (a bit like the forward method in a nn.Module)
  • a decodes method that is applied when you call the decode method of the transform, if you need 至 decode anything for showing purposes (like converting ids 至 a 文本 here)
  • a setups method that sets some inner state of the Transform (这里不需要,所以我们跳过它)
class TransformersTokenizer(Transform):
    def __init__(self, 至 kenizer): self.至 kenizer = 至 kenizer
    def encodes(self, x): 
        至 ks = self.至 kenizer.至 kenize(x)
        return tensor(self.至 kenizer.convert_tokens_to_ids(至 ks))
    def decodes(self, x): return TitledStr(self.至 kenizer.decode(x.cpu().numpy()))

上面的代码有两个注释:

  • in encodes we don't use the 至 kenizer.encode method since it does some additional preprocessing for the model after 至 kenizing and numericalizing (the part throwing a warning before). Here we don't need any post-processing so it's fine 至 skip it.
  • in decodes we return a TitledStr object and not just a plain string. That's a 法泰 class that adds a show method 至 the string, which will allow us 至 use all the 法泰 show methods.

然后,您可以使用此分组数据 Transform 用一个 TfmdLists. It has an s in its name because it contains the training and validation set. We indicate the indices of the training set and the validation set with splits (here all the first indices until len(df_train) and then all the remaining indices):

splits = [range_of(df_train), list(range(len(df_train), len(all_texts)))]
tls = TfmdLists(all_texts, TransformersTokenizer(至 kenizer), splits=splits, dl_type=LMDataLoader)
Token indices sequence length is longer than the specified maximum sequence length for this model (4576 > 1024). Running this sequence through the model will result in indexing errors

We specify dl_type=LMDataLoader for when we will convert this TfmdLists数据加载器s:我们将使用 LMDataLoader 因为我们有语言建模问题,而不是通常的fastai TfmdDL.

在一个 TfmdLists 您可以非常轻松地访问培训或验证集的元素:

tls.train[0],tls.valid[0]
(tensor([220, 198, 796,  ..., 198, 220, 198]),
 tensor([220, 198, 796,  ..., 198, 220, 198]))

它们看起来相同,但仅是因为它们以相同的方式开始和结束。我们可以看到形状不同:

tls.tfms(tls.train.items[0]).shape, tls.tfms(tls.valid.items[0]).shape
(torch.Size([4576]), 至 rch.Size([1485]))

我们可以看看两种解码使用 show_at:

show_at(tls.train, 0)
 
 = 2013 – 14 York City F.C. season = 
 
 The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club, a professional football club based in York, North Yorkshire, England. Their 17th @[电子邮件 protected] place finish in 2012 – 13 meant it was their second consecutive season in League Two. The season ran from 1 July 2013 至  30 June 2014. 
 Nigel Worthington, starting his first full season as York manager, made eight permanent summer signings. By the turn of the year York were only above the relegation zone on goal difference, before a 17 @[电子邮件 protected] match unbeaten run saw the team finish in seventh @[电子邮件 protected] place in the 24 @[电子邮件 protected] team 2013 – 14 Football League Two. This meant York qualified for the play @[电子邮件 protected] offs, and they were eliminated in the semi @[电子邮件 protected] final by Fleetwood Town. York were knocked out of the 2013 – 14 FA Cup, Football League Cup and Football League Trophy in their opening round matches. 
 35 players made at least one appearance in nationally organised first @[电子邮件 protected] team competition, and there were 12 different <unk>. Defender Ben Davies missed only five of the fifty @[电子邮件 protected] two competitive matches played over the season. Wes Fletcher finished as leading scorer with 13 goals, of which 10 came in league competition and three came in the FA Cup. The winner of the <unk> of the Year award, voted for by the club's supporters, was <unk> Oyebanjo. 
 
 = = Background and pre @[电子邮件 protected] season = = 
 
 The 2012 – 13 season was York City's first season back in the Football League, having won the Conference Premier play @[电子邮件 protected] offs in 2011 – 12 after <unk> years in the Football Conference. Manager Gary Mills was sacked in March 2013 following an 11 @[电子邮件 protected] match run without a victory, and was replaced by former Northern Ireland manager Nigel Worthington. Despite being in the relegation zone with three matches remaining, Worthington led the team 至  safety from relegation after a 1 – 0 win away 至  Dagenham & Redbridge on the final day of the season. York finished the season in 17th @[电子邮件 protected] place in the 2012 – 13 League Two table. 
 Following the previous season's conclusion Lee <unk>, Jon <unk>, Chris <unk>, Ben Everson, Scott Kerr, David <unk>, Patrick <unk>, Michael Potts, Jamie Reed and Jason Walker were released by York, while <unk> Blair departed for Fleetwood Town. David McGurk, <unk> Oyebanjo, Danny Parslow, Tom Platt and Chris Smith signed new contracts with the club. New players signed ahead of the start of the season were goalkeeper Chris <unk> on a season @[电子邮件 protected] long loan from Blackpool, defender Ben Davies on loan from Preston North End, midfielders Craig Clay from Chesterfield and Lewis Montrose from Gillingham, winger <unk> Puri from St <unk> and strikers Ryan Bowman from Hereford United, Richard Cresswell from Sheffield United, Wes Fletcher from Burnley and Ryan Jarvis from Torquay United. Defender Mike Atkinson and striker Chris Dickinson entered the first @[电子邮件 protected] team squad from the youth team after agreeing professional contracts. 
 York retained the previous season's home and away kits. The home kit comprised red shirts with white sleeves, light blue shorts and white socks. The away kit included light blue shirts with white sleeves, white shorts and light blue socks. <unk> Health continued as shirt sponsors for the second successive season. 
 
 = = Review = = 
 
 
 = = = August = = = 
 
 York began the season with a 1 – 0 home win over the previous season's play @[电子邮件 protected] off finalists, Northampton Town, with <unk> Jarvis scoring the winning goal in the 90th @[电子邮件 protected] minute. However, defeat came in York's match against Championship side Burnley in the first round of the League Cup, going down 4 – 0 at home. The team endured their first league defeat of the season in the following game after being beaten 2 – 0 away by Dagenham & Redbridge, the home team scoring in each half. York then held Hartlepool United 至  a 0 – 0 home draw, before being beaten 3 – 2 away by Bristol Rovers, in which Jarvis scored twice before John @[电子邮件 protected] Joe O 'Toole scored the winning goal for the home team in the 67th @[电子邮件 protected] minute. Two signings were made shortly before the transfer deadline ; defender George Taft was signed on a one @[电子邮件 protected] month loan from Leicester City, while Middlesbrough midfielder Ryan Brobbel joined on a one @[电子邮件 protected] month loan. <unk> John <unk>, who had been 至 ld he had no future with the club, departed after signing for FC Halifax Town. Jarvis gave York the lead away at Exeter City before Alan <unk> scored in each half 至  see the home team win 2 – 1. 
 
 = = = September = = = 
 
 York suffered their first home league defeat of the season after AFC Wimbledon won 2 – 0, with Michael Smith scoring in each half. Former Ipswich Town midfielder Josh Carson, who had a spell on loan with York the previous season, signed a contract until the end of 2013 – 14 and Sheffield United midfielder Elliott <unk> signed on a one @[电子邮件 protected] month loan. Brobbel opened the scoring in the second minute of his home debut against Mansfield Town, although the away team went on 至  score twice 至  win 2 – 1. York's run of four defeats ended following a 1 – 1 draw away 至  Wycombe Wanderers, in which McGurk gave York the lead before the home team levelled through Dean Morgan. Taft was sent back 至  Leicester after he fell behind McGurk, Parslow and Smith in the pecking order for a central defensive berth. York achieved their first win since the opening day of the season after beating Portsmouth 4 – 2 at home, with Fletcher ( 2 ), Montrose and Jarvis scoring. 
 
 = = = October = = = 
 
 Defender Luke O 'Neill was signed from Burnley on a 28 @[电子邮件 protected] day emergency loan. He made his debut in York's 3 – 0 win away at Torquay, which was the team's first successive win of the season. York were knocked out of the Football League Trophy in the second round after being beaten 3 – 0 at home by League One team Rotherham United, before their winning streak in the league was ended with a 3 – 0 defeat away 至  Newport County. York drew 2 – 2 away 至  Chesterfield, having taken a two @[电子邮件 protected] goal lead through O 'Neill and Jarvis, before the home team fought back through Armand <unk> and Jay O <unk>. The team then hosted Fleetwood Town, and the visitors won 2 – 0 with goals scored in each half by Gareth Evans and <unk> Matt. Scunthorpe United were beaten 4 – 1 at home 至  end York's three @[电子邮件 protected] match run without a win, with all the team's goals coming in the first half from Carson, Fletcher and Brobbel ( 2 )。 
 
 = = = November = = = 
 
 Bowman scored his first goals for York away 至  Cheltenham Town, as York twice fought back from behind 至  draw 2 – 2. York drew 3 – 3 away 至  Bristol Rovers 至  earn a first round replay in the FA Cup, taking the lead through Jarvis before Eliot Richards equalised for the home team. Carson scored a 30 yard volley 至  put York back in the lead, and after Bristol Rovers goals from Matt <unk> and Chris <unk>, Fletcher scored an 86th @[电子邮件 protected] minute equaliser for York. Bowman scored with a header from an O 'Neill cross 至  open the scoring at home 至  Plymouth Argyle, which was the first goal the visitors had conceded in 500 minutes of action. However, Plymouth equalised 11 minutes later through <unk> <unk> and the match finished a 1 – 1 draw. York were knocked out of the FA Cup after losing 3 – 2 at home 至  Bristol Rovers in a first round replay ; the visitors were 3 – 0 up by 50 @[电子邮件 protected] minutes before Fletcher pulled two back for York with a penalty and a long @[电子邮件 protected] range strike. 
 Defender Keith Lowe, of Cheltenham, and goalkeeper Nick Pope, of Charlton Athletic, were signed on loan until January 2014. They both played in York's first league defeat in four weeks, 2 – 1 away, 至  Southend United. <unk> <unk> gave Southend the lead early into the match and Bowman equalised for York with a low strike during the second half, before Luke Prosser scored the winning goal for the home side in stoppage 时间. With Pope preferred in goal, <unk> returned 至  Blackpool on his own accord, although his loan agreement would stay in place until January 2014. York then drew 0 – 0 away 至  Morecambe. After Pope was recalled from his loan by Charlton, York signed Wolverhampton Wanderers goalkeeper Aaron McCarey on loan until January 2014. McCarey kept a clean sheet in York's 0 – 0 home draw with Rochdale. 
 
 = = = December = = = 
 
 Cresswell retired from playing as a result of an eye complaint and a knee injury. York drew 1 – 1 away 至  Burton Albion, with an own goal scored by Shane <unk> @[电子邮件 protected] <unk> giving York the lead in the 64th @[电子邮件 protected] minute before the home team equalised eight minutes later through Billy <unk>. Atkinson was released after failing 至  force himself into the first team and signed for Scarborough Athletic, with whom he had been on loan. York drew 0 – 0 at home with second @[电子邮件 protected] placed Oxford United, in which Carson came closest 至  scoring with a volley that <unk> across the face of the goal. This was followed by another draw after the match away 至  Accrington Stanley finished 1 – 1, with the home team <unk> 10 minutes after a Fletcher penalty had given York the lead in the 35th @[电子邮件 protected] minute. Striker <unk> McDonald, who had been released by Peterborough United, was signed on a contract until the end of the season. York's last match of 2013 was a 2 – 1 defeat away at Bury, a result that ended York's run of consecutive draws at five. The home team were 2 – 0 up by the 19th @[电子邮件 protected] minute, before Michael Coulson scored York's goal in the 73rd @[电子邮件 protected] minute. This result meant York would begin 2014 in 22nd @[电子邮件 protected] position in the table, only out of the relegation zone on goal difference. 
 
 = = = January = = = 
 
 Jarvis scored the only goal in York's first win since October 2013, a 1 – 0 home victory over Morecambe on New Year's Day. McCarey was recalled by Wolverhampton Wanderers due 至  an injury 至  one of their <unk>, while O 'Neill was recalled by Burnley 至  take part in their FA Cup match. York achieved back @[电子邮件 protected] 至  @[电子邮件 protected] back wins for the first 时间 since October 2013 after Dagenham & Redbridge were beaten 3 – 1 at home, with Bowman opening the scoring in the second half before Fletcher scored twice. Adam Reed, who had a spell on loan with York in the previous season, was signed on a contract until the end of the season after parting company with Burton. Davies'loan was extended, while Brobbel and <unk> returned 至  their parent clubs. Cheltenham club captain Russell Penn, a midfielder, was signed on a two @[电子邮件 protected] and @[电子邮件 protected] a @[电子邮件 protected] half @[电子邮件 protected] year contract for an undisclosed fee. Lowe was subsequently signed permanently from Cheltenham on a two @[电子邮件 protected] and @[电子邮件 protected] a @[电子邮件 protected] half @[电子邮件 protected] year contract for an undisclosed fee. Having been allowed 至  leave the club on a free transfer, Ashley Chambers signed for Conference Premier club Cambridge United. 
 York achieved three successive wins for the first 时间 in 2013 – 14 after beating Northampton 2 – 0 away, with Bowman and Fletcher scoring in three @[电子邮件 protected] second half minutes. Defender John McCombe was signed on a two @[电子邮件 protected] and @[电子邮件 protected] a @[电子邮件 protected] half @[电子邮件 protected] year contract following his release from Mansfield, before Clay and Jamal <unk> left York by mutual consent. Pope returned 至  York on loan from Charlton for the remainder of the season. York's run of wins ended with a 0 – 0 draw at home 至  Bristol Rovers, before their first defeat of the year came after losing 2 – 0 away 至  Hartlepool. Preston winger Will Hayhurst, a Republic of Ireland under @[电子邮件 protected] 21 international, was signed on a one @[电子邮件 protected] month loan. York fell 至  a successive defeat for the first 时间 since September 2013 after being beaten 2 – 0 at home by Chesterfield. Shortly after the match, Smith left the club by mutual consent 至  pursue first @[电子邮件 protected] team football. 
 
 = = = February = = = 
 
 Fletcher scored a 90th @[电子邮件 protected] minute winner for York away 至  Fleetwood in a 2 – 1 win, a result that ended Fleetwood's five @[电子邮件 protected] match unbeaten run. York then drew 0 – 0 at home 至  fellow mid @[电子邮件 protected] table team Cheltenham, before beating Plymouth 4 – 0 away with goals from Fletcher, McCombe ( 2 ) and Carson as the team achieved successive away wins for the first 时间 in 2013 – 14. York went without scoring for a fourth consecutive home match after drawing 0 – 0 with Southend. Having worn the <unk> since an injury 至  McGurk, Penn was appointed captain for the rest of the season, a position that had earlier been held by Smith and Parslow. 
 
 = = = March = = = 
 
 York achieved their first home win in five matches after beating Exeter 2 – 1, with first half goals scored by McCombe and Coulson. Hayhurst's loan was extended 至  the end of the season, having impressed in his six appearances for the club. Coulson scored again with the only goal, a 41st @[电子邮件 protected] minute header, in York's 1 – 0 away win over AFC Wimbledon. Bowman scored the only goal with a 32nd @[电子邮件 protected] minute penalty as York won 1 – 0 away against Mansfield, in which Fletcher missed the opportunity 至  extend the lead when his stoppage 时间 penalty was saved by Alan Marriott. York moved one place outside the play @[电子邮件 protected] offs with a 2 – 0 home win over Wycombe, courtesy of a second Bowman penalty in as many matches and a Carson goal from the edge of the penalty area. Coulson scored York's only goal in a 1 – 0 away win over struggling Portsmouth with a low volley in the fifth @[电子邮件 protected] minute ; this result meant York moved into the play @[电子邮件 protected] offs in seventh @[电子邮件 protected] place with eight fixtures remaining. 
 Striker Calvin Andrew, who had been released by Mansfield in January 2014, was signed on a contract for the remainder of the season. He made his debut as a substitute in York's 1 – 0 home win over bottom of the table Torquay, in which Hayhurst scored the only goal in the 11th @[电子邮件 protected] minute with an 18 yard shot that <unk> off Aaron <unk>. Middlesbrough winger Brobbel rejoined on loan until the end of the season, following an injury 至  Carson. York's run of successive wins ended on six matches after a 0 – 0 home draw with Burton, and this result saw York drop out of the play @[电子邮件 protected] offs in eighth @[电子邮件 protected] place. With the team recording six wins and one draw in March 2014, including six clean sheets, Worthington was named League Two Manager of the Month. 
 
 = = = April = = = 
 
 Pope made a number of saves as York held league leaders Rochdale 至  a 0 – 0 away draw, with a point being enough 至  lift the team back into seventh @[电子邮件 protected] place. York were prevented from equalling a club record of eight consecutive clean sheets when Accrington scored a stoppage 时间 equaliser in a 1 – 1 home draw, in which York had taken earlier taken the lead with a Coulson penalty. A 1 – 0 win away win over Oxford, which was decided by a second half Coulson penalty, resulted in York moving one place above their opponents and back into seventh @[电子邮件 protected] place. York consolidated their place in a play @[电子邮件 protected] off position after beating Bury 1 – 0 at home with a fifth @[电子邮件 protected] minute goal scored by Lowe from a Hayhurst corner. The result meant York opened up a five @[电子邮件 protected] point lead over eighth @[电子邮件 protected] placed Oxford with two fixtures remaining. A place in the League Two play @[电子邮件 protected] offs was secured following a 1 – 0 win over Newport at home, in which Coulson scored the only goal in the 77th @[电子邮件 protected] minute with a 25 yard free kick. Pope earned a nomination for League Two Player of the Month for April 2014, having conceded only one goal in five matches in that period. 
 
 = = = May = = = 
 
 The league season concluded with an away match against divisional runners @[电子邮件 protected] up Scunthorpe ; having gone two goals down York fought back 至  draw 2 – 2 with goals scored by Brobbel and Andrew. This result meant York finished the season in seventh @[电子邮件 protected] place in League Two, and would thus play fourth @[电子邮件 protected] placed Fleetwood in the play @[电子邮件 protected] off semi @[电子邮件 protected] final on the back of a 17 @[电子邮件 protected] match unbeaten run. York lost 1 – 0 至  Fleetwood in the first leg at <unk> Crescent ; the goal came from former York player <unk> Blair in the 50th @[电子邮件 protected] minute, who scored from close range after Antoni <unk>'s shot was blocked on the line. A 0 – 0 draw away 至  Fleetwood in the second leg meant York were eliminated 1 – 0 on aggregate, ending the prospect of a second promotion in three seasons. At an awards night held at York Racecourse, Oyebanjo was voted <unk> of the Year for 2013 – 14. 
 
 = = Summary and aftermath = = 
 
 York mostly occupied the bottom half of the table before the turn of the year, and dropped as low as 23rd in September 2013. During February 2014 the team broke into the 至 p half of the table and with one match left were in sixth @[电子邮件 protected] place. York's defensive record was the third best in League Two with 41 goals conceded, bettered only by Southend ( 39 ) and Chesterfield ( 40 )。 Davies made the highest number of appearances over the season, appearing in 47 of York's 52 matches. Fletcher was York's 至 p scorer in the league and in all competitions, with 10 league goals and 13 in 至 tal. He was the only player 至  reach double figures, and was followed by Jarvis with nine goals. 
 After the season ended York released Tom Allan, Andrew, Dickinson, McDonald, Puri and Reed, while McGurk retired from professional football. Bowman and Oyebanjo left 至  sign for Torquay and Crawley Town respectively while Coulson signed a new contract with the club. York's summer signings included goalkeeper Jason <unk> from Tranmere Rovers, defenders <unk> <unk> from Dagenham, Marvin McCoy from Wycombe and Dave Winfield from Shrewsbury Town, midfielders <unk> <unk> from Mansfield, Anthony <unk> from Southend and Luke <unk> from Shrewsbury and striker Jake Hyde from <unk>. 
 
 = = Match details = = 
 
 League positions are sourced by <unk>, while the remaining information is referenced individually. 
 
 = = = Football League Two = = = 
 
 
 = = = League table ( part ) = = = 
 
 
 = = = FA Cup = = = 
 
 
 = = = League Cup = = = 
 
 
 = = = Football League Trophy = = = 
 
 
 = = = Football League Two play @[电子邮件 protected] offs = = = 
 
 
 = = <unk> = = 
 
 
 = = = In = = = 
 
 <unk> around club names denote the player's contract with that club had expired before he joined York. 
 
 = = = Out = = = 
 
 <unk> around club names denote the player joined that club after his York contract expired. 
 
 = = = Loan in = = = 
 
 
 = = = Loan out = = = 
 
 
 = = Appearances and goals = = 
 
 Source : 
 Numbers in parentheses denote appearances as substitute. 
 Players with names struck through and marked left the club during the playing season. 
 Players with names in italics and marked * were on loan from another club for the whole of their season with York. 
 Players listed with no appearances have been in the <unk> squad but only as unused <unk>. 
 Key 至  positions : <unk> – <unk> ; <unk> – Defender ; <unk> – <unk> ; <unk> – Forward 
 

show_at(tls.valid, 0)
 
 = Tropical Storm <unk> ( 2008 ) = 
 
 Tropical Storm <unk> was the tenth tropical storm of the 2008 Atlantic hurricane season. <unk> developed out of a strong tropical wave which moved off the African coast on August 31. The wave quickly became organized and was declared Tropical Depression Ten while located 170 mi ( 270 km ) 至  the south @[电子邮件 protected] southeast of the Cape Verde Islands on September 2. The depression was quickly upgraded 至  Tropical Storm <unk> around noon the same day. Over the next several days, <unk>搬到了西部将军@[电子邮件 protected] northwest direction and reached its peak intensity early on September 3. Strong wind shear, some due 至  the outflow of Hurricane Ike, and dry air caused the storm 至  weaken. On September 6, the combination of wind shear, dry air, and cooling waters caused <unk> 至  weaken into a tropical depression. <unk> deteriorated into a remnant low shortly after as convection continued 至  dissipate around the storm. The low ultimately dissipated while located 520 mi ( 835 km ) east of <unk> on September 10. However, the remnant moisture led 至  minor flooding on the island of St. Croix. 
 
 = = Meteorological history = = 
 
 Tropical Storm <unk> formed as a tropical wave that emerged off the west coast of Africa near the end of August 2008. It tracked south of Cape Verde and slowly developed, and on September 2 the disturbance became Tropical Depression Ten while located south @[电子邮件 protected] southeast of the Cape Verde islands. As the depression became more organized, an eye @[电子邮件 protected] like feature developed in the upper levels of the system. The depression was upgraded 至  Tropical Storm <unk> six hours after forming. <unk> was located in an area which was supportive for rapid intensification but was not forecast 至  intensify quickly. 
 <unk> continued 至  intensify throughout the afternoon as the storm became more symmetrical. However, due 至  the location of the storm, there was a lack of accurate wind speed readings, and the National Hurricane Center was uncertain of its actual intensity. Despite the lack of wind shear around the storm, the center became slightly exposed and ceased further intensification. The storm was also heading into an area where shear was <unk> 至  significantly increase due 至  an upper @[电子邮件 protected] level trough diving southward. Despite convection being partially removed from the center of <unk>, the storm intensified slightly in the early morning hours on September 3 as thunderstorm activity 至  the south of the center became more organized. The intensification was forecast 至  be short in duration as the trough 至  the north was deepening, causing the wind shear 至  the west 至  become stronger. 
 <unk> reached its peak intensity of 65 mph ( 100 km / h ) around 8 a.m. ( <unk> ) as it continued 至  become more organized. However, there were indications that it had already begun 至  weaken. <unk> 至 wards the north was becoming restricted and arc clouds began emanating from the storm, a sign that dry air was entering the system. During the afternoon hours, the structure of <unk> began 至  rapidly deteriorate as strong wind shear and dry air 至 ok their 至 ll. By the late night, the center was almost completely exposed and only a band of convection persisted near the center. 
 Despite continuing effects from the strong wind shear, a large, deep burst of convection formed in the northern <unk> of <unk>. The center was found 至  have shifted 至 wards the new convection leading 至  an increase in intensity. The forecast showed a slight decrease in wind shear as <unk> continued westward and no change in intensity over the 5 @[电子邮件 protected] day forecast was predicted. However, the convection decreased once more and the low became completely exposed by the late morning hours and <unk> weakened again. By the afternoon, the center of <unk> was only a <unk> of clouds, devoid of convection. During the overnight hours on September 4 into the morning of September 5, convection associated with <unk> began 至  <unk> somewhat, mostly 至  the north of the circulation, due 至  the strong <unk> wind shear. By mid @[电子邮件 protected] morning, <unk> re @[电子邮件 protected] intensified slightly due 至  the redevelopment of some convection. However, the redevelopment was short lived and wind shear again 至 ok its 至 ll on <unk> by late morning. The convection around the system became <unk> from the center and <unk> weakened slightly. 
 The weakening trend continued through the afternoon as the storm was being affected by strong <unk> shear. <unk> became almost fully devoid of any convection by mid @[电子邮件 protected] afternoon and the storm weakened 至  40 mph ( 65 km / h ), barely holding on 至  tropical storm status. <unk> regained a small amount of convection in the late night hours, but not enough 至  still be classified a tropical storm. Due 至  the lack of convection, <unk> was downgraded 至  a Tropical Depression at <unk> ( <unk> ) with winds of 35 mph ( 55 km / h )。 Since there was no convection around the system, it would have normally been classified a remnant low but, due 至  the possibility of the storm <unk> over the next several days, it was considered a tropical depression. The next morning, <unk> was downgraded 至  a remnant low as strong wind shear and dry air caused the demise of the storm. No redevelopment was expected with <unk> as it began 至  move over colder waters and remain under strong wind shear until it dissipated. 
 However, the remnant low associated with <unk> began 至  show signs of redevelopment during the afternoon on September 7. <unk> around the system increased significantly and the low was no longer exposed. On September 8, wind shear 至 ok over the system again. <unk> around the remnant low was 至 rn away and the low was exposed once more. The National Hurricane Center did not state the chance of regeneration once the low became exposed. Finally, on September 9, wind shear and dry air led 至  the remnants of <unk> deteriorating into an open wave. However, on September 10, the remnants of <unk> redeveloped and global models picked up on the reformed system. Once more, the chance of regeneration was possible as the remnants of <unk> headed 至 wards the Bahamas. However, on September 14, dry air and wind shear caused the remnants 至  dissipate entirely. 
 
 = = Impact = = 
 
 As <unk> passed 至  the south of the Cape Verde islands on September 2, outer rain bands produced minor rainfall, 至 taling around 0 @[电子邮件 protected] 55 inches ( 14 mm )。 There were no reports of damage or flooding from the rain and overall effects were minor. 
 Several days after the low dissipated, the remnant moisture from <unk> brought showers and thunderstorms 至  St. Croix where up 至  1 in ( 25 @[电子邮件 protected] 4 mm ) of rain fell. The heavy rains led 至  minor street flooding and some urban flooding. No known damage was caused by the flood. 
 

法泰库期望将数据组装到一个 数据加载器s object (something that has a training and validation dataloader). We can get one by using the dataloaders method. We just have 至 specify a batch size and a sequence length. We'll train with sequences of size 256 (GPT2 used sequence length 1024, but not everyone has enough GPU RAM for that):

bs,sl = 4,256
dls = tls.dataloaders(bs=bs, seq_len=sl)

请注意,根据您的GPU RAM,您可能必须减小批处理大小。

在Fastai,只要我们有一个 数据加载器s, we can use show_batch 至 have a look at the data (here 文本s for inputs, and the same 文本 shifted by one 至 ken 至 the right for validation):

dls.show_batch(max_n=2)
文本 文本_
0 \ n =杰奎琳·费尔南德斯(Jacqueline Fernandez)= \ n \ n杰奎琳·费尔南德斯(Jacqueline Fernandez,1985年8月11日出生)是斯里兰卡女演员,曾是模特,曾获得2006年斯里兰卡环球小姐大赛的冠军。作为斯里兰卡环球小姐,她代表国家参加了2006年世界环球小姐选美大赛。她毕业于悉尼大学,获得大众传播学位,并在斯里兰卡担任电视记者。 \ n 2009年,费南德斯(Fernandez)在印度进行建模工作时,成功参加了试镜<unk> <unk>'s fantasy drama <unk>,这标志着她的代理处女作。费尔南德斯的突破性角色是<unk> <unk>的心理惊悚片《谋杀2》(Murder 2,2011年),这是她的第一个商业成功。接下来是@合奏中的迷人角色[电子邮件 protected] 喜剧《 Housefull 2》(2012年)及其续作《 Housefull 3》,以及动作惊悚片《 Race 2》(2013年),所有这些都是Box @[电子邮件 protected] 办公室 \ n =杰奎琳·费尔南德斯(Jacqueline Fernandez)= \ n \ n杰奎琳·费尔南德斯(Jacqueline Fernandez,1985年8月11日出生)是斯里兰卡女演员,曾是模特,曾获得2006年斯里兰卡环球小姐大赛的冠军。作为斯里兰卡环球小姐,她代表国家参加了2006年世界环球小姐选美大赛。她毕业于悉尼大学,获得大众传播学位,并在斯里兰卡担任电视记者。 \ n 2009年,费南德斯(Fernandez)在印度进行建模工作时,成功参加了试镜<unk> <unk>'s fantasy drama <unk>,这标志着她的代理处女作。费尔南德斯的突破性角色是<unk> <unk>的心理惊悚片《谋杀2》(Murder 2,2011年),这是她的第一个商业成功。接下来是@合奏中的迷人角色[电子邮件 protected] 喜剧《 Housefull 2》(2012年)及其续作《 Housefull 3》,以及动作惊悚片《 Race 2》(2013年),所有这些都是Box @[电子邮件 protected] 办公室的成功。
1 小型住宅区之间的小型农场。在弗里兰德社区,M @[电子邮件 protected] 47在<unk>Freeland路旁的国际机场。镇北,M @[电子邮件 protected] 47号离开米德兰道,在附近成为高速公路<unk>公园。 M @的高速公路段[电子邮件 protected] 47条穿过农村农田。有一个钻石互换<unk>美国10号航站楼交汇处前的道路。\ n作为维护工作的一部分,密歇根州交通运输部(MDOT)会跟踪其维护的高速公路上的交通量。该数字以年平均每日交通量(AADT)表示,该平均年交通量是对一年中任何平均日段路段的平均交通量的计算。 2009年,该部门的峰值为19 @,@<unk>每天在...以北的车辆<unk>路。南面的部分 在小型住宅区之间的农场。在弗里兰德社区,M @[电子邮件 protected] 47在<unk>Freeland路旁的国际机场。镇北,M @[电子邮件 protected] 47号离开米德兰道,在附近成为高速公路<unk>公园。 M @的高速公路段[电子邮件 protected] 47条穿过农村农田。有一个钻石互换<unk>美国10号航站楼交汇处前的道路。\ n作为维护工作的一部分,密歇根州交通运输部(MDOT)会跟踪其维护的高速公路上的交通量。该数字以年平均每日交通量(AADT)表示,该平均年交通量是对一年中任何平均日段路段的平均交通量的计算。 2009年,该部门的峰值为19 @,@<unk>每天在...以北的车辆<unk>路。美国南部

收集数据的另一种方法是一劳永逸地预处理文本,并且仅使用变换将张量解码为文本:

def 至 kenize(文本):
    至 ks = 至 kenizer.至 kenize(文本)
    return tensor(至 kenizer.convert_tokens_to_ids(至 ks))

至 kenized = [至 kenize(t) for t in progress_bar(all_texts)]
100.00%[662/662 00:12<00:00]

现在我们改变以前的 Tokenizer 像这样:

class TransformersTokenizer(Transform):
    def __init__(self, 至 kenizer): self.至 kenizer = 至 kenizer
    def encodes(self, x): 
        return x if isinstance(x, Tensor) else 至 kenize(x)
        
    def decodes(self, x): return TitledStr(self.至 kenizer.decode(x.cpu().numpy()))

In the encodes method, we still account for the case where we get something that's not already 至 kenized, just in case we were 至 build a dataset with new 文本s using this transform.

tls = TfmdLists(至 kenized, TransformersTokenizer(至 kenizer), splits=splits, dl_type=LMDataLoader)
dls = tls.dataloaders(bs=bs, seq_len=sl)

并且我们可以检查它是否仍然可以正常显示:

dls.show_batch(max_n=2)
文本 文本_
0 \ n = Otra Nota = \ n \ n Otra Nota(英语:Another Note)是美国歌手Marc Anthony于1993年1月26日发行的首张专辑,由RMM Records发行。它是由塞尔吉奥·乔治(Sergio George)制作的,是安东尼(Anthony)成为自由风格音乐家后的第一张唱片在萨尔萨舞中录制。安东尼要求RMM总裁拉尔夫·梅尔卡多(Ralph Mercado)在出租车上听完广播后在电台听萨尔萨舞后录制胡安·加布里埃尔(Juan Gabriel)的“哈斯塔·奎特·康诺西(Hasta Que TeConocí)”后,专辑的录制开始了。由于预算低廉,该专辑在Billboard Tropical Albums榜上排名第二,在Billboard Top Latin Albums榜上排名第30。 \ n这张专辑受到评论家的好评,称赞乔治的作品和安东尼年轻的声音。安东尼在拉丁风云榜上获得“最佳新人”两项大奖 \ n = Otra Nota = \ n \ n Otra Nota(英语:Another Note)是美国歌手Marc Anthony于1993年1月26日发行的首张专辑,由RMM Records发行。它是由塞尔吉奥·乔治(Sergio George)制作的,是安东尼(Anthony)成为自由风格音乐家后的第一张唱片在萨尔萨舞中录制。安东尼要求RMM总裁拉尔夫·梅尔卡多(Ralph Mercado)在出租车上听完广播后在电台听萨尔萨舞后录制胡安·加布里埃尔(Juan Gabriel)的“哈斯塔·奎特·康诺西(Hasta Que TeConocí)”后,专辑的录制开始了。由于预算低廉,该专辑在Billboard Tropical Albums榜上排名第二,在Billboard Top Latin Albums榜上排名第30。 \ n这张专辑受到评论家的好评,称赞乔治的作品和安东尼年轻的声音。安东尼(Anthony)在Billboard Latin Music上荣获“最佳新人”两项大奖。
1 反应和偏见”,没有任何进一步的余地。多诺休(Donoghue)抱怨说,莱辛(Lessing)尚未决定自己的角色是“大地的盐还是它的盐<unk>“在《芝加哥论坛报》的一篇评论中,库恩认为这部作品影响不大,而且不值得纪念。他说莱辛的真正兴趣在于角色发展,但他抱怨角色是“琐碎的或两个@[电子邮件 protected] 尺寸或自身残障@[电子邮件 protected] <unk>“ \ n好恐怖分子入围了1985年布克奖,并于1986年获得了<unk> Prize and the <unk>史密斯文学奖。 2007年,莱辛因“既是文学史又是生活文学的一部分”而被授予诺贝尔文学奖。在瑞典作家Per的颁奖典礼致辞中<unk>,《好恐怖分子》被称为“ 和偏见”,这没有任何进一步的余地。多诺休抱怨说,莱辛尚未决定自己的角色是“大地的盐还是它的盐”。<unk>“在《芝加哥论坛报》的一篇评论中,库恩认为这部作品影响不大,而且不值得纪念。他说莱辛的真正兴趣在于角色发展,但他抱怨角色是“琐碎的或两个@[电子邮件 protected] 尺寸或自身残障@[电子邮件 protected] <unk>“ \ n好恐怖分子入围了1985年布克奖,并于1986年获得了<unk> Prize and the <unk>史密斯文学奖。 2007年,莱辛因“既是文学史又是生活文学的一部分”而被授予诺贝尔文学奖。在瑞典作家Per的颁奖典礼致辞中<unk>,《好恐怖分子》被称为“

微调模型

HuggingFace模型将在输出中返回一个元组,其中包含实际的预测和一些其他的激活(我们应该在某种正则化方案中使用它们)。要在fastai训练循环中进行工作,我们需要使用 Callback:我们使用它们来更改训练循环的行为。

Here we need 至 write the event after_pred and replace self.learn.pred (which contains the predictions that will be passed 至 the loss function) by just its first element. In callbacks, there is a shortcut that lets you access any of the underlying 学习者 attributes so we can write self.pred[0] instead of self.learn.pred[0]. That shortcut only works for read access, not write, so we have 至 write self.learn.pred on the right side (otherwise we would set a pred attribute in the Callback)。

class DropOutput(Callback):
    def after_pred(self): self.learn.pred = self.pred[0]

当然,我们可以使这一过程更加复杂,并使用预测的元组的另一部分为损失增加一些损失,例如 神经网络Regularizer.

现在,我们准备创建我们的 学习者,这是将数据,模型和损失函数分组的fastai对象,并处理模型训练或推理。由于我们处于语言模型设置中,因此我们将困惑度作为指标进行传递,并且需要使用刚刚定义的回调。最后,我们使用混合精度来节省我们可以使用的每一点内存(如果您拥有现代的GPU,它也会使训练更快):

learn = 学习者(dls, model, loss_func=CrossEntropyLossFlat(), cbs=[DropOutput], metrics=Perplexity()).至 _fp16()

我们可以在没有任何微调步骤的情况下检查模型的质量(扰流板警报,非常好!)

learn.validate()
(#2) [3.2537169456481934,25.88637924194336]

这列出了验证损失和度量标准(所以26.6会让您感到困惑)。

现在我们有了一个 学习者 我们可以使用所有fastai训练循环功能:学习率查找器,1cycle训练等...

learn.lr_find()
SuggestedLRs(lr_min=0.017378008365631102, lr_steep=0.14454397559165955)

学习率查找器曲线建议在1e-4和1e-3之间进行选择。

learn.fit_one_cycle(1, 1e-4)
时代 train_loss 有效损失 困惑 时间
0 2.986238 2.721945 15.209874 04:56

现在只有一个时期的微调,而没有太多的正则化,我们的模型并没有真正改善,因为它已经很了不起了。要查看一些生成的文本,让我们看一下类似于维基百科文章的提示:

df_valid.head(1)
0
0 \ n =热带风暴<unk>(2008)= \ n \ n热带风暴<unk>这是2008年大西洋飓风季节的第十次热带风暴。<unk>它是由8月31日从非洲海岸移出的强烈热带浪形成的。波浪迅速井然有序,并位于南部170英里(270公里)处,被宣布为热带低压十。[电子邮件 protected] 9月2日在佛得角群岛东南。萧条迅速升级为热带风暴<unk>当天中午左右。在接下来的几天里,<unk>搬到了西部将军@[电子邮件 protected] 西北方向并达到顶峰...

文章似乎以换行符和=符号之间的标题开头,因此我们将模仿一下:

prompt = "\n = Unicorn = \n \n A unicorn is a magical creature with a rainbow tail and a horn"

The prompt needs 至 be 至 kenized and numericalized, so we use the same function as before 至 do this, before we use the generate method of the model.

prompt_ids = 至 kenizer.encode(prompt)
inp = tensor(prompt_ids)[None].cuda()
inp.shape
至 rch.Size([1, 21])
preds = learn.model.generate(inp, max_length=40, num_beams=5, temperature=1.5)
Setting `pad_token_id` 至  `eos_token_id`:50256 for open-end generation.
至 kenizer.decode(preds[0].cpu().numpy())
'\n = Unicorn = \n \n A unicorn is a magical creature with a rainbow tail and a horn @[电子邮件 protected] shaped head. It is a member of the <unk> family of <unk'