一点点spaCy思想食物：易于使用的NLP框架

pip install spacy

python -m spacy download en_core_web_lg

import spacynlp = spacy.load(‘en_core_web_lg’)

sample_text = “Mark Zuckerberg took two days to testify before members of Congress last week, and he apologised for privacy breaches on Facebook. He said that the social media website didnot take a broad enough view of its responsibility, which was a big mistake. He continued to take responsibility for Facebook, saying that he started it, runs it, and he is responsible for what happens at the company. Illinois Senator Dick Durbin asked Zuckerberg whether he would be comfortable sharing the name of the hotel where he stayed the previous night, or the names of the people who he messaged that week. The CEO was startled by the question, and he took about 7 seconds to respond with no.”doc = nlp(sample_text)

sentences = list(doc3.sents)for i in range(len(sentences)): print(sentences[i].text)  print(“Number of characters:”, len(sentences[i].text)) print(“ — — — — — — — — — — — — — — — — — -”)

Mark Zuckerberg took two days to testify before members of Congress last week, and he apologised for privacy breaches on Facebook.Number of characters: 130-----------------------------------He said that the social media website did not take a broad enough view of its responsibility, which was a big mistake.Number of characters: 118-----------------------------------He continued to take responsibility for Facebook, saying that he started it, runs it, and he is responsible for what happens at the company.Number of characters: 140-----------------------------------Illinois Senator Dick Durbin asked Zuckerberg whether he would be comfortable sharing the name of the hotel where he stayed the previous night, or the names of the people who he messaged that week.Number of characters: 197-----------------------------------The CEO was startled by the question, and he took about 7 seconds to respond with no.Number of characters: 85-----------------------------------

from spacy import displacydisplacy.render(doc, style=’ent’, jupyter=True)

for token in doc: print(“{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}\t{7}”.format( token.text, token.idx, token.lemma_, token.is_punct, token.is_space, token.shape_, token.pos_, token.tag_ ))

Mark0markFalseFalseXxxxPROPNNNPZucker.5zucker.FalseFalseXxxxxPROPNNNPtook16takeFalseFalsexxxxVERBVBDtwo21twoFalseFalsexxxNUMCDdays25dayFalseFalsexxxxNOUNNNSto30toFalseFalsexxPARTTOtestify33testifyFalseFalsexxxxVERBVBbefore41beforeFalseFalsexxxxADPINmembers48memberFalseFalsexxxxNOUNNNSof56ofFalseFalsexxADPIN

text: token itselfidx: starting byte of the tokenlemma_: root of the wordis_punct: is it a punctuation symbol or notis_space: is it a space or notshape_: shape of the token to show which letter is the capitalpos_: the simple part of speech tagtag_: the detailed part of speech tag

[-1.7310e-01,  2.0663e-01,  1.6543e-02, ....., -7.3803e-02]

from scipy import spatialcosine_similarity = lambda x, y: 1 — spatial.distance.cosine(x, y)print(“apple vs banana: “, cosine_similarity(nlp.vocab[‘apple’].vector, nlp.vocab[‘banana’].vector))print(“car vs banana: “, cosine_similarity(nlp.vocab[‘car’].vector, nlp.vocab[‘banana’].vector))print(“car vs bus: “, cosine_similarity(nlp.vocab[‘car’].vector, nlp.vocab[‘bus’].vector))print(“tomatos vs banana: “, cosine_similarity(nlp.vocab[‘tomatos’].vector, nlp.vocab[‘banana’].vector))print(“tomatos vs cucumber: “, cosine_similarity(nlp.vocab[‘tomatos’].vector, nlp.vocab[‘cucumber’].vector))

apple vs banana:  0.5831844210624695car vs banana:  0.16172660887241364car vs bus:  0.48169606924057007tomatos vs banana:  0.38079631328582764tomatos vs cucumber:  0.5478045344352722

from scipy import spatial cosine_similarity = lambda x, y: 1 — spatial.distance.cosine(x, y) man = nlp.vocab[‘man’].vectorwoman = nlp.vocab[‘woman’].vectorqueen = nlp.vocab[‘queen’].vectorking = nlp.vocab[‘king’].vectorcalculated_king = man — woman + queenprint(“similarity between our calculated king vector and real king vector:”, cosine_similarity(calculated_king, king))

similarity between our calculated king vector and real king vector: 0.771614134311676