파이썬 멀티프로세싱 겅부중...!
import multiprocessing as mp
from multiprocessing import Pool
import spacy
import time
nlp = spacy.load("ko_core_news_lg")
def multi(text):
start = time.time()
print(mp.cpu_count())
with Pool(mp.cpu_count()) as pool:
pool_top_k_data = pool.map(multi_spacy, text)
pool.close()
pool.join()
print('>>>>>>>>>>>>>>> spacy ', '__________', time.time() - start)
def multi_spacy(text):
doc = nlp(text)
print(doc.text)
for token in doc:
print(token.text, token.pos_, token.ent_type_)
def spacy(text):
start = time.time()
for a in text:
doc = nlp(a)
print(doc.text)
for token in doc:
print(token.text, token.pos_, token.ent_type_)
print('>>>>>>>>>>>>>>> spacy ', '__________', time.time() - start)
'Study' 카테고리의 다른 글
| local에서 ssh 서버에 띄운 jupyter notebook 붙기 (0) | 2022.02.06 |
|---|---|
| BERT 이해하기 (0) | 2021.09.15 |
| Neural Search 관련 공부 (2) | 2021.07.07 |
| 머신러닝 개념 정리 (0) | 2021.05.22 |
