Presentacion tema memoria v1
-
Upload
camilo-lopez-aravena -
Category
Technology
-
view
111 -
download
3
description
Transcript of Presentacion tema memoria v1
Diseño y construcción de una plataforma de clasificación y calificación de post para una red de blogs basada en textmining para Betazeta
Networks S.A.
Camilo López A.
Objetivo General
El objetivo general del trabajo es el apoyo al procesamiento manual de grandes volúmenes de publicaciones en la red de blogs de Betazeta mediante el diseño e implementación de un prototipo para la categorización automática de
estos datos utilizando text mining.
Objetivos Específicos
1. Entender a fondo la problemática y el contexto de la empresa junto con los conocimientos necesarios respecto a text mining, modelos y metodologías necesarias.
2. Selección de los datos históricos, las consultas sobre éstos y los modelos que permitan realizar predicciones exitosas de categorización.
3. Establecer métodos y métricas para la evaluación de la solución propuesta.
4. Utilizando el conocimiento adquirido en los objetivos anteriores, diseñar el proceso de categorización automático de posts.
Objetivos Específicos
5. Diseñar e implementar un prototipo que permita al usuario ingresar información en forma adecuada para su análisis y a la empresa procesarla, filtrarla y publicarla en base a criterios del negocio.
6. Implementación de la metodología de evaluación.
El Problema
betazeta
7,5 millones
Visitas Mensuales
User Generated Content
Volumen
Google Categories
Spam
Filtro de contenido
Filtro de contenido
Categorizar
Background
Teórico
Data Mining
Limpieza del texto
Stemming
Stop Words
Diccionario
Latent Dirichlet Allocation
LDA Latent Dirichlet Allocation
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.
It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using 'Content here, content here', making it look like readable English. Many desktop publishing packages and web page editors now use Lorem Ipsum as their default model text, and a search for 'lorem ipsum' will uncover many web sites still in their infancy. Various versions have evolved over the years, sometimes by accident, sometimes on purpose (injected humour and the like).
Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32 and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good and Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics, very popular during the Renaissance. The first line of Lorem Ipsum, "Lorem ipsum dolor sit amet..", comes from a line in section 1.10.32.
The standard chunk of Lorem Ipsum used since the 1500s is reproduced below for those interested. Sections 1.10.32 and 1.10.33 from "de Finibus Bonorum et Malorum" by Cicero are also reproduced in their exact original form, accompanied by English versions from the 1914 translation by H. Rackham.
There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected anything embarrassing hidden in the middle of text. All the Lorem Ipsum generators on the Internet tend to repeat predefined chunks as necessary, making this the first true generator on the Internet. It uses a dictionary of over 200 Latin words, combined with a handful of model sentence structures, to generate Lorem Ipsum which looks reasonable. The generated Lorem Ipsum is therefore always free from repetition, injected humour, or non-characteristic words etc.
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.
It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using 'Content here, content here', making it look like readable English. Many desktop publishing packages and web page editors now use Lorem Ipsum as their default model text, and a search for 'lorem ipsum' will uncover many web sites still in their infancy. Various versions have evolved over the years, sometimes by accident, sometimes on purpose (injected humour and the like).
Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32 and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good and Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics, very popular during the Renaissance. The first line of Lorem Ipsum, "Lorem ipsum dolor sit amet..", comes from a line in section 1.10.32.
The standard chunk of Lorem Ipsum used since the 1500s is reproduced below for those interested. Sections 1.10.32 and 1.10.33 from "de Finibus Bonorum et Malorum" by Cicero are also reproduced in their exact original form, accompanied by English versions from the 1914 translation by H. Rackham.
There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected anything embarrassing hidden in the middle of text. All the Lorem Ipsum generators on the Internet tend to repeat predefined chunks as necessary, making this the first true generator on the Internet. It uses a dictionary of over 200 Latin words, combined with a handful of model sentence structures, to generate Lorem Ipsum which looks reasonable. The generated Lorem Ipsum is therefore always free from repetition, injected humour, or non-characteristic words etc.
A
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.
It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using 'Content here, content here', making it look like readable English. Many desktop publishing packages and web page editors now use Lorem Ipsum as their default model text, and a search for 'lorem ipsum' will uncover many web sites still in their infancy. Various versions have evolved over the years, sometimes by accident, sometimes on purpose (injected humour and the like).
Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32 and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good and Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics, very popular during the Renaissance. The first line of Lorem Ipsum, "Lorem ipsum dolor sit amet..", comes from a line in section 1.10.32.
The standard chunk of Lorem Ipsum used since the 1500s is reproduced below for those interested. Sections 1.10.32 and 1.10.33 from "de Finibus Bonorum et Malorum" by Cicero are also reproduced in their exact original form, accompanied by English versions from the 1914 translation by H. Rackham.
There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected anything embarrassing hidden in the middle of text. All the Lorem Ipsum generators on the Internet tend to repeat predefined chunks as necessary, making this the first true generator on the Internet. It uses a dictionary of over 200 Latin words, combined with a handful of model sentence structures, to generate Lorem Ipsum which looks reasonable. The generated Lorem Ipsum is therefore always free from repetition, injected humour, or non-characteristic words etc.
A B
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.
It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using 'Content here, content here', making it look like readable English. Many desktop publishing packages and web page editors now use Lorem Ipsum as their default model text, and a search for 'lorem ipsum' will uncover many web sites still in their infancy. Various versions have evolved over the years, sometimes by accident, sometimes on purpose (injected humour and the like).
Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32 and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good and Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics, very popular during the Renaissance. The first line of Lorem Ipsum, "Lorem ipsum dolor sit amet..", comes from a line in section 1.10.32.
The standard chunk of Lorem Ipsum used since the 1500s is reproduced below for those interested. Sections 1.10.32 and 1.10.33 from "de Finibus Bonorum et Malorum" by Cicero are also reproduced in their exact original form, accompanied by English versions from the 1914 translation by H. Rackham.
There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected anything embarrassing hidden in the middle of text. All the Lorem Ipsum generators on the Internet tend to repeat predefined chunks as necessary, making this the first true generator on the Internet. It uses a dictionary of over 200 Latin words, combined with a handful of model sentence structures, to generate Lorem Ipsum which looks reasonable. The generated Lorem Ipsum is therefore always free from repetition, injected humour, or non-characteristic words etc.
A B
C
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.
A
A B C
A
B
C LDA
Descripción de la plataforma
Entrenamiento
Entrenamiento
Data Histórica
Entrenamiento
Data Histórica Limpieza
Entrenamiento
Data Histórica Limpieza Entrenamiento
Entrenamiento
Data Histórica Limpieza Entrenamiento
Scrapping
Entrenamiento
Data Histórica Limpieza Entrenamiento
Stop Words
Frecuencia 1
Frecuencia Transversal
Entrenamiento
Data Histórica Limpieza Entrenamiento
LDA
Almacenamiento
Filtrado
Categorización
Categorización
Texto Plano
Categorización
Limpieza Texto Plano
Categorización
Limpieza Clasificación Texto Plano
Categorización
Limpieza Clasificación Texto Plano
Stop Words
Categorización
Limpieza Clasificación Texto Plano
Stop Words Modelo LDA
Naïve Bayes Multinomial
Otros métodos
Python
Python
Django
Python
Django
Modelo Vista Controlador
Python
Django
Modelo Vista Controlador
MySql
Python
Django
Modelo Vista Controlador
MySql
Web Service
Resultados
Entrenamiento
4.000 Post
1000 FayerWayer
1000 WayerLess
1000 Belelu
1000 Ferplei
5 Temáticas detectadas
chile colo equipo copa partido barcelona futbol universidad jugador partidos
seleccion ex jugadores alexis goles club tecnico fecha sanchez chileno real torneo
madrid final america nacional catolica gran primera argentino estadio jugar
mundial san ahora luego volante delantero clausura campeon agosto nuevo gol
argentina primer sostuvo carlos mejor liga
93% Ferplei
Validación
350 Post
Suma Frecuencia TFIDF NB NBM
Precission 88% 85% 83% 91% 92%
Recall 96% 97% 91% 94% 93%
F-Measure 92% 91% 87% 93% 93%
FayerWayer y Wayerless
Suma Frecuencia TFIDF NB NBM
Precission 91% 93% 79% 86% 84%
Recall 74% 67% 62% 85% 86%
F-Measure 82% 78% 70% 85% 85%
Belelú
Suma Frecuencia TFIDF NB NBM
Precission 96% 94% 91% 100% 100%
Recall 96% 96% 96% 89% 94%
F-Measure 96% 95% 94% 94% 97%
Ferplei
Entrenamiento
14.400 Post
Validación
1.600 Post
Video Juegos Futbol
Musica, Fiestas y Panoramas Telefonia Movil
Automoviles Pareja y Vida Social
Medio Ambiente Global Ciencia y Tecnologia
Medio Ambiente Pequeña Escala Mujer y Sexualidad Familia y Sociedad
Investigación Espacial Tech Gadgets
Servicios y Tecnologia Automoviles: Top Gear
Complejidad
Trabajo Futuro
Mejorar limpieza del texto
Sistema multimodelo
Predicción de Blog
Mejoras del modelo en el tiempo
Incorporar Stemming
Muchas Gracias