Detección de anomalías con embeddings

# @title Licensed under the Apache License, Version 2.0 (the "License");
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

%pip install -U -q google-genai

import re
import tqdm
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from google import genai
from google.genai import types

# Used to securely store your API key
from google.colab import userdata

from sklearn.datasets import fetch_20newsgroups
from sklearn.manifold import TSNE

# Or use `os.getenv('GEMINI_API_KEY')` to fetch an environment variable.
GEMINI_API_KEY=userdata.get('GEMINI_API_KEY')

client = genai.Client(api_key=GEMINI_API_KEY)

for m in client.models.list():
  if 'embedContent' in m.supported_actions:
    print(m.name)

models/embedding-001
models/text-embedding-004
models/gemini-embedding-exp-03-07
models/gemini-embedding-exp
models/gemini-embedding-001

MODEL_ID = "gemini-embedding-001" # @param ["gemini-embedding-2-preview", "gemini-embedding-001"] {"allow-input":true, isTemplate: true}

newsgroups_train = fetch_20newsgroups(subset='train')

# View list of class names for dataset
newsgroups_train.target_names

['alt.atheism',
 'comp.graphics',
 'comp.os.ms-windows.misc',
 'comp.sys.ibm.pc.hardware',
 'comp.sys.mac.hardware',
 'comp.windows.x',
 'misc.forsale',
 'rec.autos',
 'rec.motorcycles',
 'rec.sport.baseball',
 'rec.sport.hockey',
 'sci.crypt',
 'sci.electronics',
 'sci.med',
 'sci.space',
 'soc.religion.christian',
 'talk.politics.guns',
 'talk.politics.mideast',
 'talk.politics.misc',
 'talk.religion.misc']

idx = newsgroups_train.data[0].index('Lines')
print(newsgroups_train.data[0][idx:])

Lines: 15

 I was wondering if anyone out there could enlighten me on this car I saw
the other day. It was a 2-door sports car, looked to be from the late 60s/
early 70s. It was called a Bricklin. The doors were really small. In addition,
the front bumper was separate from the rest of the body. This is 
all I know. If anyone can tellme a model name, engine specs, years
of production, where this car is made, history, or whatever info you
have on this funky looking car, please e-mail.

Thanks,
- IL
   ---- brought to you by your neighborhood Lerxst ----

# Apply functions to remove names, emails, and extraneous words from data points in newsgroups.data
newsgroups_train.data = [re.sub(r'[\w\.-]+@[\w\.-]+', '', d) for d in newsgroups_train.data] # Remove email
newsgroups_train.data = [re.sub(r"\([^()]*\)", "", d) for d in newsgroups_train.data] # Remove names
newsgroups_train.data = [d.replace("From: ", "") for d in newsgroups_train.data] # Remove "From: "
newsgroups_train.data = [d.replace("\nSubject: ", "") for d in newsgroups_train.data] # Remove "\nSubject: "

# Cut off each text entry after 5,000 characters
newsgroups_train.data = [d[0:5000] if len(d) > 5000 else d for d in newsgroups_train.data]

# Put training points into a dataframe
df_train = pd.DataFrame(newsgroups_train.data, columns=['Text'])
df_train['Label'] = newsgroups_train.target
# Match label to target name index
df_train['Class Name'] = df_train['Label'].map(newsgroups_train.target_names.__getitem__)

df_train

Text  Label  \
0       WHAT car is this!?\nNntp-Posting-Host: rac3.w...      7   
1       SI Clock Poll - Final Call\nSummary: Final ca...      4   
2       PB questions...\nOrganization: Purdue Univers...      4   
3       Re: Weitek P9000 ?\nOrganization: Harris Comp...      1   
4       Re: Shuttle Launch Question\nOrganization: Sm...     14   
...                                                  ...    ...   
11309    Re: Migraines and scans\nDistribution: world...     13   
11310  Screen Death: Mac Plus/512\nLines: 22\nOrganiz...      4   
11311   Mounting CPU Cooler in vertical case\nOrganiz...      3   
11312   Re: Sphere from 4 points?\nOrganization: Cent...      1   
11313   stolen CBR900RR\nOrganization: California Ins...      8   

                     Class Name  
0                     rec.autos  
1         comp.sys.mac.hardware  
2         comp.sys.mac.hardware  
3                 comp.graphics  
4                     sci.space  
...                         ...  
11309                   sci.med  
11310     comp.sys.mac.hardware  
11311  comp.sys.ibm.pc.hardware  
11312             comp.graphics  
11313           rec.motorcycles  

[11314 rows x 3 columns]

# Take a sample of each label category from df_train
SAMPLE_SIZE = 150
df_train = (df_train.groupby('Label', as_index = False)
                    .apply(lambda x: x.sample(SAMPLE_SIZE))
                    .reset_index(drop=True))

# Choose categories about science
df_train = df_train[df_train['Class Name'].str.contains('sci')]

# Reset the index
df_train = df_train.reset_index()
df_train

/tmp/ipykernel_100019/406673449.py:4: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  .apply(lambda x: x.sample(SAMPLE_SIZE))

index                                               Text  Label  \
0     1650   Privacy & Anonymity on the Internet FAQ \nSup...     11   
1     1651   The source of that announcement\nOrganization...     11   
2     1652   Would "clipper" make a good cover for other e...     11   
3     1653   What the clipper nay-sayers sound like to me....     11   
4     1654   alt.security.pgp\nNntp-Posting-Host: bootes.c...     11   
..     ...                                                ...    ...   
595   2245   Re: Terraforming Venus: can it be done "cheap...     14   
596   2246   Re: *Doppelganger* \nArticle-I.D.: mojo.1qkn6...     14   
597   2247   Re: What if the USSR had reached the Moon fir...     14   
598   2248   Re: DC-X Rollout Report\nArticle-I.D.: topaz....     14   
599   2249   Re: Vandalizing the sky.\nOrganization: Locus...     14   

    Class Name  
0    sci.crypt  
1    sci.crypt  
2    sci.crypt  
3    sci.crypt  
4    sci.crypt  
..         ...  
595  sci.space  
596  sci.space  
597  sci.space  
598  sci.space  
599  sci.space  

[600 rows x 4 columns]

df_train['Class Name'].value_counts()

Class Name
sci.crypt          150
sci.electronics    150
sci.med            150
sci.space          150
Name: count, dtype: int64

Genera los embeddings

En esta sección, verás cómo generar embeddings para los diferentes textos en el dataframe usando los embeddings de la API de Gemini.

El modelo de embedding de Gemini admite varios tipos de tareas, cada uno adaptado a un objetivo específico. Aquí tienes una descripción general de los tipos disponibles y sus aplicaciones:

from tqdm.auto import tqdm
tqdm.pandas()

def make_embed_text_fn(model):

  def embed_fn(text: str) -> list[float]:
    # Set the task_type to CLUSTERING.
    result = client.models.embed_content(model=model,
                                            contents=text,
                                            config=types.EmbedContentConfig(
                                                task_type="CLUSTERING"))
    return np.array(result.embeddings[0].values)

  return embed_fn

def create_embeddings(df):
  df['Embeddings'] = df['Text'].progress_apply(make_embed_text_fn(MODEL_ID))
  return df

df_train = create_embeddings(df_train)
df_train.drop('index', axis=1, inplace=True)

100%|██████████| 600/600 [06:11<00:00,  1.62it/s]

Reducción de dimensionalidad

La dimensión del vector de embedding del documento es 3072. Para visualizar cómo se agrupan los documentos embebidos, deberás aplicar la reducción de dimensionalidad, ya que solo puedes visualizar los embeddings en un espacio 2D o 3D. Los documentos contextualmente similares deben estar más cerca en el espacio en comparación con los documentos que no son tan similares.

len(df_train['Embeddings'][0])

3072

# Convert df_train['Embeddings'] Pandas series to a np.array of float32
X = np.array(df_train['Embeddings'].to_list(), dtype=np.float32)
X.shape

(600, 3072)

Aplicarás el enfoque de t-Distributed Stochastic Neighbor Embedding (t-SNE) para realizar la reducción de dimensionalidad. Esta técnica reduce el número de dimensiones, mientras preserva los clústeres (los puntos que están cerca permanecen cerca). Para los datos originales, el modelo intenta construir una distribución sobre la cual otros puntos de datos son "vecinos" (por ejemplo, comparten un significado similar). Luego optimiza una función objetivo para mantener una distribución similar en la visualización.

tsne = TSNE(random_state=0)
tsne_results = tsne.fit_transform(X)

df_tsne = pd.DataFrame(tsne_results, columns=['TSNE1', 'TSNE2'])
df_tsne['Class Name'] = df_train['Class Name'] # Add labels column from df_train to df_tsne
df_tsne

TSNE1      TSNE2 Class Name
0    39.326805 -15.291752  sci.crypt
1    25.044193  -2.996266  sci.crypt
2    30.786301  -6.652494  sci.crypt
3    26.256178  -9.720546  sci.crypt
4    41.185246  -8.945529  sci.crypt
..         ...        ...        ...
595 -25.796329   8.991899  sci.space
596 -20.514021   0.129148  sci.space
597 -22.480268   6.861408  sci.space
598 -23.748753  14.050385  sci.space
599 -22.166462   4.321325  sci.space

[600 rows x 3 columns]

fig, ax = plt.subplots(figsize=(8,6)) # Set figsize
sns.set_style('darkgrid', {"grid.color": ".6", "grid.linestyle": ":"})
sns.scatterplot(data=df_tsne, x='TSNE1', y='TSNE2', hue='Class Name', palette='Set2')
sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
plt.title('Scatter plot of news using t-SNE')
plt.xlabel('TSNE1')
plt.ylabel('TSNE2')

Text(0, 0.5, 'TSNE2')

<Figure size 800x600 with 1 Axes>

Detección de valores atípicos

Para determinar qué puntos son anómalos, determinarás qué puntos son valores internos y valores atípicos. Comienza por encontrar el centroide, o la ubicación que representa el centro del clúster, y usa la distancia para determinar los puntos que son valores atípicos.

Comienza por obtener el centroide de cada categoría.

def get_centroids(df_tsne):
  # Get the centroid of each cluster
  centroids = df_tsne.groupby('Class Name').mean()
  return centroids

centroids = get_centroids(df_tsne)
centroids

TSNE1      TSNE2
Class Name                           
sci.crypt        31.146711  -6.366441
sci.electronics   7.322525  19.477852
sci.med          -9.485087 -22.437294
sci.space       -23.256557   9.471162

def get_embedding_centroids(df):
  emb_centroids = dict()
  grouped = df.groupby('Class Name')
  for c in grouped.groups:
    sub_df = grouped.get_group(c)
    # Get the centroid value of dimension 768
    emb_centroids[c] = np.mean(sub_df['Embeddings'], axis=0)

  return emb_centroids

emb_c = get_embedding_centroids(df_train)

Traza cada centroide que has encontrado contra el resto de los puntos.

# Plot the centroids against the cluster
fig, ax = plt.subplots(figsize=(8,6)) # Set figsize
sns.set_style('darkgrid', {"grid.color": ".6", "grid.linestyle": ":"})
sns.scatterplot(data=df_tsne, x='TSNE1', y='TSNE2', hue='Class Name', palette='Set2');
sns.scatterplot(data=centroids, x='TSNE1', y='TSNE2', color="black", marker='X', s=100, label='Centroids')
sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
plt.title('Scatter plot of news using t-SNE with centroids')
plt.xlabel('TSNE1')
plt.ylabel('TSNE2');

<Figure size 800x600 with 1 Axes>

Elige un radio. Cualquier cosa más allá de este límite desde el centroide de esa categoría se considera un valor atípico.

def calculate_euclidean_distance(p1, p2):
  return np.sqrt(np.sum(np.square(p1 - p2)))

def detect_outlier(df, emb_centroids, radius):
  for idx, row in df.iterrows():
    class_name = row['Class Name'] # Get class name of row
    # Compare centroid distances
    dist = calculate_euclidean_distance(row['Embeddings'],
                                        emb_centroids[class_name])
    df.at[idx, 'Outlier'] = dist > radius

  return len(df[df['Outlier'] == True])

range_ = np.arange(0.3, 0.75, 0.02).round(decimals=2).tolist()
num_outliers = []
for i in range_:
  num_outliers.append(detect_outlier(df_train, emb_c, i))

# Plot range_ and num_outliers
fig = plt.figure(figsize = (14, 8))
plt.rcParams.update({'font.size': 12})
plt.bar(list(map(str, range_)), num_outliers)
plt.title("Number of outliers vs. distance of points from centroid")
plt.xlabel("Distance")
plt.ylabel("Number of outliers")
for i in range(len(range_)):
  plt.text(i, num_outliers[i], num_outliers[i], ha = 'center')

plt.show()

<Figure size 1400x800 with 1 Axes>

Dependiendo de cuán sensible quieras que sea tu detector de anomalías, puedes elegir qué radio deseas usar. Por ahora, se usa 0.58, pero puedes cambiar este valor.

# View the points that are outliers
RADIUS = 0.54
detect_outlier(df_train, emb_c, RADIUS)
df_outliers = df_train[df_train['Outlier'] == True]
df_outliers.head()

Text  Label  \
90   Cryptography FAQ 03/10 - Basic Cryptology\nOrg...     11   
97    [Rubick] Shortest Path Algorithm - Status?\nO...     11   
147   Re: Trinomial-Based PRNG\nOrganization: Schoo...     11   
181  Electric power line "balls"\nArticle-I.D.: alm...     12   
237  Electrical wiring FAQ (was: A question about 1...     12   

          Class Name                                         Embeddings  \
90         sci.crypt  [-0.03167741, -0.00045672545, -0.01068269, -0....   
97         sci.crypt  [-0.022985524, -0.024591176, -0.011402696, -0....   
147        sci.crypt  [-0.025036471, -0.005380305, -0.00593743, -0.0...   
181  sci.electronics  [-0.014121288, 0.00074380956, 0.007506555, -0....   
237  sci.electronics  [-0.0092245415, -0.022044295, 0.0032056107, -0...   

    Outlier  
90     True  
97     True  
147    True  
181    True  
237    True

# Use the index to map the outlier points back to the projected TSNE points
outliers_projected = df_tsne.loc[df_outliers['Outlier'].index]

Traza los valores atípicos y denótalos usando un color rojo transparente.

fig, ax = plt.subplots(figsize=(8,6)) # Set figsize
plt.rcParams.update({'font.size': 10})
sns.set_style('darkgrid', {"grid.color": ".6", "grid.linestyle": ":"})
sns.scatterplot(data=df_tsne, x='TSNE1', y='TSNE2', hue='Class Name', palette='Set2');
sns.scatterplot(data=centroids, x='TSNE1', y='TSNE2', color="black", marker='X', s=100, label='Centroids')
# Draw a red circle around the outliers
sns.scatterplot(data=outliers_projected, x='TSNE1', y='TSNE2', color='red', marker='o', alpha=0.5, s=90, label='Outliers')
sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
plt.title('Scatter plot of news with outliers projected with t-SNE')
plt.xlabel('TSNE1')
plt.ylabel('TSNE2')

Text(0, 0.5, 'TSNE2')

<Figure size 800x600 with 1 Axes>

Usa los valores de índice de los dataframes para imprimir algunos ejemplos de cómo pueden verse los valores atípicos en cada categoría. Aquí, se imprime el primer punto de datos de cada categoría. Explora otros puntos en cada categoría para ver datos que se consideran valores atípicos o anomalías.

sci_electronics_outliers = df_outliers[df_outliers['Class Name'] == 'sci.electronics']
print(sci_electronics_outliers['Text'].iloc[0])

Electric power line "balls"
Article-I.D.: almaden.19930406.142616.248
Lines: 4

Power lines and airplanes don't mix. In areas where lines are strung very
high, or where a lot of crop dusting takes place, or where there is danger
of airplanes flying into the lines, they place these plastic balls on the
lines so they are easier to spot.

sci_space_outliers = df_outliers[df_outliers['Class Name'] == 'sci.space']
print(sci_space_outliers['Text'].iloc[0])

LARSONIAN Astronomy and Physics
Organization: University of Wisconsin Eau Claire
Lines: 552



                      LARSONIAN Astronomy and Physics

               Orthodox physicists, astronomers, and astrophysicists 
          CLAIM to be looking for a "Unified Field Theory" in which all 
          of the forces of the universe can be explained with a single 
          set of laws or equations.  But they have been systematically 
          IGNORING or SUPPRESSING an excellent one for 30 years! 

               The late Physicist Dewey B. Larson's comprehensive 
          GENERAL UNIFIED Theory of the physical universe, which he 
          calls the "Reciprocal System", is built on two fundamental 
          postulates about the physical and mathematical natures of 
          space and time: 
    
                "The physical universe is composed ENTIRELY of ONE 
          component, MOTION, existing in THREE dimensions, in DISCRETE 
          UNITS, and in two RECIPROCAL forms, SPACE and TIME." 
    
                "The physical universe conforms to the relations of 
          ORDINARY COMMUTATIVE mathematics, its magnitudes are 
          ABSOLUTE, and its geometry is EUCLIDEAN." 
    
               From these two postulates, Larson developed a COMPLETE 
          Theoretical Universe, using various combinations of 
          translational, vibrational, rotational, and vibrational-
          rotational MOTIONS, the concepts of IN-ward and OUT-ward 
          SCALAR MOTIONS, and speeds in relation to the Speed of Light 
          . 
      
               At each step in the development, Larson was able to 
          MATCH objects in his Theoretical Universe with objects in the 
          REAL physical universe, , even objects NOT YET 
          DISCOVERED THEN . 
          
               And applying his Theory to his NEW model of the atom, 
          Larson was able to precisely and accurately CALCULATE inter-
          atomic distances in crystals and molecules, compressibility 
          and thermal expansion of solids, and other properties of 
          matter. 

               All of this is described in good detail, with-OUT fancy 
          complex mathematics, in his books. 
    


          BOOKS of Dewey B. Larson
          
               The following is a complete list of the late Physicist 
          Dewey B. Larson's books about his comprehensive GENERAL 
          UNIFIED Theory of the physical universe.  Some of the early 
          books are out of print now, but still available through 
          inter-library loan. 
    
               "The Structure of the Physical Universe"  
    
               "The Case AGAINST the Nuclear Atom" 
    
               "Beyond Newton"  
    
               "New Light on Space and Time"  
    
               "Quasars and Pulsars"  
    
               "NOTHING BUT MOTION"  
                    [A $9.50 SUBSTITUTE for the $8.3 BILLION "Super 
                                                            Collider".] 
                    [The last four chapters EXPLAIN chemical bonding.]

               "The Neglected Facts of Science"  
     
               "THE UNIVERSE OF MOTION" 
                    [FINAL SOLUTIONS to most ALL astrophysical
                                                            mysteries.] 
      
               "BASIC PROPERTIES OF MATTER" 

               All but the last of these books were published by North 
          Pacific Publishers, P.O. Box 13255, Portland, OR  97213, and 
          should be available via inter-library loan if your local 
          university or public library doesn't have each of them. 

               Several of them, INCLUDING the last one, are available 
          from: The International Society of Unified Science , 
          1680 E. Atkin Ave., Salt Lake City, Utah  84106.  This is the 
          organization that was started to promote Larson's Theory.  
          They have other related publications, including the quarterly 
          journal "RECIPROCITY". 

          

          Physicist Dewey B. Larson's Background
    
               Physicist Dewey B. Larson was a retired Engineer 
          .  He was about 91 years old when he 
          died in May 1989.  He had a Bachelor of Science Degree in 
          Engineering Science from Oregon State University.  He 
          developed his comprehensive GENERAL UNIFIED Theory of the 
          physical universe while trying to develop a way to COMPUTE 
          chemical properties based only on the elements used. 
    
               Larson's lack of a fancy "PH.D." degree might be one 
          reason that orthodox physicists are ignoring him, but it is 
          NOT A VALID REASON.  Sometimes it takes a relative outsider 
          to CLEARLY SEE THE FOREST THROUGH THE TREES.  At the same 
          time, it is clear from his books that he also knew ORTHODOX 
          physics and astronomy as well as ANY physicist or astronomer,

Próximos pasos

¡Ahora has creado un detector de anomalías usando embeddings! Intenta usar tus propios datos textuales para visualizarlos como embeddings, y elige algún límite para que puedas detectar valores atípicos. Puedes realizar una reducción de dimensionalidad para completar el paso de visualización. Ten en cuenta que t-SNE es bueno para agrupar entradas, pero puede tardar más tiempo en converger o quedarse atascado en mínimos locales.

Para aprender a usar otros servicios en la API de Gemini, consulta la guía Primeros pasos.