From 3adcc90d04f007fc81425c9236f80756bcfee673 Mon Sep 17 00:00:00 2001 From: Corydon Baylor Date: Thu, 18 Dec 2025 16:36:44 -0500 Subject: [PATCH] added two docs --- modules/snowflake-analytics/nav.adoc | 1 + .../pages/neo4j-insurance-fraud.adoc | 599 ++++++++++++++++++ .../pages/neo4j-retail-recs.adoc | 359 +++++++++++ 3 files changed, 959 insertions(+) create mode 100644 modules/snowflake-analytics/pages/neo4j-insurance-fraud.adoc create mode 100644 modules/snowflake-analytics/pages/neo4j-retail-recs.adoc diff --git a/modules/snowflake-analytics/nav.adoc b/modules/snowflake-analytics/nav.adoc index 265aa55c..6672af66 100644 --- a/modules/snowflake-analytics/nav.adoc +++ b/modules/snowflake-analytics/nav.adoc @@ -1,4 +1,5 @@ ** xref:index.adoc[Graph Analytics for Snowflake] +*** xref:neo4j-retail-recs.adoc[Create Better Recommendations with Node Similarity] *** xref:neo4j-fraud.adoc[Discover Fraudulent Communities] *** xref:neo4j-insurance-fraud.adoc[Discover Fraudulent Insurance Claims with Embeddings] *** xref:neo4j-manufacturing.adoc[Manage Risk with a Digital Twin] diff --git a/modules/snowflake-analytics/pages/neo4j-insurance-fraud.adoc b/modules/snowflake-analytics/pages/neo4j-insurance-fraud.adoc new file mode 100644 index 00000000..d5c5083a --- /dev/null +++ b/modules/snowflake-analytics/pages/neo4j-insurance-fraud.adoc @@ -0,0 +1,599 @@ +:author: corydon baylor +:id: neo4j-insurance-fraud +:categories: snowflake-site:taxonomy/product/analytics, snowflake-site:taxonomy/snowflake-feature/business-intelligence, snowflake-site:taxonomy/industry/financial-services +:summary: How to find fraud using fast_rp and knn in Neo4j Graph Analytics for Snowflake +:environments: web +:status: Published +:feedback-link: https://github.com/Snowflake-Labs/sfguides/issues +:language: en + += Detecting Insurance Fraud Using Graph Algorithms with Neo4j + +Neo4j helps insurance organizations uncover hidden connections and subtle behavioral similarities across claims, policies, and entities at scale. +Neo4j Graph Analytics for Snowflake brings the power of graph embeddings directly into Snowflake, enabling teams to represent complex fraud behavior +as vectors and use k-nearest neighbors to identify suspiciously similar claims, accounts, and actors. With 65{plus} ready-to-use graph algorithms, +organizations can surface emerging fraud rings and coordinated activity—without moving data or leaving Snowflake. + +==== Prerequisites + +The Native App +https://app.snowflake.com/marketplace/listing/GZTDZH40CN/neo4j-neo4j-graph-analytics[Neo4j +Graph Analytics] for Snowflake + +==== What You Will Need: + +* A https://signup.snowflake.com/[Snowflake account] with appropriate +access to databases and schemas. +* Neo4j Graph Analytics application installed from the Snowflake +marketplace. Access the marketplace via the menu bar on the left hand +side of your screen, as seen below: + +image:marketplace.png[] + + +==== What You Will Build: + +* A method to compare complex insurance cases to one another and +identify the ones that are at risk of being fraudulent. + +==== What You Will Learn: + +* How to prepare and project your data for graph analytics +* How to use Weakly Connected Components to identify potential clusters +of fraudulent activity +* How to create node embeddings to understand the structure of the graph +* How to use K-nearest neighbors algorithm to find highly similar nodes +in the graph +* How to read and write directly from and to your Snowflake tables + +=== Loading the Data + +This dataset is designed to model and analyze insurance claims for the +purpose of identifying fraudulent activity using graph analytics. Given +the complexity of the data model, all contextual relationships captured +in the graph will be leveraged when comparing claims. This will enable +deeper insights beyond isolated data points. + +For the purposes of the demo, the database will be named `I++_++DEMO`. +Using the CSV, `insurance++_++claims++_++full.csv`, found +https://github.com/neo4j-product-examples/snowflake-graph-analytics/blob/main/insurance-fraud/insurance_claims_full.csv[here], +we are going to create a new table called +`insurance++_++claims++_++full` via the Snowsight data upload method. + +Follow through this Snowflake +https://docs.snowflake.com/en/user-guide/data-load-web-ui[documentation] +on creating a table from '`Load data using the web interface`'. + +In the pop up, 1. Upload the CSV `insurance++_++claims++_++full.csv` +using the browse option. 2. Under +`Select or create a database and schema`, please create a database with +name `I++_++DEMO`. 3. Under `Select or create a table`, please click on +the '`{plus}`' symbol and create a new table named +`insurance++_++claims++_++full`. + +Now, a new table named `insurance++_++claims++_++full` will be created +under `i++_++demo.public` with the provided CSV. + +==== Import the Notebook + +* We’ve provided a Colab notebook to walk you through each SQL and +Python step—no local setup required! +* Download the .ipynb found +https://github.com/neo4j-product-examples/snowflake-graph-analytics/blob/main/insurance-fraud/Insurance-fraud-snowflake.ipynb[here], +and import the notebook into snowflake. + +==== Permissions + +One of the most usefull aspects of Snowflake is the ability to have +roles with specific permissions, so that you can have many people +working in the same database without worrying about security. The Neo4j +app requires the creation of a few different roles. But before we get +started granting different roles, we need to ensure that you are using +`accountadmin` to grant and create roles. Lets do that now: + +.... +USE ROLE ACCOUNTADMIN; +.... + +Next we can set up the necessary roles, permissions, and resource access +to enable Graph Analytics to operate on the demo data within the +`i++_++demo.public` schema (this schema is where the data will be stored +by default). + +We will create a consumer role (gds++_++role) for users and +administrators, grant the gds++_++role and GDS application access to +read from and write to tables and views, and ensure the future tables +are accessible. We will also provide the application with access to the +compute pool and warehouse resources required to run the graph +algorithms at scale. + +.... +-- Create an account role to manage the GDS application +CREATE ROLE IF NOT EXISTS gds_role; +GRANT APPLICATION ROLE neo4j_graph_analytics.app_user TO ROLE gds_role; +GRANT APPLICATION ROLE neo4j_graph_analytics.app_admin TO ROLE gds_role; + +--Grant permissions for the application to use the database +GRANT USAGE ON DATABASE i_demo TO APPLICATION neo4j_graph_analytics; +GRANT USAGE ON SCHEMA i_demo.public TO APPLICATION neo4j_graph_analytics; + +--Create a database role to manage table and view access +CREATE DATABASE ROLE IF NOT EXISTS gds_db_role; + +GRANT ALL PRIVILEGES ON FUTURE TABLES IN SCHEMA i_demo.public TO DATABASE ROLE gds_db_role; +GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA i_demo.public TO DATABASE ROLE gds_db_role; + +GRANT ALL PRIVILEGES ON FUTURE VIEWS IN SCHEMA i_demo.public TO DATABASE ROLE gds_db_role; +GRANT ALL PRIVILEGES ON ALL VIEWS IN SCHEMA i_demo.public TO DATABASE ROLE gds_db_role; + +GRANT CREATE TABLE ON SCHEMA i_demo.public TO DATABASE ROLE gds_db_role; + + +--Grant the DB role to the application and admin user +GRANT DATABASE ROLE gds_db_role TO APPLICATION neo4j_graph_analytics; +GRANT DATABASE ROLE gds_db_role TO ROLE gds_role; + +GRANT USAGE ON DATABASE I_DEMO TO ROLE GDS_ROLE; +GRANT USAGE ON SCHEMA I_DEMO.PUBLIC TO ROLE GDS_ROLE; + +GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA I_DEMO.PUBLIC TO ROLE GDS_ROLE; +GRANT CREATE TABLE ON SCHEMA I_DEMO.PUBLIC TO ROLE GDS_ROLE; +GRANT SELECT, INSERT, UPDATE, DELETE ON FUTURE TABLES IN SCHEMA I_DEMO.PUBLIC TO ROLE GDS_ROLE; +.... + +Now we will switch to the role we just created: + +.... +use warehouse NEO4J_GRAPH_ANALYTICS_APP_WAREHOUSE; +use role gds_role; +use database i_demo; +use schema public; +.... + +=== Cleaning Our Data + +Duration 5 + +We need our data to be in a particular format in order to work with +Graph Analytics. In general, it should be like so: + +*For the tables representing nodes:* The first column should be called +`nodeId`, which uniquely identifies each node in the graph. + +*For the tables representing relationships:* We need to have columns +called `sourceNodeId` and `targetNodeId`, representing the start and end +nodes of each relationship. + +To get ready for Graph Analytics, reshape your tables as follows: + +==== *For Nodes* + +* *Policy* — one node for each unique policy (policy++_++number). +* *PoliceReportAvailability* — represents whether a police report was +available. +* *Witnesses* — represents witness status related to claims. +* *VehicleMake* — describes the make of the vehicle involved. +* *PolicyState* — represents the U.S. state where the policy was issued. +* *PolicyCSL* — describes the liability coverage (Combined Single Limit) +of the policy. +* *TotalClaimAmountBucket* — categorizes claim severity into Low, +Medium, or High. +* *MonthsAsCustomerBucket* — groups policyholders into Short, Medium, or +Long tenure. + +==== *For Relationships* + +* (Policy)-++[++:HAS++_++POLICE++_++REPORT++]++-++>++(PoliceReportAvailability) +— links each policy to whether a police report was available. +* (Policy)-++[++:HAS++_++WITNESSES++]++-++>++(Witnesses) — links each +policy to its witness status. +* (Policy)-++[++:INVOLVES++_++VEHICLE++]++-++>++(VehicleMake) — connects +each policy to the vehicle make. +* (Policy)-++[++:REGISTERED++_++IN++]++-++>++(PolicyState) — connects +each policy to the issuing state. +* (Policy)-++[++:HAS++_++CSL++]++-++>++(PolicyCSL) — connects each +policy to its CSL coverage level. +* (Policy)-++[++:HAS++_++CLAIM++_++AMOUNT++_++BUCKET++]++-++>++(TotalClaimAmountBucket) +— links each policy to its total claim amount bucket. +* (Policy)-++[++:HAS++_++CUSTOMER++_++TENURE++_++BUCKET++]++-++>++(MonthsAsCustomerBucket) +— links each policy to its customer tenure bucket. + +.... +CREATE OR REPLACE TABLE node_policies AS +SELECT DISTINCT policy_number +FROM i_demo.public.insurance_claims_full; + + +CREATE OR REPLACE TABLE node_police_report_available AS +SELECT DISTINCT police_report_available +FROM i_demo.public.insurance_claims_full; + + +CREATE OR REPLACE TABLE node_witnesses AS +SELECT DISTINCT witnesses +FROM i_demo.public.insurance_claims_full; + + +CREATE OR REPLACE TABLE node_auto_make AS +SELECT DISTINCT auto_make +FROM i_demo.public.insurance_claims_full; + + +CREATE OR REPLACE TABLE node_policy_state AS +SELECT DISTINCT policy_state +FROM i_demo.public.insurance_claims_full; + + +CREATE OR REPLACE TABLE node_policy_csl AS +SELECT DISTINCT policy_csl +FROM i_demo.public.insurance_claims_full; + + +CREATE OR REPLACE TABLE policy_states AS +SELECT ROW_NUMBER() OVER (ORDER BY policy_state) AS state_id, policy_state +FROM ( + SELECT DISTINCT policy_state FROM i_demo.public.insurance_claims_full +); + +CREATE OR REPLACE TABLE node_total_claim_amount_bucket AS +SELECT DISTINCT + CASE + WHEN total_claim_amount < 40000 THEN 'Low' + WHEN total_claim_amount BETWEEN 40000 AND 70000 THEN 'Medium' + WHEN total_claim_amount > 70000 THEN 'High' + ELSE 'Unknown' + END AS total_claim_amount_bucket +FROM i_demo.public.insurance_claims_full; + +CREATE OR REPLACE TABLE node_months_as_customer_bucket AS +SELECT DISTINCT + CASE + WHEN months_as_customer < 100 THEN 'Short (<100m)' + WHEN months_as_customer BETWEEN 100 AND 300 THEN 'Medium (100-300m)' + WHEN months_as_customer > 300 THEN 'Long (>300m)' + ELSE 'Unknown' + END AS months_as_customer_bucket +FROM i_demo.public.insurance_claims_full; +.... + +Now, we will merge all the node tables to a single +`all++_++nodes++_++tbl` + +.... +CREATE OR REPLACE TABLE all_nodes AS +SELECT DISTINCT policy_number::STRING AS nodeid FROM node_policies +UNION +SELECT DISTINCT police_report_available::STRING AS nodeid FROM node_police_report_available +UNION +SELECT DISTINCT witnesses::STRING AS nodeid FROM node_witnesses +UNION +SELECT DISTINCT total_claim_amount_bucket::STRING AS nodeid FROM node_total_claim_amount_bucket +UNION +SELECT DISTINCT auto_make::STRING AS nodeid FROM node_auto_make +UNION +SELECT DISTINCT policy_state::STRING AS nodeid FROM node_policy_state +UNION +SELECT DISTINCT policy_csl::STRING AS nodeid FROM node_policy_csl +UNION +SELECT DISTINCT months_as_customer_bucket::STRING AS nodeid FROM node_months_as_customer_bucket; +.... + +Below we create the relationship tables: + +.... +CREATE OR REPLACE TABLE rel_policy_police_report_available AS +SELECT + policy_number, + police_report_available +FROM i_demo.public.insurance_claims_full; + +CREATE OR REPLACE TABLE rel_policy_witnesses AS +SELECT + policy_number, + witnesses +FROM i_demo.public.insurance_claims_full; + +CREATE OR REPLACE TABLE rel_policy_auto_make AS +SELECT + policy_number, + auto_make +FROM i_demo.public.insurance_claims_full; + +CREATE OR REPLACE TABLE rel_policy_policy_state AS +SELECT + policy_number, + policy_state +FROM i_demo.public.insurance_claims_full; + +CREATE OR REPLACE TABLE rel_policy_policy_csl AS +SELECT + policy_number, + policy_csl +FROM i_demo.public.insurance_claims_full; + +CREATE OR REPLACE TABLE rel_policy_total_claim_amount_bucket AS +SELECT + policy_number, + CASE + WHEN total_claim_amount < 40000 THEN 'Low' + WHEN total_claim_amount BETWEEN 40000 AND 70000 THEN 'Medium' + WHEN total_claim_amount > 70000 THEN 'High' + ELSE 'Unknown' + END AS total_claim_amount_bucket +FROM i_demo.public.insurance_claims_full; + +CREATE OR REPLACE TABLE rel_policy_months_as_customer_bucket AS +SELECT + policy_number, + CASE + WHEN months_as_customer < 100 THEN 'Short (<100m)' + WHEN months_as_customer BETWEEN 100 AND 300 THEN 'Medium (100-300m)' + WHEN months_as_customer > 300 THEN 'Long (>300m)' + ELSE 'Unknown' + END AS months_as_customer_bucket +FROM i_demo.public.insurance_claims_full; +.... + +We will merge all relationships into one big relationship table for +easier analyses. + +.... +CREATE OR REPLACE TABLE all_relationships AS +SELECT policy_number::STRING AS sourcenodeid, police_report_available::STRING AS targetnodeid +FROM rel_policy_police_report_available +UNION +SELECT policy_number::STRING, witnesses::STRING +FROM rel_policy_witnesses +UNION +SELECT policy_number::STRING, total_claim_amount_bucket::STRING +FROM rel_policy_total_claim_amount_bucket +UNION +SELECT policy_number::STRING, auto_make::STRING +FROM rel_policy_auto_make +UNION +SELECT policy_number::STRING, policy_state::STRING +FROM rel_policy_policy_state +UNION +SELECT policy_number::STRING, policy_csl::STRING +FROM rel_policy_policy_csl +UNION +SELECT policy_number::STRING, months_as_customer_bucket::STRING +FROM rel_policy_months_as_customer_bucket; +.... + +=== Insurance Claims Embeddings and Similarity + +Uncovering complex fraud patterns in insurance claims requires more than +tracing obvious links between entities. To detect subtle signals of +collusion or anomalous behavior, we turn to *structural embeddings* — +numerical summaries that capture how each claim fits within the broader +network. + +By transforming the graph structure into a vector space, we can: + +* Detect clusters of claims that fulfill similar structural roles +* Surface outliers whose behavior deviates from typical claim patterns +* Flag candidates for further review based on similarity to known +fraudulent activity + +Our approach leverages two key graph algorithms: + +*Fast Random Projection (FastRP):* This algorithm generates a concise +16-dimensional vector for each claim, reflecting the shape of its +surrounding network. Claims embedded in similar structural contexts — +such as being part of a fraud ring — will yield similar vectors. + +*K-Nearest Neighbors (KNN):* Once embeddings are in place, KNN finds the +most structurally similar claims using cosine similarity. This allows us +to identify networks of claims that may not be directly connected but +exhibit comparable behavior. + +By combining structural embeddings with similarity search, we move +beyond surface-level connections and begin to model how fraud operates +across the entire claims graph. + +You can find more information about these algorithms in our +https://neo4j.com/docs/snowflake-graph-analytics/current/algorithms/[documentation]. + +==== Fast Random Projection (FastRP) + +Fraud patterns often hide behind complex, indirect relationships. FastRP +allows us to translate each claim’s graph position into a compact vector +— a structural fingerprint that captures its role in the broader claims +network. + +These embeddings aren’t directly interpretable, but when two claims have +very similar embeddings, it strongly suggests they occupy comparable +positions in the network. They may share the same types of connections +to incidents, entities, or locations — potentially indicating +coordinated behavior or copycat strategies. + +We compute embeddings as follows: + +.... +CALL Neo4j_Graph_Analytics.graph.fast_rp('CPU_X64_XS', { + 'project': { + 'defaultTablePrefix': 'i_demo.public', + 'nodeTables': ['all_nodes'], + 'relationshipTables': { + 'all_relationships': { + 'sourceTable': 'all_nodes', + 'targetTable': 'all_nodes', + 'orientation': 'UNDIRECTED' + + } + } + }, + 'compute': { + 'mutateProperty': 'embedding', + 'embeddingDimension': 128, + 'randomSeed': 1234 + }, + 'write': [{ + 'nodeLabel': 'all_nodes', + 'outputTable': 'i_demo.public.all_nodes_fast_rp', + 'nodeProperty': 'embedding' + }] +}); +.... + +We can take a look at our embeddings like so: + +.... +SELECT + nodeid, + embedding +FROM i_demo.public.all_nodes_fast_rp; +.... + +[cols=",",options="header",] +|=== +|NODEID |EMBEDDING +|514065 |++[++ 7.488563656806946e-03, -8.751728385686874e-02, …++]++ +|235220 |++[++ 1.217467859387398e-01, -1.637900769710541e-01, …++]++ +|420815 |++[++ 5.846929550170898e-02, -8.481042832136154e-02, …++]++ +|=== + +Now that we have generated node embeddings, we can now proceed to use +these in KNN similarity detection algorithm. + +==== K-Nearest Neighbors (KNN) + +With embeddings in place, KNN helps us find structurally similar claims +— even if they’re not directly connected. It compares the cosine +similarity of embeddings to rank the top matches for each node. + +This is especially useful in fraud detection, where collusive claims may +appear unrelated on the surface but exhibit parallel structural +behavior: similar entity relationships, involvement in incidents with +mirrored patterns, or indirect ties to the same clusters of providers. + +In the context of cosine similarity in the KNN algorithm, a score of: + +* 1.0 means the vectors point in exactly the same direction (perfect +similarity). +* 0.0 means orthogonal (no similarity). +* –1.0 means completely opposite. + +.... +CALL Neo4j_Graph_Analytics.graph.knn('CPU_X64_XS', { + 'project': { + 'defaultTablePrefix': 'i_demo.public', + 'nodeTables': [ 'all_nodes_fast_rp' ], + 'relationshipTables': {} + }, + 'compute': { + 'nodeProperties': ['EMBEDDING'], + 'topK': 3, + 'mutateProperty': 'score', + 'mutateRelationshipType': 'SIMILAR_TO' + }, + 'write': [{ + 'outputTable': 'i_demo.public.claims_knn_similarity', + 'sourceLabel': 'all_nodes_fast_rp', + 'targetLabel': 'all_nodes_fast_rp', + 'relationshipType': 'SIMILAR_TO', + 'relationshipProperty': 'score' + }] +}); +.... + +And now we look at the results: + +.... +SELECT + score, + COUNT(*) AS row_count +FROM i_demo.public.claims_knn_similarity +GROUP BY score +ORDER BY score DESC + +.... + +[cols=",",options="header",] +|=== +|SCORE |ROW++_++COUNT +|1 |144 +|0.9910849541836564 |2 +|0.9909591414045124 |2 +|0.990797909587815 |2 +|0.9907855573714457 |2 +|0.990772737863948 |2 +|0.9907504528107439 |2 +|=== + +In our example dataset, the KNN results show that many nodes have very +high structural similarity scores (mostly above 0.92), indicating they +occupy very similar positions in the graph. This suggests that these +claims or entities may share common patterns or connections, potentially +signaling coordinated behavior. High-scoring pairs are good candidates +for closer review to detect possible collusion or fraud. + +=== Finding Additional Fraud + +We now have pairwise similarity scores between different claims. Let’s +take a look at our original table and find claims that appear to be +structurally the same as fraudulent claims. We are looking for claims +that satisfy two conditions: + +[arabic] +. They are not currently marked as fraudulent +. They have a knn score of "`1`" with a claim that has already been +marked as fraudulent + +This can give us an idea of the universe of potentially missed +fraudulent claims. + +.... +SELECT + icf.policy_number, + icf.fraud_reported +FROM I_DEMO.PUBLIC.INSURANCE_CLAIMS_FULL icf +JOIN I_DEMO.PUBLIC.CLAIMS_KNN_SIMILARITY knn + ON CAST(icf.policy_number AS VARCHAR) = knn.targetnodeid +WHERE knn.score = 1 + AND icf.fraud_reported <> 'Y' + AND EXISTS ( + SELECT 1 + FROM I_DEMO.PUBLIC.INSURANCE_CLAIMS_FULL icf_src + WHERE CAST(icf_src.policy_number AS VARCHAR) = knn.sourcenodeid + AND icf_src.fraud_reported = 'Y' + ); +.... + +[cols=",",options="header",] +|=== +|POLICY++_++NUMBER |FRAUD++_++REPORTED +|866805 |FALSE +|804219 |FALSE +|795004 |FALSE +|731450 |FALSE +|116700 |FALSE +|=== + +=== Conclusion and Resources + +Duration 2 + +In this quickstart, you learned how to bring the power of graph insights +into Snowflake using Neo4j Graph Analytics. + +==== What You Learned + +By working with a Insurance Claims dataset, you were able to: + +[arabic] +. Set up the Neo4j Graph Analytics application within Snowflake. +. Prepare and project your data into a graph model (users as nodes, +transactions as relationships). +. Ran Weakly Connected Components to identify potential clusters of +fraudulent activity. +. Ran Node Embeddings and K Nearest Neighbors to identify the structure +of nodes in the graph and identify highly similar claims. + +==== Resources + +* https://neo4j.com/docs/snowflake-graph-analytics/[Neo4j Graph +Analytics Documentation] +* https://neo4j.com/docs/snowflake-graph-analytics/installation/[Installing +Neo4j Graph Analytics on SPCS] diff --git a/modules/snowflake-analytics/pages/neo4j-retail-recs.adoc b/modules/snowflake-analytics/pages/neo4j-retail-recs.adoc new file mode 100644 index 00000000..c5cf0fc2 --- /dev/null +++ b/modules/snowflake-analytics/pages/neo4j-retail-recs.adoc @@ -0,0 +1,359 @@ + + += Better Recommendations Using Graph Analytics + +Recommendations are big business. Amazon reports that 35% of its revenue +comes from recommendations. Even more surprisingly, Netflix and YouTube +report that 75% and 70% of what people watch on their platforms comes +from recommendations. That means the majority of what we buy, watch, or +even listen to is shaped by algorithms working quietly in the +background. + +We will be using Neo4j Graph Analytics for Snowflake to build our +recommendations. Graph powered recommendations go deeper than +traditional methods because they intuitively model user behavior. + +In our example, we will be looking at co-purchasing behavior built off +of data sampled from Instakart. We will discover how simply looking at +items that are most frequently purchased together isn’t enough to build +a good recommendation, and interestingly might cause us to recommend +products that customers were already planning on buying without our +intervention. + +So how do we build a good recommendation engine? What techniques power +these systems, and how can you start applying them yourself? Well, +follow along to find out! + +== Overview + +=== What Is Neo4j Graph Analytics For Snowflake? + +Neo4j helps organizations find hidden relationships and patterns across +billions of data connections deeply, easily, and quickly. *Neo4j Graph +Analytics for Snowflake* brings to the power of graph directly to +Snowflake, allowing users to run 65{plus} ready-to-use algorithms on +their data, all without leaving Snowflake! + +=== Prerequisites + +* The Native App +https://app.snowflake.com/marketplace/listing/GZTDZH40CN[Neo4j Graph +Analytics] for Snowflake + +=== What You Will Need + +* A https://signup.snowflake.com/?utm_cta=quickstarts[Snowflake account] +with appropriate access to databases and schemas. +* Neo4j Graph Analytics application installed from the Snowflake +marketplace. Access the marketplace via the menu bar on the left hand +side of your screen, as seen below: +image:/Users/corydonbaylor/Documents/md2adoc/md/assets/marketplace.png[image] + +=== What You Will Build + +* A method to identify communities that are at high risk of fraud in P2P +networks + +=== What You Will Learn + +* How to prepare and project your data for graph analytics +* How to use community detection to identify fraud +* How to read and write directly from and to your snowflake tables + +== Loading The Data + +Dataset overview : This dataset is a subset of instakart data and can be +found +https://github.com/neo4j-product-examples/aura-graph-analytics/tree/main/better_recommendations/data[here]. + +Let’s name our database `RETAIL++_++RECS`. We are going to add five new +tables: + +* One called `aisles` based on the aisles.csv +* One called `baskets` based on baskets.csv +* One called `departments` based on departments.csv +* One called `order++_++history` based on order++_++history.csv +* One called `products` based on products.csv + +Follow the steps found +https://docs.snowflake.com/en/user-guide/data-load-web-ui[here] to load +in your data. + +== Simple Similarity using Co-purchase Patterns + +We need our data to be in a particular format in order to work with +Graph Analytics. Let’s start by switching to our new database: + +.... +USE DATABASE RETAIL_RECS; +USE SCHEMA PUBLIC; +USE ROLE ACCOUNTADMIN; +.... + +First, let’s create a co-purchase table to understand what items are +currently co-purchased together. + +.... +-- One row per unordered product pair that appeared in the same order +CREATE OR REPLACE TABLE COPURCHASE AS +WITH DISTINCT_LINES AS ( + SELECT DISTINCT order_id, product_id + FROM BASKETS +), +PAIRS AS ( + SELECT + LEAST(b1.product_id, b2.product_id) AS product_id_a, + GREATEST(b1.product_id, b2.product_id) AS product_id_b + FROM DISTINCT_LINES b1 + JOIN DISTINCT_LINES b2 + ON b1.order_id = b2.order_id + AND b1.product_id < b2.product_id -- avoid self & duplicates +) +SELECT + p.product_id_a, + pa.product_name AS product_name_a, + p.product_id_b, + pb.product_name AS product_name_b, + COUNT(*)::FLOAT AS co_count +FROM PAIRS p +JOIN PRODUCTS pa + ON p.product_id_a = pa.product_id +JOIN PRODUCTS pb + ON p.product_id_b = pb.product_id +GROUP BY + p.product_id_a, + pa.product_name, + p.product_id_b, + pb.product_name; +.... + +.... +select * from copurchase order by co_count desc; +.... + +[width="100%",cols="18%,29%,16%,27%,10%",options="header",] +|=== +|PRODUCT++_++ID++_++A |PRODUCT++_++NAME++_++A |PRODUCT++_++ID++_++B +|PRODUCT++_++NAME++_++B |CO++_++COUNT +|21903 |Organic Baby Spinach |24852 |Banana |24 + +|13176 |Bag of Organic Bananas |47209 |Organic Hass Avocado |22 + +|13176 |Bag of Organic Bananas |21137 |Organic Strawberries |19 + +|24852 |Banana |47766 |Organic Avocado |16 + +|21137 |Organic Strawberries |47209 |Organic Hass Avocado |15 +|=== + +You’ll notice that four out of five of the top co-purchased pairs +include bananas. Logically, I suppose this means that grocery stores +should nearly always recommend bananas to customers – but should they +really? + +== Cleaning our Data + +Next, we are going to put our data into two tables: one for nodes and +one for relationships. We will use these tables later to run a graph +algorithm! + +.... +-- products as nodes; add numeric properties if useful +CREATE OR REPLACE VIEW PRODUCTS_NODES AS +SELECT + product_id AS nodeId +FROM PRODUCTS; +.... + +And for relationships: + +.... +CREATE OR REPLACE TABLE COPURCHASE_EDGES AS +SELECT + a AS SOURCENODEID, + b AS TARGETNODEID, + co_count / NULLIF(pa.cnt + pb.cnt - co_count, 0) AS WEIGHT +FROM ( + SELECT + LEAST(b1.product_id, b2.product_id) AS a, + GREATEST(b1.product_id, b2.product_id) AS b, + COUNT(*)::FLOAT AS co_count + FROM ( + SELECT DISTINCT order_id, product_id FROM BASKETS + ) b1 + JOIN ( + SELECT DISTINCT order_id, product_id FROM BASKETS + ) b2 + ON b1.order_id = b2.order_id + AND b1.product_id < b2.product_id + GROUP BY 1,2 +) pc +JOIN ( + SELECT product_id, COUNT(*)::FLOAT AS cnt + FROM (SELECT DISTINCT order_id, product_id FROM BASKETS) + GROUP BY 1 +) pa ON pa.product_id = pc.a +JOIN ( + SELECT product_id, COUNT(*)::FLOAT AS cnt + FROM (SELECT DISTINCT order_id, product_id FROM BASKETS) + GROUP BY 1 +) pb ON pb.product_id = pc.b; +.... + +== Granting Permissions + +Next, we will grant the necessary permissions for our app to run. Make +sure you are account admin before running this block: + +.... +-- Create a consumer role for users and admins of the GDS application +CREATE ROLE IF NOT EXISTS gds_user_role; +CREATE ROLE IF NOT EXISTS gds_admin_role; +GRANT APPLICATION ROLE neo4j_graph_analytics.app_user TO ROLE gds_user_role; +GRANT APPLICATION ROLE neo4j_graph_analytics.app_admin TO ROLE gds_admin_role; + +CREATE DATABASE ROLE IF NOT EXISTS gds_db_role; +GRANT DATABASE ROLE gds_db_role TO ROLE gds_user_role; +GRANT DATABASE ROLE gds_db_role TO APPLICATION neo4j_graph_analytics; + +-- Grant access to consumer data +GRANT USAGE ON DATABASE RETAIL_RECS TO ROLE gds_user_role; +GRANT USAGE ON SCHEMA RETAIL_RECS.PUBLIC TO ROLE gds_user_role; + +-- Required to read tabular data into a graph +GRANT SELECT ON ALL TABLES IN DATABASE RETAIL_RECS TO DATABASE ROLE gds_db_role; + +-- Ensure the consumer role has access to created tables/views +GRANT ALL PRIVILEGES ON FUTURE TABLES IN SCHEMA RETAIL_RECS.PUBLIC TO DATABASE ROLE gds_db_role; +GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA RETAIL_RECS.PUBLIC TO DATABASE ROLE gds_db_role; +GRANT CREATE TABLE ON SCHEMA RETAIL_RECS.PUBLIC TO DATABASE ROLE gds_db_role; +GRANT CREATE VIEW ON SCHEMA RETAIL_RECS.PUBLIC TO DATABASE ROLE gds_db_role; +GRANT ALL PRIVILEGES ON FUTURE VIEWS IN SCHEMA RETAIL_RECS.PUBLIC TO DATABASE ROLE gds_db_role; +GRANT ALL PRIVILEGES ON ALL VIEWS IN SCHEMA RETAIL_RECS.PUBLIC TO DATABASE ROLE gds_db_role; + +-- Compute and warehouse access +GRANT USAGE ON WAREHOUSE NEO4J_GRAPH_ANALYTICS_APP_WAREHOUSE TO APPLICATION neo4j_graph_analytics; +.... + +Then switch to the role we just created: + +.... +use role gds_role; +.... + +== Running Node Similiarity + +When we recommend items, we should consider some items like hinges to +others. If baby spinach and bananas are bought together and bananas and +avocados are bought together, then perhaps someone who buys avocados +also would want spinach. + +But here’s the rub. If every grocery basket has bananas in it, then it +isn’t a very good hinge. It doesn’t provide a personalized +recommendation about what else someone might want to buy. When bananas +are in every basket, their presence is not a good predictor of what +other items will be in the basket. It’s just noise. + +.... +CALL Neo4j_Graph_Analytics.graph.node_similarity('CPU_X64_XS', { + 'defaultTablePrefix': 'RETAIL_RECS.PUBLIC', + 'project': { + 'nodeTables': ['PRODUCTS_NODES'], + 'relationshipTables': { + 'COPURCHASE_EDGES': { + 'sourceTable': 'PRODUCTS_NODES', + 'targetTable': 'PRODUCTS_NODES' + } + } + }, + 'compute': { + 'mutateProperty': 'score', + 'mutateRelationshipType': 'SIMILAR', + 'topK': 10, + 'similarityMetric': 'JACCARD' + }, + 'write': [{ + 'outputTable': 'PRODUCT_SIMILARITY_JACCARD', + 'sourceLabel': 'PRODUCTS_NODES', + 'targetLabel': 'PRODUCTS_NODES', + 'relationshipType': 'SIMILAR', + 'relationshipProperty': 'score' + }] +}); +.... + +Next, let’s look at the least similar items in our table: + +.... +SELECT + p1.product_name AS source_product_name, + p2.product_name AS target_product_name, + s.SCORE AS similarity_score +FROM PRODUCT_SIMILARITY_JACCARD AS s +JOIN PRODUCTS AS p1 + ON p1.product_id = s.SOURCENODEID +JOIN PRODUCTS AS p2 + ON p2.product_id = s.TARGETNODEID +ORDER BY s.SCORE ASC +LIMIT 5; +.... + +[width="99%",cols="56%,23%,21%",options="header",] +|=== +|SOURCE++_++PRODUCT++_++NAME |TARGET++_++PRODUCT++_++NAME +|SIMILARITY++_++SCORE +|Chocolate Bar Milk Stevia Sweetened Salted Almond |Bag of Organic +Bananas |0.001303780964797914 + +|DairyFree Cheddar Style Wedges |Bag of Organic Bananas +|0.001303780964797914 + +|Grapes Certified Organic California Black Seedless |Bag of Organic +Bananas |0.001303780964797914 + +|Baking Chopped Pecans |Banana |0.001388888888888889 + +|Vegan Crunchy Peanut Butter |Banana |0.001388888888888889 +|=== + +Notice how bananas top the list least similar items. Why? Because their +presence in a basket doesn’t really signal that the customer wants any +other specific item — they’re just a frequent, general-purpose purchase. + +So while bananas are everywhere, they tell us almost nothing about +co-purchase patterns — and *`nodeSimilarity`* correctly learns to +downweight them. Plus, since bananas are in nearly every shopping cart, +our theoretical customer was likely to buy them regardless of whether or +not we recommended them, which is why using `nodeSimilarity` provides +some value over simply looking at what items are co-purchased together +the most. + +== A Better Recommendation + +At this point, we’ve seen what makes a bad recommendation — but what +makes a good one? Let’s look at what coupons our system would suggest if +a customer bought Peanut Butter Cereal (34), Organic Bananas (13176), +and Cauliflower (5618). + +[width="99%",cols="27%,51%,22%",options="header",] +|=== +|BASKET++_++PRODUCT |SIMILAR++_++PRODUCT |SIMILARITY++_++SCORE +|Cauliflower |Organic Pepper Jack Cheese |0.9736842105263158 +|Cauliflower |White Cheddar Snack Crackers Cheddar Bunnies |0.925 +|Peanut Butter Cereal |Nectarines |0.4166666666666667 +|Peanut Butter Cereal |Bread Double Fiber |0.34210526315789475 +|Bag of Organic Bananas |Organic Strawberries |0.22397476340694006 +|Bag of Organic Bananas |Organic Baby Spinach |0.1772679874869656 +|=== + +A weaker model might recommend something like Organic Strawberries, +simply because they frequently appear alongside bananas. But a +graph-based approach looks deeper. It recognizes that the similarity +score for strawberries is driven by a universally popular item — bananas +— which doesn’t tell us much about this specific shopper. + +Instead, the algorithm surfaces Organic Pepper Jack Cheese — a +connection rooted in Cauliflower, an item that’s more distinctive to our +customers’ preferences. In other words, node similarity filters out +noisy, generic associations (like “bananas go with everything”) and +highlights patterns that are more meaningful and personalized.