You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Plan: Migrate graphify from networkx to Apache Arrow
1. Motivation
The current codebase stores its entire knowledge graph in a single graph.json
file (NetworkX node_link_data JSON). This works but has known limitations:
No graph-native queries — every read loads the full JSON into a NetworkX Graph in memory, then walks it with BFS/DFS in Python. There is no MATCH, WHERE, or OPTIONAL matching.
No incremental reads — you cannot ask "which nodes have community 5"
without deserializing the entire graph.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Plan: Migrate graphify from networkx to Apache Arrow
1. Motivation
The current codebase stores its entire knowledge graph in a single
graph.jsonfile (NetworkX
node_link_dataJSON). This works but has known limitations:No graph-native queries — every read loads the full JSON into a NetworkX
Graphin memory, then walks it with BFS/DFS in Python. There is noMATCH,WHERE, orOPTIONALmatching.No incremental reads — you cannot ask "which nodes have community 5"
without deserializing the entire graph.
Similar issues here.
Similar discussion about scalability, large repos: Are there reliable benchmarks showing Graphify improves coding agent performance on large repos? #1328, Establish a Knowledge Base #1340
Full plan: https://gist.github.com/adsharma/c0fd0c9163f1fc60669cacfe955e152f
All reactions