Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
Takahiro Inoue, profile picture
Uploaded byTakahiro Inoue
PDF, PPTX4,918 views

MongoDB: Intro & Application for Big Data

The document discusses NoSQL databases and MongoDB in particular. It provides examples of MongoDB documents with different data types like objects, arrays, etc. It also shows how data is stored across multiple shard servers and accessed via mongos query routers. Configuration servers store sharded cluster metadata. Log data can be stored in MongoDB but only aggregated results data may be needed. HDFS is better for temporary raw log storage.

Technology◦Entertainment & Humor◦
Download as PDF, PPTX
http://blog.nahurst.com/visual-guide-to-nosql-systems
{ "_id" : ObjectId("4dcd3ebc9278000000005158"), "timestamp" : ISODate("2011-05-13T14:22:46.777Z"), "binary" : BinData(0,""), "string" : "abc", "number" : 3, "subobj" : {"subA": 1, "subB": { "subsubC": 2 }}, "array" : [1, 2, 3], "dbref" : [_id1, _id2, _id3] }
{ "_id" : ObjectId("4dcd3ebc9278000000005158"), "nickname" : "doryokujin" },{ "_id" : ObjectId("4dcd3ebc9278000000005159"), "firstname" : "Takahiro", "lastname" : "Inoue", "mail" : "mr.stoicman@gmail.com", "twitter" : "@doryokujin" },...
{ "_id" : ObjectId("4dcd3ebc9278000000005158"), "timestamp" : ISODate("2011-05-13T14:22:46.777Z"), "binary" : BinData(0,""), "string" : "abc", "number" : 3, "subobj" : {"subA": 1, "subB": 2 }, "array" : [1, 2, 3], padding }
{ "_id" : ObjectId("4dcd3ebc9278000000005158"), "timestamp" : ISODate("2011-05-13T14:22:46.777Z"), "binary" : BinData(0,""), "string" : "def", "number" : 4, "subobj" : {"subA": 1, "subB": 2 }, "array" : [1, 2, 3, 4, 5, 6], "newkey" : "In-place" }
{ "_id" : ObjectId("4dcd3ebc9278000000005158"), "timestamp" : ISODate("2011-05-13T14:22:46.777Z"), "binary" : BinData(0,""), "string" : "abc", "number" : 3, "subobj" : {"subA": 1, "subB": 2 }, "array" : [1, 2, 3], }
Cluster Shard Servers (Data) config Servers (Shard Configration) shard1 shard2 shard3 [ a, f ) [ k, n) [ o, t ) Chunk [ f, k ) [ n, o ) [ t, } ) mongos Servers (Routers)
Shard 1 ( mongos primary primary ) Shard cinfig Shard mongos
Every Server: Large HDD(500GB ) Large Memory(16GB ) Slave Delay in Master Data on Preparation for AmazonS3 User Error Master Data on S3 Non Sharding, Replica Set
From Text Logs: (Large) From mySQL: (Small) From Other NoSQL: (Middle) Temporary Raw data storage is HDFS, not MongoDB We only need result data to discover new features
Reduction First Second Aggregation Aggregation
ScientificPython
Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector Service
Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector Service Sadayuki Furuhashi Treasure Data, Inc. @frsyuki
“2011-04-01 host1 myapp: cmessage size=12MB user=me” 2011-04-01 myapp.message { “on_host”: ”host1”, 2011-04-01 myapp.message { ”combined”: true, “on_host”: ”host1”, ”combined”: true, “size”: 12000000, “size”: 12000000, “user”: “me” “user”: “me” } }
log log log log
aggregate aggregate aggregate aggregate log log log log key1 key2 key3 shuffle aggregate aggregate aggregate aggregate
MongoDB: Intro & Application for Big Data
MongoDB: Intro & Application for Big Data