-
-
- Segments +
- Segments -
- Data -
- Querying
-
+
- Querying -
- Components
-
-
- Druid Components +
- Druid Components -
- Considerations & Tools @@ -155,7 +146,7 @@
- Manually coded HyperLogLog in js
Return of Experience
-
+
Return of Experience
- Ingestion still in PHP (600 msg/s max) @@ -294,88 +285,24 @@
- Dimension columns: strings (used to filter or to group)
- Metric columns: used for aggregations (count, sum, mean, etc…)
Roll-up
-Example
-
-timestamp page ... added deleted -2011-01-01T00:01:35Z Justin Bieber 10 65 -2011-01-01T00:03:63Z Justin Bieber 15 62 -2011-01-01T01:04:51Z Justin Bieber 32 45 -2011-01-01T01:01:00Z Ke$ha 17 87 -2011-01-01T01:02:00Z Ke$ha 43 99 -2011-01-01T02:03:00Z Ke$ha 12 53
-timestamp page ... nb added deleted -2011-01-01T00:00:00Z Justin Bieber 2 25 127 -2011-01-01T01:00:00Z Justin Bieber 1 32 45 -2011-01-01T01:00:00Z Ke$ha 2 60 186 -2011-01-01T02:00:00Z Ke$ha 1 12 53as SQL
-
-GROUP BY timestamp, page, nb, added, deleted - :: nb = COUNT(1) - , added = SUM(added) - , deleted = SUM(deleted)In practice can dramatically reduce the size (up to x100)
-Sharding
-Segments
-
-sampleData_2011-01-01T01:00:00:00Z_2011-01-01T02:00:00:00Z_v1_0
-2011-01-01T01:00:00Z Justin Bieber 1 20 45 -2011-01-01T01:00:00Z Ke$ha 1 30 106
-sampleData_2011-01-01T01:00:00:00Z_2011-01-01T02:00:00:00Z_v1_0
-2011-01-01T01:00:00Z Justin Bieber 1 12 45 -2011-01-01T01:00:00Z Ke$ha 2 30 80Core Data Structure
-
-
-
-
- dictionary -
- a bitmap for each value -
- a list of the columns values encoded using the dictionary -
Dictionary
-
-{ "Justin Bieber": 0 -, "Ke$ha": 1 -}Columnn Data
-
-[ 0 -, 0 -, 1 -, 1 -]Bitmaps
-one for each value of the column
-
-value="Justin Bieber": [1,1,0,0] -value="Ke$ha": [0,0,1,1]Data
-Indexing Data
+Indexing
- Immutable snapshots of data
- data structure highly optimized for analytic queries
- Each column is stored separately
- Indexes data on a per shard (segment) level
Loading data
+Loading
- Real-Time
- Batch
Querying the data
+Querying
- JSON over HTTP
- Single Table Operations, no joins.
Columnar Storage
-Index
--
-
- Values are dictionary encoded -
-{"USA" 1, "Canada" 2, "Mexico" 3, ...}-
-
- Bitmap for every dimension value (used by filters) -
-"USA" -> [0 1 0 0 1 1 0 0 0]-
-
- Column values (used by aggergation queries) -
-[2,1,3,15,1,1,2,8,7]Data Segments
+Segments
- Per time interval
-
@@ -392,6 +319,61 @@ value="Ke$ha": [0,0,1,1]
- Read-write concurrency
Roll-up
+Example
+
+timestamp page ... added deleted +2011-01-01T00:01:35Z Cthulhu 10 65 +2011-01-01T00:03:63Z Cthulhu 15 62 +2011-01-01T01:04:51Z Cthulhu 32 45 +2011-01-01T01:01:00Z Azatoth 17 87 +2011-01-01T01:02:00Z Azatoth 43 99 +2011-01-01T02:03:00Z Azatoth 12 53
+timestamp page ... nb added deleted +2011-01-01T00:00:00Z Cthulhu 2 25 127 +2011-01-01T01:00:00Z Cthulhu 1 32 45 +2011-01-01T01:00:00Z Azatoth 2 60 186 +2011-01-01T02:00:00Z Azatoth 1 12 53as SQL
+
+GROUP BY timestamp, page, nb, added, deleted + :: nb = COUNT(1) + , added = SUM(added) + , deleted = SUM(deleted)In practice can dramatically reduce the size (up to x100)
+Segments
+Sharding
+
+sampleData_2011-01-01T01:00:00:00Z_2011-01-01T02:00:00:00Z_v1_0
+timestamp page ... nb added deleted +2011-01-01T01:00:00Z Cthulhu 1 20 45 +2011-01-01T01:00:00Z Azatoth 1 30 106
+sampleData_2011-01-01T01:00:00:00Z_2011-01-01T02:00:00:00Z_v1_0
+timestamp page ... nb added deleted +2011-01-01T01:00:00Z Cthulhu 1 12 45 +2011-01-01T01:00:00Z Azatoth 2 30 80Core Data Structure
+
+ -
+
- dictionary +
- a bitmap for each value +
- a list of the columns values encoded using the dictionary +
Example
+
+dictionary: { "Cthulhu": 0 + , "Azatoth": 1 } + +column data: [0, 0, 1, 1] + +bitmaps (one for each value of the column): +value="Cthulhu": [1,1,0,0] +value="Azatoth": [0,0,1,1]Example (multiple matches)
+dictionary: { "Cthulhu": 0 + , "Azatoth": 1 } + +column data: [0, [0,1], 1, 1] + +bitmaps (one for each value of the column): +value="Cthulhu": [1,1,0,0] +value="Azatoth": [0,1,1,1]Real-time ingestion
- Via Real-Time Node and Firehose @@ -411,13 +393,11 @@ value="Ke$ha": [0,0,1,1]
- File based (HDFS, S3, …)
Real-time Ingestion
-
-Task 1: [ Interval ][ Window ] -Task 2: [ ] ----------------------------------------> - timeMinimum indexing slots =
-
- Data Sources × Partitions × Replicas × 2Querying
+
+Task 1: [ Interval ][ Window ] +Task 2: [ ] +-----------------------------------------------------> + timeQuerying
Query types
- Group by: group by multiple dimensions @@ -427,26 +407,27 @@ Task 2: [ ]
- Time Boundary: Find available data timeframe
- Metadata queries
Tip
--
-
- Prefer
topNovergroupBy
- - Prefer
timeseriesovertopN
- - Use limits (and priorities) -
Query Spec
--
-
- Data source -
- Dimensions -
- Interval -
- Filters -
- Aggergations -
- Post Aggregations -
- Granularity -
- Context (query configuration) -
- Limit -
Example(s)
-TODO
+
+{"queryType": "groupBy", + "dataSource": "druidtest", + "granularity": "all", + "dimensions": [], + "aggregations": [ + {"type": "count", "name": "rows"}, + {"type": "longSum", "name": "imps", "fieldName": "impressions"}, + {"type": "doubleSum", "name": "wp", "fieldName": "wp"} + ], + "intervals": ["2010-01-01T00:00/2020-01-01T00"]}Result
+[ { + "version" : "v1", + "timestamp" : "2010-01-01T00:00:00.000Z", + "event" : { + "imps" : 5, + "wp" : 15000.0, + "rows" : 5 + } +} ]Caching
- Historical node level @@ -458,15 +439,10 @@ Task 2: [ ]
- By segment and query
groupByis disabled on purpose!
- - By default - local caching +
- By default: local caching
Load Rules
--
-
- Can be defined -
- What can be set -
Components
-Druid Components
+Druid Components
+Druid
- Real-time Nodes
- Historical Nodes @@ -477,56 +453,60 @@ Task 2: [ ]
- Overlord
- Middle Manager
- - Deep Storage -
Metadata Storage
+
Also
+-
+
- Deep Storage (S3, HDFS, …) +
- Metadata Storage (SQL)
- Load Balancer -
Cache
+- Cache
Coordinator
-Manage Segments
-Real-time Nodes
-
-
- Pulling data in real-time -
- Indexing it -
__
