325 lines
11 KiB
Markdown
325 lines
11 KiB
Markdown
# TODO: choose a title
|
|
|
|
TODO: tl;dr: ... (3 sentences max)
|
|
|
|
TODO: introduction (20 lines max)
|
|
|
|
## Plan
|
|
|
|
TODO: Remove the detailled plan
|
|
|
|
- Start with the end
|
|
- show a pulse
|
|
- explain what is simple / hard
|
|
- The situation before
|
|
- pb with volume (MongoDB / PHP, etc...)
|
|
- securities issues
|
|
- pb with abilities
|
|
- angular complexity
|
|
- refactoring issues
|
|
- deployment issues
|
|
- The choices
|
|
- why clojure?
|
|
- why Haskell?
|
|
- why not full Haskell?
|
|
- why reagent?
|
|
- why Kafka?
|
|
- why Mesos / Marathon?
|
|
- why Druid?
|
|
- why still MongoDB?
|
|
- The firsts weeks
|
|
- first impressions
|
|
- what was harder?
|
|
- what was easier?
|
|
- Once used to clojure
|
|
- how does it feels?
|
|
- was it a mistake?
|
|
- Do we have any doubts?
|
|
- One year later (maintenance and impressions)
|
|
|
|
## The Elephant Graveyard
|
|
|
|
Imagine you can get all twitter data in realtime.
|
|
|
|
Imagine you need to count tweets.
|
|
Imagine you need to filter them by keywords.
|
|
Imagine you have to answer complex questions about all this data in realtime.
|
|
For instance, how many tweets from women, containing the word `clojure` expressing a positive sentiment were submitted last hour.
|
|
Now, Imagine the same question, but you have to deal with a year of data.
|
|
|
|
How would you do it?
|
|
|
|
First you'll need to absorb tweets in realtime.
|
|
The twitter streaming API is here for that.
|
|
However you are limited to 1% of all twitter volume.
|
|
If you want not to be limited, you either need to deal directly with twitter or use GNIP.
|
|
|
|
Next, you'll need to keep only tweet of interest and discard the rest.
|
|
For instance, you'll have to filter them by keyword.
|
|
|
|
Right after that, you have to add extra data for each received tweet.
|
|
For instance, Twitter doesn't provide gender information so you have to guess it somehow.
|
|
The same applies for the sentiment expressed in the Tweet.
|
|
|
|
In the end you'll need to display all these informations in real-time.
|
|
By real-time we mean with very low latency.
|
|
|
|
Under the minute is generally acceptable.
|
|
Our processing pipeline usually process Tweets in only less than half a second of latency.
|
|
|
|
However total latency is due to twitter streaming API (about 2s) or gnip endpoints (about 15s).
|
|
|
|
### The Elephant
|
|
When we got the job, we inherited the prototypes builded by externs.
|
|
back then, everything was done in PHP.
|
|
Yes, in PHP! From retrieving tweets, to aggregates generation and display.
|
|
Let's not even talk about code quality.
|
|
|
|
At the beginning nothing was saved over time, the system could only display data
|
|
since midninght.
|
|
If you only need to count how many tweets were received and you were the
|
|
only client looking at the website, this "architecture" could handle about 200 tweets/s max.
|
|
That was achieved on very large machines and which, knowing how everything was coded, wasn't so bad.
|
|
|
|
But if you needed to add enrichments to achieve complex drilldowns,
|
|
for instance how many men or women,
|
|
how many positive, negative or informative tweets, then the system couldn't handle more than 7 tweets by second.
|
|
|
|
### MongoDB the Destroyer!
|
|
|
|
![MongoDB the destroyer](img/mongodb-the-destroyer.png)
|
|
|
|
Behind the scene data was saved in MongoDB.
|
|
Saved was a big word talking about MongoDB.
|
|
|
|
More seriously, Mongo was a great DB, really easy to start with.
|
|
Really fast and nice to use.
|
|
|
|
Nice until you reach the hard limit.
|
|
At that time it was, Mongo 2.6.
|
|
So there was a **Database Level Lock**.
|
|
|
|
Yes, I repeat: **Database Level Lock**.
|
|
Each time you read or write, nobody could read or write at the same time.
|
|
|
|
And even using very expensive clusters, they couldn't handle these hard limits.
|
|
|
|
As a result, when we asked Mongo to read and write a lot at the same time, we started to witness data loss...
|
|
If you can't write them, let's destroy them.
|
|
Furthermore the code dealing with tweet insertion in MongoDB was really hard to manipulate. No correct error handling. In the end, data loss...
|
|
|
|
There is a lot to say about MongoDB and a lot was already written.
|
|
But the main point is yes.
|
|
MongoDB couldn't be trusted nor used for intensive data manipulation.
|
|
|
|
Now, the situation might have changed.
|
|
But there are better tools for the same job.
|
|
|
|
When we got the job, many customers were already on board
|
|
And many new products were planned.
|
|
|
|
So what we did first is:
|
|
|
|
- create an incremental map reduce system for mongoDB with node.js.
|
|
- use HHVM to somewhat boost PHP performance
|
|
- create a real data API to be called with Angular JS
|
|
- lots of code optimizations
|
|
|
|
In the end our system was able to deal with _far_ more volume than before.
|
|
It could display all the informations we talked about before with a volume of about 120 tweets/min.
|
|
Which was about x17 progress. But as in the optimisation of old nitendo.
|
|
We reached the maximal of power we could from this old legacy system.
|
|
|
|
let's not even speak about our frontend.
|
|
Code was a mess, to deal with it, we had to convert from a single file with 10 000 lines of JS code to an Angular JS application.
|
|
Anyway, we quickly started to experience complexity limits with our Angular JS architecture.
|
|
|
|
It was clear that each new component we created, added more than a linear complexity to all the project.
|
|
It was more about quadratic complexity.
|
|
We started to experience weird bug very hard to reproduce.
|
|
The lack of expliciteness was also a real problem.
|
|
|
|
### 9 months of gestation
|
|
|
|
It was clear from here that nothing could work correctly and everything should be rewritten.
|
|
|
|
The startup had, at that time, only two people to rewrite everything from scratch.
|
|
The chosen language was Clojure for multiple reasons.
|
|
|
|
Something what made this evolution possible was the time taken to explore new technologies.
|
|
About half a day per week was focused toward experimentation of new technologies.
|
|
|
|
For instance, we created a Haskell tweet absorber.
|
|
It was clear that it could handle thousands of tweets per seconds.
|
|
In fact, at that time the system was able to absorb about 6000 tweets/s.
|
|
That means barely the full firhose on a single machine.
|
|
|
|
We thought that is was certainly a good idea to use it instead of another java client.
|
|
|
|
Next we experimented clojure projects intended to create a restful API.
|
|
And the experience was great.
|
|
It was really fast to develop and create new things.
|
|
|
|
Also lot of success stories with clojure we noticed throughout the internet made us confident we could use it
|
|
in production.
|
|
|
|
At that time, Haskell wasn't suitable for production.
|
|
The cabal hell was really problematic.
|
|
We had to use `cabal freeze` often.
|
|
There were other problems.
|
|
It was hard to install, compile and deploy.
|
|
|
|
NOW, Thank to `stack` this is no more the case[^1].
|
|
|
|
[^1]: Just a great thank you to FPComplete and in particular Michael Snoyman!
|
|
|
|
Further more, dealing with realtime processing at that time was all about java ecosystem.
|
|
There was Storm, Kafka, Zookeeper, etc...
|
|
|
|
So using a language which could use all the java libraries seemed very important to us.
|
|
With that in mind we simply had to choose between Scala and Clojure.
|
|
Looking at scala, it was clear that it will be to Java what C++ is to C.
|
|
While Clojure being a descendant from the LISP family, we found that everything was simple, easier to read and understand.
|
|
Clojure community sounded great.
|
|
|
|
Furthermore Storm was first written in Clojure.
|
|
So go go go!
|
|
|
|
It was during summer that most technical choices were made.
|
|
|
|
1. We wanted to deploy using Mesos / Marathon,
|
|
2. Use Kafka and Storm,
|
|
3. Use Druid and ElasticSearch for tweets DB,
|
|
4. Still use MongoDB for resources (users, projects, keywords by projects, etc...).
|
|
5. Use `compojure-api` for the API server
|
|
6. Go on with Haskell to absorb tweets
|
|
7. Use reagent instead of Angular for the frontend
|
|
|
|
Each choice was balanced.
|
|
In the end some of those choices were changed throught practice.
|
|
|
|
For instance, we discarded Storm.
|
|
The power of `core.async` was more than enough efficiently exploit all the juice of our machines,
|
|
Storm added some unnecessary latency and complexity.
|
|
|
|
Today you can see a result here:
|
|
|
|
<div class="wrap" style="height: 630px; width: 100%;">
|
|
<iframe src="https://dev.pulse.vigiglo.be/#/vgteam/TV_Shows/dashboard" style="width:200%; border:solid 2px #DDD; padding: none; margin: 20px 0; height: 1200px; -ms-zoom:0.5; -moz-transform: scale(0.5); -moz-transform-origin: 0 0; -o-transform: scale(0.5); -o-transform-origin: 0 0; -webkit-transform: scale(0.5); -webkit-transform-origin: 0 0"></iframe>
|
|
</div>
|
|
|
|
|
|
## Long live the new flesh
|
|
|
|
![Long Live the new Flesh](img/videodrome.jpg)
|
|
|
|
Difficulties with the new mindset.
|
|
As everything new, there is a period of adaptation.
|
|
Typically the most difficult part was to deal with reversed arrays.
|
|
|
|
In javascript one would write
|
|
|
|
~~~ {.javascript}
|
|
foo["a"]="value-for-a"
|
|
foo["b"]="value-for-b"
|
|
foo["c"]="value-for-c"
|
|
|
|
foreach (i in foo) {v[foo[i]]=i;}
|
|
~~~
|
|
|
|
Or doing things like:
|
|
|
|
~~~ {.javascript}
|
|
var foo = [[1,2,3],[4,5,6]];
|
|
tmp=0;
|
|
foreach (i in foo) {
|
|
foreach (j in foo[i]) {
|
|
tmp += foo[i][j] + 2;
|
|
}
|
|
}
|
|
return tmp;
|
|
~~~
|
|
|
|
Now that I am used to reduce and filters this is like a second nature.
|
|
And the new solution is far better.
|
|
|
|
For example the preceeding example write:
|
|
|
|
~~~ {.clojure}
|
|
(def foo [[1 2 3] [4 5 6]])
|
|
(defn plus2 [x y] (+ x 2))
|
|
(defn sum [l] (reduce + 0 l))
|
|
(sum (map (fn [l] (reduce plus2 0 l)) foo))
|
|
|
|
;; or
|
|
|
|
(->> foo
|
|
(map #(reduce plus2 0 %))
|
|
(sum))
|
|
~~~
|
|
|
|
The resulting code is much better: modular, easier to read and to modify.
|
|
|
|
- Java null pointer exception!
|
|
- Unreadable stacktrace
|
|
|
|
What were the immediate wins!
|
|
|
|
### Deep access
|
|
|
|
For the brave an true there is the lenses Haskell library.
|
|
But for clojurist, the basic access function should be good enough.
|
|
|
|
Let's compare Javascript with Clojure:
|
|
|
|
~~~ {.javascript}
|
|
foo={"db": [{"name":"John Doe","age":30},{"name":"Rich","age":40},{"age":20}]
|
|
// other stuff , ....
|
|
}
|
|
|
|
var val = function() {
|
|
x = foo[db];
|
|
if (x) {
|
|
let y = x[1];
|
|
if (y) {
|
|
return y.age;
|
|
} else return nil;
|
|
} else return nil;
|
|
}();
|
|
~~~
|
|
|
|
Yes, you have to manually check at each level if the value is null
|
|
or not. Without this manual check, your code is going to crash at runtime!
|
|
|
|
Now lets compare the situation with clojure:
|
|
|
|
~~~ {.clojure}
|
|
(-> foo :db second :age)
|
|
~~~
|
|
|
|
Yes, that's all. The default value in case of problem is `nil`.
|
|
|
|
### Merges
|
|
|
|
**Seriously!!!!!**
|
|
|
|
~~~ {.clojure}
|
|
(into map1 map2)
|
|
~~~
|
|
|
|
I don't even want to compare to javascript as it would be ridiculous.
|
|
Mainly, you can't[^2], or you need jQuery and its ugly.
|
|
|
|
[^2]: <http://stackoverflow.com/questions/171251/how-can-i-merge-properties-of-two-javascript-objects-dynamically>
|
|
|
|
### Syntax
|
|
|
|
Learning Clojure syntax takes about 3 minutes, thanks to homoiconicity
|
|
It is clean, no _fucking_ comma, semicolons, etc...
|
|
|
|
- Arrays: `[a b c]` in javascript `[a,b,c]` (why the commas?)
|
|
- Hash Map (Associative arrays): `{:key1 value1 :key2 value2}` in javascript you need to define an Object and keys are generally strings: `{"key1":value1, "key2":value2}`. Multiline object declaration always have bad number of commas.
|
|
- Set: `#{:a :b :c}` in javascript sets doesn't even exists you have to simulate them with Objects: `{"a":true, "b":true, "c":true}`
|
|
- inline function declaration; compare `#(* % 2)` in clojure with `function(x){return x * 2;}` in javascript
|
|
|
|
TODO: faudrait m'aider à en rajouter quelques tonnes. Avec de meilleurs exemples.
|