yogsototh/clj-kafka
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
136 commits 1 branch 4 tags 157 KiB
Commit graph

136 commits

This branch
This branch
All branches
Author SHA1 Message Date
Paul Lam
7ec63751f1 example hostname 2013-06-06 21:58:59 +01:00
Paul Lam
276903b4ee update consumer 2013-06-06 17:22:54 +01:00
Paul Lam
398f002e74 update readme and project.clj for 0.8 2013-06-06 17:09:27 +01:00
Paul Ingles
e97faeb733 releasing 0.0.7 with pipe size, thanks @DAddYE 2013-05-15 09:32:20 +01:00
Paul Ingles
1485d4845d Merge pull request #7 from DAddYE/patch-1 
Prevent Out of Memory with large streams
 2013-05-15 01:31:44 -07:00
Paul Ingles
b7abaa16ee add travis-ci build badge to README 2013-05-15 09:27:57 +01:00
Paul Ingles
242e57d5b3 add travis-ci yaml 2013-05-15 09:27:25 +01:00
Davide D'Agostino
2383459067 Prevent Out of Memory with large streams 
This version caps the queue and prevent OOM and obviously uses much less memory.

My app that use some like:

  (doseq [group (partition resolution (kafka/messages consumer topic))

Where resolution is a value of ~60000

Without this patch the memory usage was constantly around ~4gb

After dealing with OOMs and inspecting all vm snapshots I come that the problem was in the pipe, I checked your code then the linked article and was confirmed (in one comment) with the same solution.
 2013-05-15 01:17:31 -06:00
Paul Ingles
231a739e66 release 0.0.6-0.7 with producer type hints to avoid reflection 2013-05-03 10:09:26 +01:00
Paul Ingles
7485a60472 change type hint in message 2013-05-03 10:08:47 +01:00
Paul Bergeron
787897402b Add type annotations for producer 2013-04-08 16:21:27 -07:00
Paul Ingles
3ce92430bb release 0.0.5-0.7 2012-12-18 10:24:26 +00:00
Paul Ingles
914a0e2389 add ToMessage protocol to help convert objects to kafka.message.Message objects 2012-10-04 09:46:45 +01:00
Paul Ingles
ab2cefe1df lein 2 compatible project.clj 2012-10-04 09:46:32 +01:00
Paul Ingles
6dfb60fcb6 update README for a 0.0.5-0.7-SNAPSHOT release 2012-05-18 12:38:44 +01:00
Paul Ingles
1fca1666ec use threading macro to tidy consumer.zk/messages 2012-05-18 12:28:58 +01:00
Paul Ingles
f9574228db little refactoring in consumer.zk/messages 2012-05-18 12:28:03 +01:00
Paul Ingles
8fc0df82e3 Change handling of multiple topics- assoc a topic key that names where the message came from... not so keen on having parser/handler functions being injected 2012-05-18 12:26:07 +01:00
Paul Ingles
eb50e39b58 Revert "Update the example + allow the zookeeper consumer to specify additional functions that can be used to interpret the messages before they're aggregated into the sequence" 
This reverts commit cdc32becff.
 2012-05-18 12:24:09 +01:00
Paul Ingles
cdc32becff Update the example + allow the zookeeper consumer to specify additional functions that can be used to interpret the messages before they're aggregated into the sequence 2012-05-18 12:22:36 +01:00
Paul Ingles
55eefc1314 refactor consumer.zk/messages a little 2012-05-18 11:20:19 +01:00
Paul Ingles
7a3d348089 allow consumer.zk/messages to be passed multiple topics and aggregate into single seq 2012-05-18 11:02:22 +01:00
Paul Ingles
f8c72e9e75 Add topics to the consumer.zk ns, provides a list of all registered topics 2012-03-14 16:07:10 +00:00
Paul Ingles
9840461b7e bump version- fix for payload data extraction 2012-03-14 12:11:39 +00:00
Paul Ingles
cc139ef82b Merge branch 'master' of github.com:pingles/clj-kafka 2012-03-14 12:11:26 +00:00
Paul Ingles
df01625a56 fix a bug with extracting the byte array of the payload: didn't honour the length of the payload, just extracted the underlying array. 2012-03-14 12:10:39 +00:00
Paul Ingles
d4963d0795 Merge pull request #1 from strongh/patch-1 
Update README.md
 2012-03-10 01:13:15 -08:00
Homer Strong
9606b5c8ec Update README.md 2012-03-09 18:15:15 -08:00
Paul Ingles
49e0cb5f7f Add a note about Kafka binaries 2012-01-26 11:04:00 +00:00
Paul Ingles
9d6681cb48 add link to the autodoc API docs 2012-01-26 11:00:38 +00:00
Paul Ingles
0fb2673049 updated autodoc 2012-01-26 10:59:48 +00:00
Paul Ingles
d6d9d52402 as-bytes doesnt exist 2012-01-26 10:59:00 +00:00
Paul Ingles
efee73f42d Update README; pull with-resource into core; add producer 2012-01-26 10:56:32 +00:00
Paul Ingles
9ddb351353 Add namespaces for connecting with the ZookeeperConnector and SimpleConsumer consumers; updated version to 0.0.2-0.7-SNAPSHOT 2012-01-26 10:20:44 +00:00
Paul Ingles
89032ffc4b clojure syntax readme 2012-01-22 23:24:41 +00:00
Paul Ingles
df8cb2f747 first commit 2012-01-22 23:01:36 +00:00