yogsototh/clj-kafka
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
67 commits 1 branch 4 tags 157 KiB
Commit graph

37 commits

Author SHA1 Message Date
Paul Ingles
be2cd0329f add queue-capacity option for zookeeper messages: can be used to limit LinkedBlockingQueue queue size with large messages 2013-06-07 22:11:42 +01:00
Paul Ingles
84c813ae48 add message piping back; to-clojure conversion to KafkaMessage added by default 2013-06-07 22:03:11 +01:00
Paul Ingles
ec1fb0671c rename MessagePayload to MessageValue 2013-06-07 21:10:37 +01:00
Paul Ingles
6437dcd18d delete unused to-clojure definition 2013-06-07 21:09:02 +01:00
Paul Lam
cd525fa6bd fix hanging param from merge 2013-06-07 16:36:42 +01:00
Paul Lam
20683d7e18 type hint consumer.simple 2013-06-07 16:35:48 +01:00
Paul Lam
c01ffa4a96 added brokers in producer to retrieve list of brokers from zookeeper, 
cool
 2013-06-07 16:35:48 +01:00
Paul Ingles
590f303a49 fix reflection warning 2013-06-07 16:30:16 +01:00
Paul Ingles
391f790cde remove reflection warning in producer code 2013-06-07 16:27:08 +01:00
Paul Ingles
cfa6b0854f remove some reflection warnings from the zookeeper consumer code 2013-06-07 16:24:33 +01:00
Paul Ingles
9945751c2f remove example; will replace in README later 2013-06-07 16:20:20 +01:00
Paul Ingles
989965a2d1 add example for simple consumer 2013-06-07 14:57:41 +01:00
Paul Ingles
f84c7dcc83 remove unused fns 2013-06-07 14:20:24 +01:00
Paul Ingles
9f23aec1df fix cleanup after test run; add MessagePayload protocol and second test 2013-06-07 14:17:38 +01:00
Paul Ingles
fa0de3cae0 update code to pass test; consumes messages but clj-kafka.consumer.zk/messages can't map the contents for us... 2013-06-07 13:55:17 +01:00
Paul Ingles
f4a1ebe88a more work on updating for 0.8 api; clj-kafka.test.consumer is WIP and hangs currently when creating the messages sequence 2013-06-07 13:03:55 +01:00
Paul Ingles
2492aee358 Revert "work on updating the api to work against 0.8" 
This reverts commit 5286bbfbb6.
 2013-06-07 10:48:08 +01:00
Paul Ingles
5286bbfbb6 work on updating the api to work against 0.8 2013-06-07 10:23:13 +01:00
Paul Ingles
f661446647 start work on adding some tests; update message tests for 0.8 api 2013-06-07 08:33:51 +01:00
Paul Lam
276903b4ee update consumer 2013-06-06 17:22:54 +01:00
Davide D'Agostino
2383459067 Prevent Out of Memory with large streams 
This version caps the queue and prevent OOM and obviously uses much less memory.

My app that use some like:

  (doseq [group (partition resolution (kafka/messages consumer topic))

Where resolution is a value of ~60000

Without this patch the memory usage was constantly around ~4gb

After dealing with OOMs and inspecting all vm snapshots I come that the problem was in the pipe, I checked your code then the linked article and was confirmed (in one comment) with the same solution.
 2013-05-15 01:17:31 -06:00
Paul Ingles
7485a60472 change type hint in message 2013-05-03 10:08:47 +01:00
Paul Bergeron
787897402b Add type annotations for producer 2013-04-08 16:21:27 -07:00
Paul Ingles
914a0e2389 add ToMessage protocol to help convert objects to kafka.message.Message objects 2012-10-04 09:46:45 +01:00
Paul Ingles
1fca1666ec use threading macro to tidy consumer.zk/messages 2012-05-18 12:28:58 +01:00
Paul Ingles
f9574228db little refactoring in consumer.zk/messages 2012-05-18 12:28:03 +01:00
Paul Ingles
8fc0df82e3 Change handling of multiple topics- assoc a topic key that names where the message came from... not so keen on having parser/handler functions being injected 2012-05-18 12:26:07 +01:00
Paul Ingles
eb50e39b58 Revert "Update the example + allow the zookeeper consumer to specify additional functions that can be used to interpret the messages before they're aggregated into the sequence" 
This reverts commit cdc32becff.
 2012-05-18 12:24:09 +01:00
Paul Ingles
cdc32becff Update the example + allow the zookeeper consumer to specify additional functions that can be used to interpret the messages before they're aggregated into the sequence 2012-05-18 12:22:36 +01:00
Paul Ingles
55eefc1314 refactor consumer.zk/messages a little 2012-05-18 11:20:19 +01:00
Paul Ingles
7a3d348089 allow consumer.zk/messages to be passed multiple topics and aggregate into single seq 2012-05-18 11:02:22 +01:00
Paul Ingles
f8c72e9e75 Add topics to the consumer.zk ns, provides a list of all registered topics 2012-03-14 16:07:10 +00:00
Paul Ingles
df01625a56 fix a bug with extracting the byte array of the payload: didn't honour the length of the payload, just extracted the underlying array. 2012-03-14 12:10:39 +00:00
Paul Ingles
d6d9d52402 as-bytes doesnt exist 2012-01-26 10:59:00 +00:00
Paul Ingles
efee73f42d Update README; pull with-resource into core; add producer 2012-01-26 10:56:32 +00:00
Paul Ingles
9ddb351353 Add namespaces for connecting with the ZookeeperConnector and SimpleConsumer consumers; updated version to 0.0.2-0.7-SNAPSHOT 2012-01-26 10:20:44 +00:00
Paul Ingles
df8cb2f747 first commit 2012-01-22 23:01:36 +00:00