yogsototh/clj-kafka
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
70 commits 1 branch 4 tags 157 KiB
Commit graph

70 commits

This branch
This branch
All branches
Author SHA1 Message Date
Paul Ingles
5ba3d9b67b delete MessageValue protocol- clients should know whether they're providing a byte array or not and they can implement their own protocols. 2013-06-09 08:17:48 +01:00
Paul Ingles
24f182c824 change producer send-message to receive a KeyedMessage rather than topic/value; rename keyed-message to just message and have pulled message-value out of send-message: clients should know whether they're providing a byte array or not. 2013-06-09 08:15:03 +01:00
Paul Ingles
b373042058 delete ToClojure extension for ByteBuffer: didn't make any sense given it just extracts the bytes 2013-06-09 08:01:47 +01:00
Paul Ingles
f75c4b339a change log4j to append to ./logs/kafka.log; tidy the zk consumer test 2013-06-09 07:51:34 +01:00
Paul Ingles
257c25293d remove println; move test broker config into a Var 2013-06-09 07:43:24 +01:00
Paul Ingles
76ede151a4 pull paren up 2013-06-08 11:52:07 +01:00
Paul Ingles
1e014986c5 refactor test.utils/with-test-broker to receive config map specifying ports + topic name 2013-06-08 11:49:52 +01:00
Paul Ingles
be2cd0329f add queue-capacity option for zookeeper messages: can be used to limit LinkedBlockingQueue queue size with large messages 2013-06-07 22:11:42 +01:00
Paul Ingles
84c813ae48 add message piping back; to-clojure conversion to KafkaMessage added by default 2013-06-07 22:03:11 +01:00
Paul Ingles
57a9f17dfa rename with-broker to with-test-broker 2013-06-07 21:55:11 +01:00
Paul Ingles
ec1fb0671c rename MessagePayload to MessageValue 2013-06-07 21:10:37 +01:00
Paul Ingles
6437dcd18d delete unused to-clojure definition 2013-06-07 21:09:02 +01:00
Paul Ingles
8643566f5c add log4j properties to reduce logging in dev/test; delete tmp directory before test run too 2013-06-07 20:58:23 +01:00
Paul Lam
cd525fa6bd fix hanging param from merge 2013-06-07 16:36:42 +01:00
Paul Lam
20683d7e18 type hint consumer.simple 2013-06-07 16:35:48 +01:00
Paul Lam
c01ffa4a96 added brokers in producer to retrieve list of brokers from zookeeper, 
cool
 2013-06-07 16:35:48 +01:00
Paul Ingles
590f303a49 fix reflection warning 2013-06-07 16:30:16 +01:00
Paul Ingles
391f790cde remove reflection warning in producer code 2013-06-07 16:27:08 +01:00
Paul Ingles
cfa6b0854f remove some reflection warnings from the zookeeper consumer code 2013-06-07 16:24:33 +01:00
Paul Ingles
9945751c2f remove example; will replace in README later 2013-06-07 16:20:20 +01:00
Paul Ingles
989965a2d1 add example for simple consumer 2013-06-07 14:57:41 +01:00
Paul Ingles
f84c7dcc83 remove unused fns 2013-06-07 14:20:24 +01:00
Paul Ingles
c44ddf8dba rename test 2013-06-07 14:19:41 +01:00
Paul Ingles
4c10bf6c48 both tests are the same 2013-06-07 14:18:46 +01:00
Paul Ingles
9f23aec1df fix cleanup after test run; add MessagePayload protocol and second test 2013-06-07 14:17:38 +01:00
Paul Ingles
fa0de3cae0 update code to pass test; consumes messages but clj-kafka.consumer.zk/messages can't map the contents for us... 2013-06-07 13:55:17 +01:00
Paul Ingles
f4a1ebe88a more work on updating for 0.8 api; clj-kafka.test.consumer is WIP and hangs currently when creating the messages sequence 2013-06-07 13:03:55 +01:00
Paul Ingles
2492aee358 Revert "work on updating the api to work against 0.8" 
This reverts commit 5286bbfbb6.
 2013-06-07 10:48:08 +01:00
Paul Ingles
5286bbfbb6 work on updating the api to work against 0.8 2013-06-07 10:23:13 +01:00
Paul Ingles
f3d01cbfba remove println 2013-06-07 09:43:14 +01:00
Paul Ingles
72cec16cd6 move with-broker testing stuff into test.utils ns 2013-06-07 09:42:04 +01:00
Paul Ingles
a89a454d3b create embedded zookeeper server 2013-06-07 09:22:55 +01:00
Paul Ingles
79544297fe clean the kafka log directory before using the broker in tests 2013-06-07 09:01:38 +01:00
Paul Ingles
72f8daed9e add system-time to implement Time interface for testing 2013-06-07 08:53:01 +01:00
Paul Ingles
f661446647 start work on adding some tests; update message tests for 0.8 api 2013-06-07 08:33:51 +01:00
Paul Lam
276903b4ee update consumer 2013-06-06 17:22:54 +01:00
Paul Lam
398f002e74 update readme and project.clj for 0.8 2013-06-06 17:09:27 +01:00
Paul Ingles
e97faeb733 releasing 0.0.7 with pipe size, thanks @DAddYE 2013-05-15 09:32:20 +01:00
Paul Ingles
1485d4845d Merge pull request #7 from DAddYE/patch-1 
Prevent Out of Memory with large streams
 2013-05-15 01:31:44 -07:00
Paul Ingles
b7abaa16ee add travis-ci build badge to README 2013-05-15 09:27:57 +01:00
Paul Ingles
242e57d5b3 add travis-ci yaml 2013-05-15 09:27:25 +01:00
Davide D'Agostino
2383459067 Prevent Out of Memory with large streams 
This version caps the queue and prevent OOM and obviously uses much less memory.

My app that use some like:

  (doseq [group (partition resolution (kafka/messages consumer topic))

Where resolution is a value of ~60000

Without this patch the memory usage was constantly around ~4gb

After dealing with OOMs and inspecting all vm snapshots I come that the problem was in the pipe, I checked your code then the linked article and was confirmed (in one comment) with the same solution.
 2013-05-15 01:17:31 -06:00
Paul Ingles
231a739e66 release 0.0.6-0.7 with producer type hints to avoid reflection 2013-05-03 10:09:26 +01:00
Paul Ingles
7485a60472 change type hint in message 2013-05-03 10:08:47 +01:00
Paul Bergeron
787897402b Add type annotations for producer 2013-04-08 16:21:27 -07:00
Paul Ingles
3ce92430bb release 0.0.5-0.7 2012-12-18 10:24:26 +00:00
Paul Ingles
914a0e2389 add ToMessage protocol to help convert objects to kafka.message.Message objects 2012-10-04 09:46:45 +01:00
Paul Ingles
ab2cefe1df lein 2 compatible project.clj 2012-10-04 09:46:32 +01:00
Paul Ingles
6dfb60fcb6 update README for a 0.0.5-0.7-SNAPSHOT release 2012-05-18 12:38:44 +01:00
Paul Ingles
1fca1666ec use threading macro to tidy consumer.zk/messages 2012-05-18 12:28:58 +01:00