@ajorgensen How much does the following one-line fix help?

diff --git a/heron/common/src/cpp/network/mempool.h b/heron/common/src/cpp/network/mempool.h
index a8467df..875e0ac 100644
--- a/heron/common/src/cpp/network/mempool.h
+++ b/heron/common/src/cpp/network/mempool.h
@@ -75,6 +75,7 @@ class MemPool {
     }
     B* t = pool.back();
     pool.pop_back();
+    pool.shrink_to_fit();
     return static_cast<M*>(t);
   }

I still can't reproduce it locally with AckingTopology.

I can try tomorrow. Ill also try to work out a reproducible test case for you if I can.

Just in case, make sure you didn't change the default config for tuple cache:

# The frequency in ms to drain the tuple cache in stream manager
heron.streammgr.cache.drain.frequency.ms: 10

# The sized based threshold in MB for draining the tuple cache
heron.streammgr.cache.drain.size.mb: 100

No we have both of those still at the default.

@congwang what would happen if the spout could produce tuples faster than the bolt could consume them and the maxSpoutPending value was high? Would the spout know to stop calling nextTuple or would the buffer in the stream manager continue to increase until it ran out of memory?

The stream manager will send back pressure to the spout in this case.

@congwang were you able to reproduce the issue with the topology I sent you?

@ajorgensen I am still trying figure out how to build your topology

Oh ok. You should be able to create a simple pom file and build it with maven. Let me see if i can put one together for you.

@congwang sorry about that. I've emailed you the same project but with a working pom.xml file now. Let me know if you have any trouble building the topology

I guess we need the following fix:

diff --git a/heron/common/src/cpp/network/connection.cpp b/heron/common/src/cpp/network/connection.cpp
index c03ea8d..90cfbf3 100644
--- a/heron/common/src/cpp/network/connection.cpp
+++ b/heron/common/src/cpp/network/connection.cpp
@@ -240,6 +240,8 @@ void Connection::handleDataWritten() {
 
 sp_int32 Connection::readFromEndPoint(sp_int32 fd) {
   sp_int32 bytesRead = 0;
+  if (mUnderBackPressure)
+    return 0;
   while (1) {
     sp_int32 read_status = mIncomingPacket->Read(fd);
     if (read_status == 0) {

@ajorgensen I am trying it, definitely not sure yet, when back pressure happens, spout should already stop sending data, only the pending data is still being transmitted. At least it could help us to rule out the case.

@congwang has informed @kramasamy that this is not a issue that could be resolved soon. Delay this to 0.15.0.