You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
def __call__(self, past_key_values, attn_score_cache):
self._update_hh_score(attn_score_cache)
if past_key_values is None:
return None
seq_len = past_key_values[0].size(self.k_seq_dim)
if seq_len <= self.cache_size:
return past_key_values
, I find the program always falls in the line seq_len < self.cache_size and directly return without running the following h2o code. However, the cache_size is set to 2048 as default. Isn't that a little long?
Or we can consider that h2o will not work in the short context?
The text was updated successfully, but these errors were encountered:
def __call__(self, past_key_values, attn_score_cache):
self._update_hh_score(attn_score_cache)
if past_key_values is None:
return None
seq_len = past_key_values[0].size(self.k_seq_dim)
if seq_len <= self.cache_size:
return past_key_values
, I find the program always falls in the line seq_len < self.cache_size and directly return without running the following h2o code. However, the cache_size is set to 2048 as default. Isn't that a little long?
Or we can consider that h2o will not work in the short context?
When I run
bash scripts/streaming/eval.sh h2o
, I find h2o is not working and acting exactly the same as the full. When I check the codeH2O/h2o_hf/utils_real_drop/modify_llama.py
Line 68 in 8bb3fe1
, I find the program always falls in the line
seq_len < self.cache_size
and directly return without running the following h2o code. However, thecache_size
is set to 2048 as default. Isn't that a little long?Or we can consider that h2o will not work in the short context?
The text was updated successfully, but these errors were encountered: