Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an efficient chunking generator for arrays. #29

Open
ndobromirov opened this issue Apr 11, 2019 · 1 comment
Open

Add an efficient chunking generator for arrays. #29

ndobromirov opened this issue Apr 11, 2019 · 1 comment

Comments

@ndobromirov
Copy link

ndobromirov commented Apr 11, 2019

Writing command line apps it's very common to work with streams of data and to group them to be processed in chunks.

The native PHP's array_chunk in that case is not an efficient function, as it needs the whole input array present and it is generating the full result array in chunks as well. This is at that moment more than doubling the memory overhead.

My proposal is to add the following method to ArrayUtils naming can be changed as needed.
This will allow stream data processing and data grouping without huge memory problems.

/**
   * Generate chunks of a given size from an iterable list.
   *
   * @param mixed $iterator
   *   Items to iterate (array, generator, iterator).
   * @param int $size
   *   Chunk size to group items.
   *
   * @return \generator
   *   Generated items are arrays with up-to $size elements each.
   */
  public static function chunksGenerator($iterator, $size = 1000) {
    $count = 0;
    $buffer = [];

    foreach ($iterator as $item) {
      $buffer[] = $item;
      ++$count;

      if (!($count % $size)) {
        yield $buffer;

        $buffer = [];
        $count = 0;
      }
    }

    if ($buffer) {
      yield $buffer;
    }
  }
@greg-1-anderson
Copy link
Member

Your function looks interesting and useful; however, it appears to be out of scope for consolidation/config.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants