Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use text-based columnar storage by default for big memory and processing savings #96

Closed
wants to merge 4 commits into from

Commits on Apr 25, 2015

  1. Use text-based columnar storage by default for big memory and process…

    …ing savings
    
    This changes the default data for mime-types from a JSON
    file to a set of text files, with one line per mime type, and one
    file per mime type attribute.  This only loads the content_type and
    extensions for mime-types by default, reducing default memory usage
    of mime-types from over 22MB to just over 2MB.  Additionally, it
    speeds up requiring from 0.38 seconds to about 0.10 seconds.
    
    Even when loading all of the attributes for the mime types, the
    memory usage of mime-types stays under 11MB, though load time
    increases to 0.52 seconds.  I think that is an acceptable trade off.
    
    If the txt files are shipped instead of the json file, there is also
    a disk space savings of 167KB (392KB instead of 560KB).
    
    rake convert:yaml:txt has been added for converting the yaml files
    to the necessary txt files.  In order to get the conversion to work,
    I had to make convert.rb turn on lazy loading for mime types.
    
    This should be completely backwards compatible if a path is given
    containing the types.  The only current backwards compatibility
    difference is for the "text/directory - DEPRECATED by RFC6350" mime
    type, which looks to me like an invalid mime type anyway, and
    probably should be removed.
    jeremyevans committed Apr 25, 2015
    Configuration menu
    Copy the full SHA
    d6a2581 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3aec11a View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2015

  1. Pool encoding strings

    Use a single string for each encoding.  This decreases the number of
    encoding strings stored by default from 1909 to 4, saving about 700KB
    of memory by default on 64-bit MRI.
    
    While here, reduce object allocations while loading data from the
    mime type txt files.
    jeremyevans committed Apr 26, 2015
    Configuration menu
    Copy the full SHA
    79fa9fb View commit details
    Browse the repository at this point in the history
  2. Clear cache file in lazy test

    This was causing the test-order dependent bug on JRuby.
    The clear_cache_file code was taken from the cache test.
    jeremyevans committed Apr 26, 2015
    Configuration menu
    Copy the full SHA
    3acfd6c View commit details
    Browse the repository at this point in the history