bulk inserts with django 1.4

[aburgel]

i've been working on dbindexer to get it ready for django 1.4. i've fixed a couple of issues already, one related to threading changes with connection objects which caused the whole thing to break on appengine. but i've run into a bigger problem with the insert changes in 1.4.

a quick primer on dbindexer for those of you unfamiliar with it. say you want to do a case insensitive query on field 'name', your nonrel db can't do this for you, so dbindexer will secretly add another field to your model called 'name_iexact'. when you insert that model, dbindexer runs before the real insert compiler and copies the value from the 'name' field into the 'name_iexact' field (and converting it to lower case). so now your db row contains an additional field. when you query, it secretly rewrites the query to use this additional field.

in 1.3, insert compilers would get access to the list of already prepared values, so dbindexer could copy and convert those values. with 1.4, the insert query no longer contains prepped values, but instead a list of objects that are meant to be prepared by the compiler itself.

to get around this issue, i have dbindexer replace its hidden fields with proxy objects that look like the original non-dbindexer field but have a different column name. when the real insert compiler runs, it goes through the list of fields for each model instance and uses each field to prepare the value for insertion. the proxied dbindexer field will look up the original value and then store it in the new column.

its a bit of a hack. but you can see it here 657331c

for some reason its flaky, sometimes working, sometimes not. i'm still refining. if anyone has ideas, i'd sure appreciate it.