TESVSnip Dev 4-4-0a - Out Of System Memory

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Open Skyrim.esm with no filters

What is the expected output?

It should open and all of it should be available.

What do you see instead?

Out of system memory error.  How would that be resolved I wonder?  Does this 
mean that the file has to use virtual memory?

Please use labels and text to provide additional information.

Original issue reported on code.google.com by [email protected] on 22 Sep 2012 at 8:18

[GoogleCodeExporter]

Ok and if possible I'd like it to open Skyrim.esm, Update.esm, and 
Dawnguard.esm.

Original comment by [email protected] on 22 Sep 2012 at 8:43

[GoogleCodeExporter]

For reproduce you must do
Open Skyrim.esm with no filters
Close all
Open Skyrim.esm with no filters

Original comment by smaug.nogardeht on 22 Sep 2012 at 8:44

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Le 22/09/2012 22:45, [email protected] a �crit :

Yes i think.
I had a memory leak tools but i never use it.
I'm trying to fing the memory leak.

Original comment by smaug.nogardeht on 22 Sep 2012 at 8:56

[GoogleCodeExporter]

I understand that now thank you for clarifying that.  For me it happens without 
opening Skyrim.esm with no filters, then closing all, and the opening 
Skyrim.esm again.

It happens on the first try.  I have a 32 Bit Install of Windows 7, 4gb of ram 
of which only 3GB are free because of OS limitations.  

Here is what I did:

1: Open Skyrim.esm without filters.
2: Error comes up and I click ok.
3: I started typing this message
4: I brought Snip to the foreground so that I could close it since I got the 
error.
5: I noticed it had loaded.
6: I started clicking on things and data has been loaded.  I just don't know if 
100% of Skyrim.esm is loaded and in memory or not.

NOTE: If I click "Expand All" with skyrim.esm open the system will hang trying 
to open all of the world space.  So I can't do that.  I'll make a ticket for a 
warning when you are about to do that with Skyrim.esm open.

At the moment with Snip 

Original comment by [email protected] on 22 Sep 2012 at 8:57

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I expanded all the way up to NAVI so far and everything is fine.  I skiped over 
WRLD, CELL, and DIAL.  I clicked QUST and I have the wait icon, on my system 
it's a spinning circle.

If it hangs, and keeps saying (Not Responding) I'm going to close it restart my 
system and open QUST first thing, instead of expanding all the other trees 
first.

Original comment by [email protected] on 22 Sep 2012 at 9:04

[GoogleCodeExporter]

After restarting I opened snip, and I let it error.  After it finished loading 
then I went straight to QUST and expanded it only, without expanding other 
trees.  

QUST expanded and opened normally without a wait icon.  I was able to expand 
all the trees from QUST to the last GRUP at the bottom, REVB.  No wait icon and 
no no pause when expanding them.  I did not open DIAL, CELL or WRLD yet.  I am 
going to do that now.

Original comment by [email protected] on 22 Sep 2012 at 9:21

[GoogleCodeExporter]

Okay I have about 10-15 screen shots of errors to upload give me a minute.

Original comment by [email protected] on 22 Sep 2012 at 9:46

[GoogleCodeExporter]

So it was 8 ^_^  Here they are.  I am not sure if those are parameters in the 
C# code or a result of the xml file.  I will look through the xml file now.

Original comment by [email protected] on 22 Sep 2012 at 9:55

Attachments:

• SnipWRLDErr01.jpg
• SnipWRLDErr02.jpg
• SnipWRLDErr03.jpg
• SnipWRLDErr04.jpg
• SnipWRLDErr05.jpg
• SnipWRLDErr06.jpg
• SnipWRLDErr07.jpg
• SnipWRLDErr08.jpg

[GoogleCodeExporter]

I used only the filters showing in the first screen shot.  Then I had those 
errors, and the SystemOutOfMemoryException repeated several times.

Original comment by [email protected] on 22 Sep 2012 at 10:02

[GoogleCodeExporter]

Original comment by [email protected] on 22 Sep 2012 at 10:36

• Added labels: Defect-TESVSnip
• Removed labels: Type-Defect

[GoogleCodeExporter]

In r15 I removed all of the unsigned numbers as a test.  Now I still get the 
out of memory issue but I do not get any of the other issues.

Previously, before any refactoring, the Data Size would be randomly 1-9 bytes 
different.  So Data Size calculation has always been an issue.  Part of what 
caused the initial drama with Snip in the first place.  There are also plenty 
of places I saw in the original code where there were signed and unsigned 
numbers being mixed.

Maybe StyleCop could help, if that does this, maybe it could be used to check 
that all numbers being compared are in fact signed or unsigned so that there 
isn't some kind of issue from that.

Original comment by [email protected] on 22 Sep 2012 at 11:03

[GoogleCodeExporter]

Also in the second screen shot i have the GRUP record selected Parent id 
0x00000d74.  It says there are 15193 records?  Are there?  Is it more like 
there are 1500 records but total records and subrecords within all the records 
makes it 15193?

Original comment by [email protected] on 22 Sep 2012 at 11:25

Attachments:

• SnipWRLDErr09.jpg
• SnipWRLDErr10.jpg

[GoogleCodeExporter]

More info - recordstructure.xml rev 24
------------------------------------------
Trace of crash.
Below a dump of the buffer where tesvsnip crash.
It's a LAND record. It's a compressed record.


--------------
Crash on this function
BinaryReader Decompress(Stream input, int expectedSize = 0)
expectedSize=12014

-------------------------------
Calling function
File: Record.cs
Function: internal Record(string name, uint dataSize, BinaryReader 
recordReader, bool oblivion)
name = "LAND"
dataSize = 6362
dataSizeInParam = 6366 // before the subtraction of-4

this.Flags1 = 262144
this.FormID = 41880
this.Flags2 = 3237660
this.Flags3 = 327719

compressed = true

realSize = 12014


Crach here  dataReader = ZLib.Decompress(stream, (int) realSize);

Original comment by smaug.nogardeht on 25 Sep 2012 at 8:09

Attachments:

• [dump buffer.hexa](https://storage.googleapis.com/google-code-attachments/tesv-snip/issue-1/comment-15/dump buffer.hexa)
• hexa.png

[GoogleCodeExporter]

Sorry, I didn't have time this last weekend. I'll try to take a look the next 
time I have some time. Perhaps today, but I cannot say for sure. 

Original comment by [email protected] on 25 Sep 2012 at 8:27

[GoogleCodeExporter]

It's totally cool.  It could wait till the weekend even.  It needs to be done 
that's true, but I'm starting to pull back a bit from working on thins 24/7 
it's starting to burn me out.  As always I really appreciate the work both of 
you are doing to make this a better program.  I'm looking forward to releasing 
it once it's ready.

Original comment by [email protected] on 25 Sep 2012 at 8:32

[GoogleCodeExporter]

I see you highlighted a LAND record, I just updated that as much as I could.  
Until there are repeating elements, groups, and other things as outlined 
RecordStructureFormat.txt it's the best I ca do.  Hopefully that new definition 
helps.

The only Issue I see with the new definition is that since the record might 
have only 3267 bytes, it's not divisible by 4.  So at the end of the wall of 
text that I have there saying "Rest of Data" sometimes it terminates with one, 
two or three bytes and not always an even 4.

Original comment by [email protected] on 25 Sep 2012 at 8:43

[GoogleCodeExporter]

I am looking for the error but I am not very familiar with the structure of the 
records.

I'm looking by comparing statements betwenn recordstructure.xml, tes5edit and 
tes5dump.

Original comment by smaug.nogardeht on 25 Sep 2012 at 8:47

[GoogleCodeExporter]

To save you some time, the only file to look at for Skyrim's record structures 
is:

http://code.google.com/p/skyrim-plugin-decoding-project/source/browse/TES5Edit/t
runk/wbDefinitionsTES5.pas

Original comment by [email protected] on 25 Sep 2012 at 9:00

[GoogleCodeExporter]

I'd suggest looking at the UESP Wiki but unless I say I updated it then it may 
not be as accurate of a definition that TES5Edit has.  Also don't change 
RecordStructure.xml since I am still working on it and getting the rest of the 
records updated and verified.

Original comment by [email protected] on 25 Sep 2012 at 9:02

[GoogleCodeExporter]

http://code.google.com/p/skyrim-plugin-decoding-project/wiki/UESPWikiUpdates

Those are what I have updated

Original comment by [email protected] on 25 Sep 2012 at 11:34