OutOfMemoryErrors and ObjectOutputStream

Ran into an interesting problem today - turns out if you use an ObjectOutputStream to write out lots of data the data doesn't go away after OOS.write() returns - instead it gets cached in a handles map within the OOS itself so that should it be told to write that object instance again it can instead write a handle reference (thus saving on space in the stream and avoiding circular references). Kewl, huh? Well, not entirely...

The downside to this approach is that if you're writing LOTS of objects (say 250,000) then even tho you've designed your code to avoid holding all those objects in memory every one of them will be held in memory until you finish with the OOS entirely because of that handle cache! (This explains why my simple little app eats up over 2GB of RAM even tho it's supposed to be processing data objects one object at a time). The fix?

Two came up and I'm not 100% certain which is better all the time so I'll discuss them both (in the order in which I thought of them). :)

Solution #1 - Unshared Externalizable/Serializable

There's a new interface in town, java.io.Externalizable, which is the cousin of java.io.Serializable, but with a few new quirks:

  1. First, unlike Serializable, Externalizable has methods (imagine that! methods in an interface!) - readExternal(ObjectInput) and writeExternal(ObjectOutput), typically the args are Object(Input|Output)Streams, which implement Object(Input|Output).

  2. The second (and rather weird) quirk is that your Externalizable object must have a PUBLIC no-arg constructor. Now, considering that the ctor is gotten via reflection I'm a little puzzled why it has to be public but... whatever.

So first you make your DAO implement Externalizable rather than Serializable (which is a parent of Externalizable) and then you replace all your ObjectOutputStream.writeObject() calls with writeUnshared() (and on the read side OIS.readUnshared()) which tells the streams to just ignore that whole handle map thing.

Now you might think just making your DAO unshared is enough but NO! (unless your DAO is the data itself, which is nearly impossible since all DAOs are, at some level, composed of primitives, primitive-wrappers, arrays or collections of primitive/primitive-wrappers). So while your DAO might be unshared (not cached) its parts are probably shared (cached). That's why you have to implement Externalizable or Serializable - after a bit of testing i'd just go with Serializable - no funky public ctor and no casting of ObjectInput to ObjectInputStream and so forth. Six of one, half a baker's dozen of the other... :)

Solution #2 - ObjectOutputStream with amnesia

There is a method on OOS called reset() which clears the handle cache and reduces memory use to nearly nothing (other than what you were using without the OOS). It also injects a TC_RESET marker into the output stream which, in my DAO's case, increased the output file size by 50% (e.g., what was a 4.9M file became a 7.2M file). basically if you look in the serialized file you will see your DAO's and component fields' class names between every single instance (normally they're declared once at the top). So if size of your serialized data is an issue this might not be a great fix. But if RAM usage is a concern this approach is huge - my app which used to max out at over 400M (in testing) with regular serialization dropped to 100M with unshared but then dropped all the way to less than 1M java heap with reset() between each writeUnshared() (hard to say exactly 'cause there were so few Full-GCs but i'd guess it was about 400-500K). THAT's amazing (to me).

Performance Impact

Didn't see any performance impact in calling reset() between each writeUnshard(); in fact i'd say the reset() version works faster because it avoids all those Full-GCs.

Hope this helps folks as much as learning it helped me. :)


Home Alone - and what it taught me

today was the last day of the first stretch of family-free time i've had since, well, since i started my own family back in '98. granted, kid #1 didn't arrive until 2001, so it's fair to say that was the last time the house was empty for more than a few hours. but on Thursday my wife and 2 kids went to visit her relatives and i chose, despite the points it cost me, to stay behind. and i must say it was worth it.

Day 0 (Thursday, 8/14/08)
~0900 - the very first thing i did after waving family and nanny off to the airport was to turn off everything non-essential in the house. my family has allergies and also a habit of leaving lights and TVs on so as soon as the house was mine i turned it all off - air filters, air conditioners, TVs, even the night lights in the hallway. the utility meter never had such a holiday since we moved in. only fridge, computer, and everything between it and the Internet stayed on.
~0930 - the next thing was to find a good book ("The Watchmen" in this case) and go sit on "the throne" with the door wide open. ah, the freedom of an open bathroom door - how strange it is to miss something so simple. :)
1000 - (all times are approximate anyway) - took shower and then walked around the house a la natural. yet another simple pleasure i never realized i had lost.
1030 - off to work (i have an understanding employer - they understand because i do tons of hours from home when i'm not at work).
1700 - left work and came straight home. this is unbelievably early for me to leave work - normally (i.e., when the family is home) i stay at work till 1800+. not something i'm proud of, but it does give me some sense of balance between my paying job and my Daddy job.
1710 - i seem to remember playing a lot of Rockband (i love the drums, tho i wish they were more flexible in the layout - can you imagine Neil Peart having all his drums laid out on the same level side-by-side? but still, it's fun) :)

Day #1 - Friday - didn't go to work!!!! (this is unheard of in remembered history - for me to not go to work unless deathly ill or in some other way incapacitated or unable) instead i went around the house fixing things, hanging pictures, repairing the cable in the master bedroom, etc. and best of all i didn't have to keep a weathered eye on my tools at all times to watch for mischievous little hands looking to snatch away an unattended hammer or screw driver. oh, the joy of knowing your tools won't run away or you won't be called on to do this, that, or whatever, while you're trying to finish something else which is important to you, even if others don't think that what you're doing is important. Finished the day with more Rockband and working out. Ate surprisingly little - just didn't think about it. :-/

Day #2 - Saturday - more running around the house fixing things, watering plants outside (it's been hot in WA lately), trip to Fred Meyer with a list of only my own things (typically my shopping lists are written in my wife's hand and often i have to look it over and ask a number of clarifying questions before i head out. but this time every item made sense and was clearly legible and this, too, was a simple pleasure i had forgotten from way back when). i think it was about this time i started talking to myself... i also was becoming quite used to how hot the house is without the AC running. after a while it felt comfortable to feel sweaty all the time - weird, but true.

Day #3 - Sunday (today, or more accurately yesterday) - until 0930 this morning there was a chance they would stay in AZ a few more days, but my wife informed me that things had transpired with the family which guaranteed their return today). at that point i went into high gear (until then i had actually considered if i could miss work Monday as well) because there were various things i hadn't finished and i knew once they arrived my productivity would drop like the Market on Black Monday. Helter skelter for 6 hours (the last 2 of which i started worrying like i did in the old days - day-mares of crashed planes, car accidents, all sorts of highly unlikely events that would bring my world crashing down). Luckily tho, as had been in the old days, all my worrying was unfounded and my (and their) vacations ended. And from it i learned a number of interesting facts:

1. i eat a lot less when i eat by my own schedule (i.e., alone and only when i'm hungry)
2. i can go days without working (or even using a computer) if i'm left to do other things (like Rockband) that i enjoy
3. your true nature, even if repressed for 10 years, is still your true nature when given the opportunity to express itself
4. i don't mind the heat anywhere near as much as my family does
5. a house (and my brain) is much quieter without all the background noise (mostly fans - of which i can hear 3 right now, including the fan on my computer).

oh yes, and i think i figured out something about blogs too - writing blogs is for people who want to have a diary to record thoughts but realize that keeping those records secret is a waste of effort - otherwise you should just spend the time talking to yourself. but a blog - it's like a diary that people can read - people you don't even know (and hopefully don't know you - in particular i'm hoping my family never finds my blog - otherwise i'm never going to write another thing on here). but as fun as this was i'm sure i'll forget to worry about it eventually and hopefully write something someone would find worth reading (no refunds tho - time wasted reading my drivel is time lost for good). :) now to fix those spelling errors...

oh, two good quotes i thought of today:
- "The most important thing I learned after school was that much of what I had learned in school was worthless."
- "Those who forget history are doomed to repeat it; but, then again, so are those who remember it."



i've finally taken the first step - created a blog.  fortunately, yet unfortunately, that was the easy part.  what does one put in a blog?  what to share and how much is too much?  wow.  talk about (self-induced) pressure!  when i created my website i was able to keep it simple - a shared bookmark and useful forms page.  but this is something different - something more dynamic, organic, and personal.  i'm certain many learned folks have written books about writing blogs (as ironic as that seems) but it seems wrong somehow to read up on how to write one of these things...

well, there it is - step #0.  when i'll risk the next step who can say?