Thank you for your contribution!

I'm not sure about this idea, however. The proper solution is to buffer large groups on the disk first, just like large regular results are buffered, but with groups situation is more complicated, they need a more complicated format.

I also think that number of groups is a very poor metric. In regular results limit of in-memory data before buffering is specified in rows and it is a subject to change, because it doesn't actually prevent OOME in various cases. With groups number of grouped rows has even smaller meaning, because some aggregates are very cheap and others are actually hold all values from the original rows and their count isn't limited anywhere in your PR.

I think we can accept a some new setting for the grouped and window results, but it would be nice to estimate their sizes better than there. The new setting should also be explicitly marked as experimental only in documentation, and I think the special case with group-sorted optimization should also be mentioned, I guess it isn't affected by your new setting. This setting should also be covered by a some test.

Thanks for your comments.

I completely agree with you on what you say. The better solution would be like you say to use disk for large result sets, and number of entries is a poor metric for memory consumption. The only saving grace of this approach is that because its simplistic, it was also quite simple to do. It being my first contribution, I didn't want to go for something too complicated.

If you feel there is merit in pursuing this further, I'll work on improving the metric to be closer to the actual memory being used, while keeping the calculation simple and fast. I'll also include some unit tests for it.

Regarding the setting being marked experimental, can you give me some pointer on where that would be done? Is it just a matter of saying so in SysProperties?

Apologies for asking a maybe dumb question, I am just a curious visitor: would it not be a better idea to define per Connection Setting, whether to use a Memory Based Map ("regular systems") or a Disk based HashMap ("memory constraint embedded systems") for the Grouped By Entries? Something like https://github.com/bnyeggen/lash or maybe https://github.com/OpenHFT/Chronicle-Map, if MVSore was not fast enough?

I found this one: https://github.com/lmdbjava/benchmarks/blob/master/results/20160630/README.md
It is quite dated, but at least in 2016 MVStore was not the fastest solution and caching/backing large map to the disk might need that fastest speed possible?

Is it just a matter of saying so in SysProperties?

Yes, it should be enough to mention the experimental status in the Javadoc. Most likely this setting will be removed or replaced with some other setting when better implementation of storage arrives.

@manticore-projects
Groups and window partitions aren't just values, their data structure is more complicated and cannot be persisted directly, because it can contain various objects (including third-party objects) and they aren't required to be serializable. The whole this class is a subject to change, it was written as a first draft implementation of storage for both groups and window partitions and it was based on archaic storage for groups that had the same problems.