This meeting was arranged at relatively short notice (i.e. unplanned) as we were facing some technical questions which we thought would benefit from face-to-face contact. But we also took the opportunity to treat the meeting as a mini sprint review.
The technical issues concerned (a) some identified optimizations which, while effective on a limited test query, we felt might not be effective across the range of queries CLAROS needed to perform, and (b) difficulties in replicating the updated server environment at Oxford.
Important outcomes are:
- The hypothesis that specialized indexes can resolve query performance problems is supported by some concrete evidence
- Substantial performance gains, to the extent that sub-second query performance may be achievable on current data, can be realized by materializing query results and appropriate arrangement of the queries used, but will probably not scale well to even larger data volumes (probably linear). Isolating values from about 66,000 object records takes over 0.5s
- More care is needed to ensure consistency of source data used for development and testing purposes
More detailed notes from the meeting are at http://code.google.com/p/milarq/wiki/20100611_Meeting_Project_Progress
No comments:
Post a Comment