Popular games or applications can see volumes of 1,000,000 submitted crashes a week with average attached object files (i.e. dumps or attachments) ranging from 15-20MB. Storing all these objects can easily fill up 17.5TB of storage every week, or 200TB for 90 days retention of data.
When a fingerprint has large numbers of objects, storing most objects beyond a certain amount, generally a few dozen, adds little value. Backtrace offers storage sampling to control object storage with respect to a unique combination of fingerprint and optional specified attribute(s) values. Storage sampling dynamically decides upon ingestion which files should be kept in long term storage and which can be discarded after processing is complete. Most Backtrace instances have disk quota associated to them, and storage sampling provides a way to control the usage of disk depending on your teams specific needs by limiting the number of objects stored within a fingerprint.
Storage sampling can be configured by Admin users in the UI under Main menu | Project settings | Storage | Sampling
Here is an example config:
This configuration will respect unique combinations of fingerprint and version. It will store 2 objects with 0 minutes between them, then 4 more objects with at least 1 minute between them, then 8 objects with at least 60 minutes between them. After the final backoff level, nothing is stored until the reset interval (1 day) is reached. Then, the backoffs start again from the first bucket.
Note on Sampling Backoffs: Backoffs allow us to specify minimum time intervals between storing received objects. Backoffs include a count (number of objects to be stored) and an interval (minimum number of seconds between storing a received object). Note that a backoff's time interval is a minimum and there is no guarantee of getting objects for a particular group in any particular timeframe.
Indexing of objects always occurs even when an object is deleted by sampling, so metadata will always be updated by each object. The object deletion only impacts the debugger view and object download.
Note: Storage sampling configurations include a reset interval at which the sampling counts are reset. Backtrace also provides a Retention Policy system, and generally, this reset interval for Storage Sampling should be no more than the Retention Policy's deletion interval to ensure that if objects within a given sampling group continue to be submitted, then there are always objects available in it. As of March 5, 2020, the reset interval begins when the final backoff's bucket is filled. Note that we are considering updating this behavior so that the interval begins upon a group's first stored object.
Backtrace will work with customers to enable sampling configurations that are appropriate for their instance or projects.