Quantcast
Channel: Active questions tagged https - Stack Overflow
Viewing all articles
Browse latest Browse all 1573

Architecture for paginating a large dataset

$
0
0

To give some context, we are building an application to analyze and annotate large signals (lasting up to several weeks) and can have varying frequencies (100-500Hz). These signals can have between 1 and 12 channels and can vary from Mb to up to 2Gb of size.

On the initial load of our display, we make 2 requests for each channel, one to fetch the signal information and one to fetch the labels that have already been identified.

For signals lasting less than 60 seconds, we fetch the full signal length for each channel which results in an acceptable, but heavy first batch of requests being sent (up to 24). For longer signals, we paginate the request so that when the graph scrolls to a certain time, the next page gets loaded.

We use different caching mechanisms to avoid having to make too many requests but we are wondering if a different architecture might help. We use classic HTTPS requests for the pagination and we're wondering if using websockets would increase our ability to scale users accessing different signals and lighten the load for the backend.

Any thoughts/experiences would be much appreciated.


Viewing all articles
Browse latest Browse all 1573

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>