I am fetching a large data table (around 50k rows) via bigrquery and after performing some calculation displaying that data table in shiny dashboard. The problem is bigquery is taking around 10 seconds to load the data and I want to reduce this time.
One thing I came across is Paging through data table. But I am facing trouble implementing this in R.
I also tried using max_results and start_index as combination:
df = bq_project_query(project_id,str_interp(query),use_legacy_sql = TRUE,
max_pages = Inf,destination_table = NULL,default_dataset = NULL) %>%
bq_table_download(max_results = 5000,start_index = si) %>% as_tibble()