I have just got this programme to see if I can use the data I have to get me the best info, I started with free trial but “upgraded” to Professional as limited numbers of entries.
My Data - each month I get a monthly total of “sales” which is categorised in to hundreds of categories. I can filter records and do on TWO categories only and download as a CSV. I am comfortable using Excel, pivot tables, access etc. so no problem manipulating data but it’s very timely.
So the data - each month my file contains 18 MILLION lines and 28 columns about 7.5GB file. I filter out my two categories and One is about 30K Records per month and the other 100K I then join these together as the information is the same just category different. I need an annual report so need to get 12 months worth of data into RAGIC so what is the BEST and EASIEST way to do this without spending hours in Excel? TIA
Hi Paul,
You really don’t want Ragic (or Excel) touching the full 18M-row files.
What works well at this scale is putting a small service in front that streams the file, filters the two categories you care about, and then writes the cleaned data straight into Ragic via the API. Ragic only ever sees the ~30k / 100k rows it’s good at handling.
That removes Excel completely and makes monthly (and annual) reporting pretty painless.
Hope that helps 
Who is Simone?
Hello,
So, is the problem about the CSV being too big for Ragic to handle? Your best shot is to use Python to split the CSV into smaller files.
Or you can also export the CSV from the original source with even more filters so that each resultant file would be < 200 MB.
Thanks so much for coming back the honest answer is I don’t know. My record sets import and I had free package limited to 1000 records, every time I tried to do any reporting I just got an “error” as more than 1000 records. So I bought professional unlimited package and then get exactly the same “error” so although my data is in I can not get the information I want out of it.
I need to get the information out of the data so I can use it to save my business which is under threat!
Dear @paul12
Thank you in advance for your additional feedback on your question.
And many thanks to all participants.
What I would like to suggest is the following:
1st of all, let’s take things step by step.
In your case, as you have explained
-
We have a system out of which you extract each month, and amount of data.
-
This amount of data is quite big, includes the data you want, yet it has a serious amount of data that is not needed.
-
In terms of the data needed, we know that are of a specific size (approximately) so , that amount data can be handled by Magic.
-
You have obtained a subscription in Ragic so there is no further issue initially.
From those details above if they are valid, the first step is to examine IF:
A01. We should check what is the system that you extract the data from.
A02. We should check what data extraction options the system has.
A03. We should check if the filtering of the categories you need, can be done, Before extraction of the data.
IF this can happen , then you can extract only the data you need so that the file to upload in Ragic is smaller significantly.
After finding out about these actions, we can further proceed to next steps and thoughts on solutions.
Sincerely to All.
Thanks I have a public (NHS) database which reports monthly about 2-3 months in arrears. It has about 18M records about 7.5gb per month. Inside this file there are numerous categories which I filter out and down load two categories each month.
These two files consist of 30K+ and 70K+ records each month. I need this data and therefore filtered out what I need. They export as CSV files and have about 25 columns - headers. Except for two of these headers I need all the rest of info. Each month there are new products and users from previous months.
I need to have a 12 month rolling report as there is currently a review which may end my business.
I have opened a support question with RAGIC I have given them links to data and also permission to access my files - downloads of last two month data.
My simple request is to get information on how many prescriptions and £value of each product and by which Region/Area/Postcode/Practice they are prescribed, in the least time and painless way compared to now.
So dear @paul12
Much more clear now of what is exactly the burden each month.
A question comes to mind though.
Do you 1st download the data unfiltered and then filter the two categories?
OR
You 1st filter the two categories and then download the data?
This is quite essential, because it is a significant info what is the size of the CSV file that you need each month to drop into Ragic to aggregate and analyze.
Apart from my question, it was also needed to post a request to the Ragic team, so you did well.
Thanks no way would have a programme or the capacity to download 18M Lines and 25 columns! 
So to answer you I just filter out my two sections and download 2 small CSV files for each section.
Ok cool, seems reasonable 
Then the smaller CSV files that you get are the ones that need to be uploaded to Ragic so that you can proceed with the rest of your tasks.
Now the issue comes to the notification you got when you tried to upload the files into Ragic.
What I can think 
of the issue, is the chance that your Ragic plan subscription is not enough for the size of the 100k rows,
OR
There is some other technical issue that we are both not aware of.
Yet, a CSV file of 25 fields and 100k rows I cannot easily calculate what its size must be.
Do you know what size they are even approximately?
Thanks again.
IV
Files together for each month combined file 30mb, not big, they are imported no problem I had free and upgraded to professional as it said free was limited to1000 rows.
So indeed the size is doable and the subscription seems proper.
What about the issue coming up…what is the exact error type that shows ?
And IF this error is the one for which you sent the request to the Ragic team, — have you considered the next steps for finding the data insights that you mentioned?
Have you seen the reporting capabilities of Ragic even as a demo or is it completely new for you?
IV
Thanks the”error” I keep getting is the report donations more than 1000 lines so basically I have not seen any reports as I have data but cant do a report on anything so can’t comment on reporting. I am still awaiting response from support. THANKS for your time nd input appreciated “muchly”
No problem @paul12
Happy to assist, but I cannot easily understand what the " the report donations more than 1000 lines" means.
Do you mean that - the error says “the report exceeds 1000 lines”??? So in essence it is in a Non-Viewable format?
Let me know further if and when you can. Thanks.
Neither can I 
Yes it means I can not see any reports.
So …this seems to be an implementation problem.
Because there no issue with the volume of the files but there is an issue with the structure of the Report or the effort you have made to build a report.
This means that you must add some extra categorizations in your data in order to aggregate them further, to be able see a report running properly, but as partial “pictures” of the truth each time.
If you have categories already implemented in the data such as Geo area, category of medicine or product, dates, if prescription is paid or not, if it is executed or not and so on…
You must try to take advantage of them and dissect property the data to be more presentable (viewable).
Is there any chance that you can send me just the names of the fields (columns) of the data?
So that I can propose to you some kind of structure that may be more easily visible?
I would like to check if such a solution could give you the results you need.
Just in case that even the column names are confidential, you can send them to
Thanks for clarifying. Be 
cool, you can do it.
IV
The data is in public domain here’s link to files for November the last one published here’s filter:
The filter is: BNF_CHEMICAL_SUBSTANCE and the two secondary inters are Elastic Hosiery and LYMPHODEMA GARMENTS feel free to take whatever data you need. As you can see the rest of the data for each month is there. THANKS for your help
Thanks so much @paul12
Since we are waiting for the Support Team, I will give it a try under your permission as you said, and I will let you know as soon as possible.
We will figure it out in the end.
I suppose you are in UK time, so I will have that in mind in terms of working hours or office hours.
Thanks again for the trust there is in it.
Talk again soon with more details and suggestions.
IV
Hi again @paul12
I hope you are fine in health.
A) to be honest I accidentally downloaded last night the whole 2025_11 pack of the millions of lines.
I initially had filtered the data but the button for downloading the filtered was elsewhere , lower on the right side a bit hidden. So…I made the effort again and I filtered a file of just 25k rows to check the data.
B) I have some small questions that I will provide below, so that it can be much more clear to me of what we can do about it. I have already studied the Data Dictionary and I would like to ask the following:
This is your request (phrase)
“My simple request is to get information on how many prescriptions and £value of each product and by which Region/Area/Postcode/Practice they are prescribed, in the least time and painless way compared to now.”
Q01 - When we say “Per Product”, do we mean SNOMED code? like “57488XXXXXXXX”
Q02 - When we say “Area”, do we mean “ICB_NAME” ? like “NHS GLOUCESTERSHIRE INT…”
N01- I have understood that “Region” is the “REGIONAL_OFFICE_NAME”
N02 - I have understood that “PostCode” is the “POSTCODE”
N03 - I have understood that “Practice” is the “PRACTICE_NAME”
=========================================================
Proceeding… according to the answers to be given in the above questions, I would like to make a distinction between two concepts that are related to Ragic. Both concepts I am going to tell you about, are described below in my best effort & intention to help you, and help myself also to understand your needs.
Concept-01
Ragic Data Grid - The data that are uploaded into Ragic in a “Sheet” (table) are what we both know, as the “raw data”.
While a Ragic Sheet contains raw data, the user has the available feature of Filtering that data while using the grid to display them.
This means that, if the dataset is quite large (even less than 100k rows e.g. 25k rows that I got), when applying a simple filtering on the grid, to filter 1 column (or more), the results presented via the grid can be more than 1000 lines (highly possible).
So, we both easily understand that Filtering the data grid (the Ragic Data Sheet) (our table), is not a Report, no report has been created, it is just a filtered table. That method is not using any aggregation of the data and is not really helpful for your needs.
Concept-02
The Ragic Reporting features - Within Ragic there are some features that serve the purpose of “boosting” the “Aggregation” of results, and “serving” the “Presentation” offering of Ragic so far. These features can be found when you go to CREATE a NEW REPORT
You have in Ragic several Report types to create if you like as shown below
among others.
So, IF in the recent past, you were just filtering the data grid, instead of creating a report, you must try creating one, that serves your purpose as best as it can.
But IF you have already created a report, and that report is executing several filters, yet, you get the error of Non Visibility because of the many resulted rows, then we must think of something else.
That comment above - is a another question (open one) that you need to provide me with your valuable answer in order to clarify the issue.
========================================================================
An examination of your needs & their high significance.
- You have already said that those Datasets are utmostly important for your business.
- Those datasets are open, yet, they are repetitive (monthly basis).
- Those datasets are quite heavy as non pre-processed.
- You need specific filters applied each time, and you need to see specific measures.
- You need a consistent way / method to drive results from these data & their analyses.
So, driving a single conclusion from the situation, that covers almost everything is that you need
A Data Pipeline to be built, so that you can perform simple, repetitive tasks, each month, to get the results needed, for boosting your business and making it thrive.
What about Ragic
Ragic can serve for sure 1 big part at least, of your Data Pipeline, which is the structured, storage of the data, and the calculation of any “new measures” or “new stats” that you may want to create AFTER receiving the data from the EPD.
So,
Step01 - We get the data into Ragic.
Step02 - We think of any additional calculations or dimensions (for analysis) you may need, before creating any reporting.
Step03 - We test if Ragic report Types can cover your needs.
IF NOT - we see from there on fwd.
========================================================================
I have written a lot. I apologize for the burden. Let me know also in my direct email, when you
have read all these above and you have any question on my comments or questions. For starters
the stats you require are quite easy, it is just quite many combinations of Area, Geo, Product, Practice etc…
Best Regards
IV
First of all THANKS for your time and input and also the pain of downloading the whole database and not my two data sets. I have just received this morning an email to state that most probably my data set is too large as this is only one months worth and I need 12 months rolling report on aa monthly basis.
As I was always told is stop the bleeding in a project when it can not be done. So please stop as I dont think I can do what I want. However my dad always taught me that if someone says “you can not do it” then show them you can! Thats always been meh and I WILL get this done somehow but perhaps Ragic was not my tool.
So a PERSONAL THANK YOU to you I appreciate your time and effort.
Have great weekend!