What is a suitable format for writing large amounts of data within few milliseconds
I have a c++ code that needs to store some data whenever an event is triggered. The data contains about 3000 floating point values. So each of these values needs to be written in a file when the event is triggered. On an average, the event is triggered at every 25 milliseconds. So, we need to finish writing 3000 floating point values within 25 milliseconds before another event is triggered. We can assume that each log is 10 minutes long i.e. after data for 10 minutes are logged in a file, another file is created to perform similar writing task. Which is the most suitable way to write this amount of data. csv or json can be quite time taking and data writing will not be done within 25ms. Is there any other method of file writing that can be used in c++ and can scale with such large amount of data? (Edit : ) Hardware Information : SSD : NVMe CPU : Intel Core I9 Cores : 24 RAM : 32Gb (Edit 2:) Here is a summary of how the code is working: struct DataStruct { std::vectorvec; std::mapmp; float dt; std::mapstrVec; }; void StoreData(std::ostream &outputFile, DataStruct *data[10]) { for(i = 0;i < 10;i++) // Assume that there are 10 objects { // Each object is described by 300 features. // These features are stored in data // As you can see the datatype in the structure is // heterogenous. Some of these data types are vector, some // are maps. But total number of variables in the structure // is 300 } }
I have a c++ code that needs to store some data whenever an event is triggered. The data contains about 3000 floating point values. So each of these values needs to be written in a file when the event is triggered. On an average, the event is triggered at every 25 milliseconds. So, we need to finish writing 3000 floating point values within 25 milliseconds before another event is triggered. We can assume that each log is 10 minutes long i.e. after data for 10 minutes are logged in a file, another file is created to perform similar writing task.
Which is the most suitable way to write this amount of data. csv or json can be quite time taking and data writing will not be done within 25ms. Is there any other method of file writing that can be used in c++ and can scale with such large amount of data?
(Edit : ) Hardware Information :
SSD : NVMe
CPU : Intel Core I9
Cores : 24
RAM : 32Gb
(Edit 2:) Here is a summary of how the code is working:
struct DataStruct
{
std::vectorvec;
std::mapmp;
float dt;
std::map>strVec;
};
void StoreData(std::ostream &outputFile, DataStruct *data[10])
{
for(i = 0;i < 10;i++) // Assume that there are 10 objects
{
// Each object is described by 300 features.
// These features are stored in data
// As you can see the datatype in the structure is
// heterogenous. Some of these data types are vector, some
// are maps. But total number of variables in the structure
// is 300
}
}