Main Content

Work with Big Data for Simulations

Simulation of models with many time steps and signals can involve big data that is too large to fit into the RAM of your computer. Such situations include:

  • Logging simulation data (signal logging, output port logging, and state logging)

  • Loading input signal data for simulating a model

  • Running multiple or parallel simulations

To work with big data for simulations, store the data to persistent storage in a MAT-file. Using big data techniques for simulations requires additional steps beyond what you do when the data is small enough to fit in workspace memory. As you develop a model, consider logging and loading simulation data without using persistent storage unless you discover that your model has big data requirements that overload memory.

Big Data Workflow

This example is a high-level workflow for handling big data that one simulation produces and that another simulation uses as input. For more detailed information about the major workflow tasks, see:

Tip

This example uses aSimulationDatastoreobject for streaming data into a model. Alternatively, you can stream aDatasetRefobject directly into a model.

  1. Configure two models to log several signals.

  2. Simulate the models, logging the data to persistent storage for each model.

    sim(mdl1,'LoggingToFile','on','LoggingFileName','data1.mat'); sim(mdl2,'LoggingToFile','on','LoggingFileName','data2.mat');

    Logging that involves big data requires saving the data to persistent storage as a v7.3 MAT-file. Only the data logged inDatasetformat is saved to the file. Data logged in other formats, such asStructure with time, is saved in memory, in the base workspace.

    The data that you log to persistent storage is streamed during the simulation in small chunks, to minimize memory requirements. The data is stored in a file that containsDatasetobjects for each set of logged data (for example,logsoutandxout).

  3. CreateDatasetRefobjects (dsr1anddsr2) for specific sets of logged signals. Then createSimulationDatastoreobjects (dst1anddst2) for values of elements of theDatasetRefobjects. This example code creates aSimulationDatastorefor the 12th element oflogsoutfor the first simulation. For the second simulation, the example code creates a signal with values being aSimulationDatastoreobject for the seventh element oflogsout. You can use curly braces for indexing.

    dsr1 = 金宝appSimulink.SimulationData.DatasetRef ('data1.mat','logsout'); dsr2 = Simulink.SimulationData.DatasetRef('data2.mat','logsout'); dst1 = dsr1{12}; dst2 = dsr2{7};

  4. UseSimulationDatastoreobjects as an external input for another simulation. To load theSimulationDatastoredata, include it in aDatasetobject. The datastore input is incrementally loaded from the MAT-file. The third input is atimeseriesobject, which is loaded into memory as a whole, not incrementally.

    input = Simulink.SimulationData.Dataset; input{1} = dst1; input{2} = dst2; ts = timeseries(rand(5,1),1,'Name','RandomSignals'); input{3} = ts; sim(mdl3,'ExternalInput',“输入”);

  5. Use MATLAB®big data analysis to work with theSimulationDatastoreobjects. Create atimetableobject by reading the values of aSimulationDatastoreobject. Thereadfunction reads a portion of the data. Thereadallfunction reads all the data.

    tt = dst1.Values.read;

  6. Set the MATLAB session as the global execution environment (mapreducer) for working with the talltimetable. Create a talltimetablefrom aSimulationDatastoreobject and read atimetableobject with in-memory data.

    mapreducer(0); ttt = tall(dst1.Values);

Tip

For another example showing how to work with big simulation data, seeWorking with Big Data.

See Also

Functions

Related Topics