Time series analysis

[yungyuc]

@j8xixo12 brought up an idea of implementing time-series analysis (in frequency domain) in modmesh. It consists of two parts: time-series and integral transform (analysis).

It is interesting to develop both from ground up. Existing (and mature) third-party packages can be used as references during the development.

@j8xixo12 do you have any ideas of how we may start the work?

[j8xixo12]

In my opinion, there are three parts we can start to work:

• File loader
  • Support csv file <-- Most simple and common format.
• Analysis algorithm implementation
  • FFT implementation
• Integrate with GUI
  • Select which column needs to be plotted in control panel
  • Select and apply filter or some analysis techniques in a configuration window <-- I'm not sure how to do it.

[yungyuc]

Yes, FFT and any use of less than a million entries is a good problem to develop the code. A model problem is a problem suitable for code development.

[j8xixo12]

Got it, I think I can try to implement the FFT with modmesh arrary library.
I also need to study how to write test code to ensure the implementation of FFT is correct.

[yungyuc]

We can use existing packages (including) numpy to verify.

[tigercosmos]

First we need to implement the complex system. I am now studying from Numpy

[yungyuc]

A complex-numbered array is good to have. Without it we can simply use two arrays for the real and imaginary parts to start coding.

[j8xixo12]

If we need real data for validation, here is the NASA official data portal that allows everybody to download some real data.

[yungyuc]

@j8xixo12 Would https://techport.nasa.gov/view/116144 be useful for prototyping time-series code?

[j8xixo12]

Yes, these real flight data would be useful for prototyping time-series code!
We can use these real data which contains multiple frequencies and noise to test our time-series code.

[yungyuc]

Kudos to @xingularity for adding the first prototype with PR #431.

@xingularity , after PR #431, can you use the prototype to solve the problem described in https://techport.nasa.gov/projects/116144 ? If not, what still miss?

[xingularity]

Hi @yungyuc and @j8xixo12

I think this idea containing two parts, one is to build a container (or we can call it time series DataFrame) to preserve and process time series data. The other one is to perform Fourier transform. Regarding the container part, I had a short discussion with you on Saturday. I think I can summarize below.

• Create a Time Series DataFrame to store time series data. This data frame will have an index column which stores the timestamp/epoch, and multiple columns to store time series data corresponding to these timestamps.
• This DataFrame should have capabilities to append one row or multiple rows at a time. The usage scenario of this feature would be like sensors getting one or multiple new records and these new data were appended onto this data frame (memory management consideration needed).
• Analysis capabilities. For the most simple one, user can extract one arbitrary column as a single time series, and then this time series can be passed into some Fourier transform (or other time series analysis) functions to perform analysis.
  • [Extended Idea] Do we need another container for time series data structure? It should contain one time series data and an index column (time stamp). If the extracted series is only for Fourier analysis, then we don't need it, just using SimpleArray is sufficient. Since transformed series is in frequency domain and there will be no time stamp.
• [Extended Idea] Does this Time Series DataFrame need to have abilities to attach one more series as a new column?
  • If this DataFrame can attach new columns, then we need to consider merging two time series. If we need to consider attach other time series onto an existing DataFrame. Then we need to consider merging two different index columns into one (The attaching series may not have exact same time stamps as the original data frame), and deal with null values in the DataFrame (there is null data in a column on certain time stamp).

[yungyuc]

Right. In a prototype we do not emphasize performance (runtime and memory). Profiling will be needed after the prototype proves the correctness.

[xingularity]

It is too quick to delve into the implementation plan. I agree we will need a container, but it is obvious that processing data needs a container.

It is good to dissect the requirements into the bullets. But the very first thing to do is to make a prototype for the real model problem listed in #380 (reply in thread) . It will give us a concrete sense of what should be done in the first place.

@xingularity would you please make an estimate for when you can work out some code processing the NASA dataset?

I am not trying to quickly dive into the implementation plan. I just tried to record down everything we have discussed and thoughts to avoid losing them. We can do a prototype first and improve them gradually. Good reminder though, thanks!

[yungyuc]

We are aligned.

[j8xixo12]

It seems like merging time series or append column data into DataFrame could be useful in financial field. However implementation this feature might more complex. I think we can lower its priority during the prototyping.

[yungyuc]

We should start small and grow. A key point for initial development is to quickly understand the problem. Python shines for this kind of prototyping. The architecture of modmesh has been curated to be productive to take Python prototypes into C++ and connect them together, and enables very fast learning through doing.

[xingularity]

Hi @yungyuc , @tigercosmos and @j8xixo12

Thanks yyc for merging my early prototype. However, the work has not done yet. Based on my f2f discussion with Chun-Hsu, he needs that this DataFrame to provide data in correct sequence. Current prototype implementation only reads text and organizes data in a columnar format, it does not guarantee the sequence of the data. So the next step should be implementing a sorting algorithm to check/guarantee the data sequence. There are two ways to put in the sorting algorithm I can think of, one is to sort data when the data are read or inserted, another one is to sort data only when they are retrieved. Any thoughts on this?

Collected backlog so far

• use mmap when reading large files. - yyc
• Index sequence should accept floating points. - tigercosmos
• The code will be more readable to have member data listed in init(). - yyc

[yungyuc]

Thanks, @xingularity . Could you please create an issue to track the data ordering requirement? It may need clarification.

mmap is a distant item. We do not need to consider it before the data frame code is moved to C++.