DCSF: Deep Convolutional Set Functions for Classification of Asynchronous Time Series

In AsTS, each variable is observed independently, and the observation times are not synchronized, making the length of each channel different from others. Hence, it poses a significant challenge to deep learning models that operate on equal channel lengths. In order to train a deep learning model for AsTS data, we need to model all the channels in to equal length, or to a fixed dimensional vector. For this, there have been various methodologies that are studied in the literature.

The main disadvantage of the triplets’ set is that the time dependency in the series is not explicit, but has to be learned using the time information provided which is difficult, and fails when the sub-sequences carry more information of classification rather than individual observations. On the other hand, using separate encoders for every channel in list of channels approach is computationally expensive. Imputation methods are not desired when the missing observations carry the information of their own. Also, asynchronous time series come with high sparsity (our datasets have a sparsity of around 85%), and imputing such data leads to bad predictions (we show empirically in Section IV. Hence, we propose a representation of AsTS (trivial to extend to any MTS) as a set of channels as shown in Figure 2(f).

In contrast to the set representation in [9], where instances are represented as sets of single time slices, we represent AsTS as set of channels, each channel being an irregular one-dimensional time series. Then the classification of AsTS becomes a classification of a set of channels.

An AsTS $X$ with $D$ many channels can be represented as a set of $D$ many elements: $X=\{s_1,...,s_d,...,s_D\}$. Each of its element is a tuple $s_d=(M_d,V_d,T_d)$ of i) a channel indicator $M_d\in N^P$ for channel $d\in \{1,2,...,D\}$, where $P$ is the dimension of the channel indicator, e.g., $P=D$ for one hot encoding, $P={\log }_{2}D$ for binary encoding, or $P=1$ for nominal encoding, ii) an observation vector $V_d\in R^{\ast }$ consisting of the values observed in channel $d$; and iii) a time vector $T_d\in {R^{+}}^{\ast }$ containing the corresponding observation times of the values in $V_d$. We construct $T_d$ in a monotonically increasing fashion to preserve the causal properties of the series. In our representation, the lengths of the vectors $T_d$ and $V_d$ are always equal ($|T_d|=|V_d|$). We denote the domain of these set elements by $\Omega :=N^P\times R^{\ast }\times {R^{+}}^{\ast }$ with $P\in N$.

A time series $X$ is considered to be fully observed if and only if $\forall d,e\in \{1,2,...,D\},T_d=T_e$, meaning all the variables are observed at any given time point. Similarly, a time series $X$ is considered to be sparse, if there exists at least two variables $d$ and $e$ such that $T_d\ne T_e$. Note that our representation covers both, TSMV and AsTS, although they are qualitatively different. In Figure 2 (f), we demonstrate our set representation for the AsTS of Figure 2 (a).

We define $F$ as a function that operates on set elements, and its output is invariant to the order in which the set elements are presented. The proposed model has three different modalities: 1) Encoder ($E$), 2) Aggregator ($A$) and 3) Classifier ($C$). We formulate $F$ for an AsTS instance $X$ with $D$ many set elements (channels) as follows:

where $E:\Omega \to R^K$, $A:R^{K\times D}\to R^K$and $C:R^K\to R^L$ where $K\in N^{+}$ is the dimensionality of the latent representation of a set element $s_d\in X$. The overall architecture of the proposed model is presented in Figure 3.

Our encoder $E$ takes a set element as input, and provides a fixed dimensional representation $R^K$. Since the length of every channel varies in AsTS, we require a model whose parameters do not depend on the length of the series presented to it. The widely used parameterized model architectures for this task are Recurrent Neural Networks (RNN), Attention Network (AtN), and Convolutional Neural Networks (CNNs). While RNN and AtN are useful for time series forecasting, state-of-the-art models for time series classification are built upon CNNs[19]. Hence, we choose a CNN based model as an encoder, and extract useful information from the channel observations. The proposed encoder architecture is presented in Figure 4. It comprises a series of $l$ many residual blocks where each residual block consists of three convolutional layers with a ReLU activation function following each layer. In order to achieve fixed representation, one needs to aggregate the latent representation over time. For this, we employ global average pooling, which is widely used in CNNs based time series classification [19] models.

As mentioned in Section II, our set elements are tuples. Since our encoder cannot directly operate on tuples, we convert our set element $s_d$ into a multi-dimensional array $s_d\in R^{(P+2)\times |T_d|}$. $s_d^{:,t}=[M_d,V_d^t,T_d^t],t\in \{1,...,|T_d|\}$ consists of 1) modality indicator of channel $d$, ($M_d\in N^P$), 2) observed value at $t$ ($V_d^t$), and 3) its observation time ($T_d^t$). Our encoder $E$ takes $s_d$ as an input, and outputs $K$ dimensional fixed representation of $s_d$. We consider $K$ to be large enough in order to memorize the temporal locations of the signals present in $s_d$. In [24], Wagstaff et. al., shows that universal function representation is guaranteed when $K\ge {max}_i|X_i|$ meaning, the dimension of latent embedding shall be greater than or equal to the number of set elements in $X$. We carefully choose $K$ that follows the criteria using hyperparameter search.

The proposed encoder share some similarity with the ResNET architecture provided in [19] in terms of using residual blocks, number of kernels, and kernel lengths in CNNs used in the residual block. However, a) we do not have a fixed number of residual blocks meaning, the number of residual blocks $l$ is a hyper parameter, b) we do not use batch normalization after each residual block as it is hindering the performance of DCSF (refer Section IV-G).

Once the embeddings for each $s_d\in X$ are computed, we aggregate those embeddings $E\left(s_d\right)$ using the aggregator function $A$. There have been various kinds of aggregations provided in the literature such as mean aggregation [29, 5], sum aggregation [29, 5], and attention based aggregation [9]. In this work, we sum all the embeddings in order to aggregate them as shown in Equation 2 and represent the aggregated value with $z$.

After aggregation of the embeddings, we receive a vector representation $z\in R^K$ of the multivariate time series. Then, one can use any model that is used for vector data classification. In this work, we use a fully connected artificial neural network.

As mentioned earlier, SEFT-ATTN presented in [9] also uses a deep set architecture for classifying time series. However, SEFT-ATTN uses triplet set modeling of time series, and those triplets are given as inputs to the encoder; an attention mechanism was used for aggregating the latent embeddings of the set elements. We observed that operating on individual observations has the disadvantage of losing sequence information even after providing time in the triplet. On the other hand, operating on sequences rather than observations not only preserves the causality of the sequence but also eases the learning process.

For the online classification we need to predict the class label at every time point. Hence, we replace normal convolutions with causal convolutions, such that the current observation cannot look ahead into the future. Because we need to classify at every time point, we cannot implement global average pooling as it is implemented on entire sequence. Instead, we use causal average pooling ($CAP$) where at every time point we average the observations made until then, and aggregate those embeddings followed by classification using dense layer. The causal average pooling at time point $\tau$ for an array $E$ is given by

We consider the parameters of the encoder $E$ and the classifier $C$ as $\theta$ and $\varphi$, respectively. Because we perform sum aggregation, we do not have any parameters for $A$. We follow [29], and train the model with all the channels present in time series. We optimize the following cost function:

where, $L$ is the loss function which is binary cross entropy for binary classification and Sigmoid cross entropy for multi-class classification tasks.

Following the literature that deals with AsTSC problems, we chose three medical and one Human Activity Recognition dataset that contain asynchronous measurements. Basic statistics of the datasets are presented in Table LABEL:tab:data_stat.

The PhysioNet Challenge 2012 dataset [22, 6] comprises asynchronous time series data extracted from the records of patients admitted to ICU. Up to 37 variables were measured for the first 48 hours after the patient’s admission, and all the time series have general descriptors like height, weight, age, and gender. The aim of this dataset is to predict whether a person dies in the hospital. The dataset comprises three different splits (a, b, and c) with 4000 observations each. Here, we combine all the splits, round the observed time points to the nearest minute, and split the 12,000 labeled observations randomly into 80% for training and 20% for testing.

Baseline models [21, 18] used only a part of the full dataset for the experiments. Hence, we would like to see how the model performs for a small part of the dataset. For this, we have taken split-c of the physionet dataset. It is also an imbalanced dataset with 4000 samples. Again, we randomly split $80\%$ of the dataset into training, and $20\%$ for testing.

The MIMIC dataset [10] also comprises records of ICU patients at Beth Israel Deaconess Medical Center. All the time series contain asynchronous measurements. Again, the task is to predict patient’s mortality after the first 48 hours of ICU admission. We follow the procedures of [9, 7] for splitting the dataset that contains around 21,000 stays with 12 physiological variables being observed.

We use the human activity dataset for the online classification of Asynchronous Time Series. It contains the AsTS collected from five individuals wearing 3D sensors positioned in the belt, chest, and ankles while performing various tasks like sitting, laying, walking, etc. The dataset comprises $6,554$ time series with $12$ channels, and channel consists of $50$ time points. We follow the data preprocessing steps mentioned in [21, 18]. The task is to classify every time point in the series into $11$ classes (activities).

We use the following baseline models for the comparison.

We randomly split off $20\%$ of training data for validation, which is used for hyperparameter search and early stopping. For training, we use Adam optimizer with learning rate chosen from $[0.001,0.00001]$ and the batch size from $\{32,64,128\}$. We also consider normalizing the time series as a hyperparameter because, for some datasets, normalization of the series before inputting to the models provides better validation results. We set the embedding length $K=128$ in order to follow the universal function approximation for set functions presented in [24].

The hyper parameters searched for all the competing models are provided in the supplementary material.

We randomly generate $10$ sets of hyperparameters for each of the competing models, and chose the setup with the best evaluation metric on the validation dataset. The models with selected hyperparameters are run independently for $5$ times. For the medical datasets, we use area under receiver operating characteristic (AUROC) as the evaluation metric, because those datasets are heavily imbalanced. For the Activity dataset, we use accuracy as the evaluation metric.

We present experimental results for classification of full AsTS using the medical datasets in Table I. The presented run time is the one that is taken by the model with best chosen hyperparameters. It could be possible that different hyperparameters may need different run times, but we use the current set up in order to be fair for all the models.

We can observe that our proposed model outperforms all the baselines in the Physionet and mini-Physionet datasets. Whereas, in the MIMIC dataset, our proposed model performs on par with the GRU-D model while outperforming all the remaining baselines. However, the run times of our proposed model are less than the GRU-D by an order of magnitude for the MIMIC dataset. Note that the second-best performing model is the Interpolation Networks, and the proposed model improves the AUROC of Interpolation Networks by around $2\%$ in the mini-Physionet and MIMIC datasets. The Latent-ODE model could be slightly underperforming, because we run the dataset with the default hyperparameters provided in [18]. This is because of high computational complexity and run times. Especially, for the MIMIC and Physionet datasets, it took around an hour for each epoch, which is also observed in [9].

The results of the Physionet and MIMIC dataset are similar to that of the results published in [9], whereas the results on mini-Physionet dataset deviate significantly from the published results in [21]. Our results, in most cases, are better than the published ones because, unlike the procedure in [21, 18], we balance both the positive and negative class time series while sampling a batch for training in order to avoid bias towards the negative samples. Also, we could not reproduce the results published in [21] with the provided experimental setup and hyperparameters. We emailed the authors regarding the issue, and did not receive any information from them. The same issue has been raised by multiple people regarding their inability to reproduce the results for both Physionet and MIMIC datasets. ¹¹1https://github.com/reml-lab/mTAN/issues.

The Activity dataset is an online classification dataset where one needs to provide class labels at every time point. While this is similar to segmentation in the time domain, since many researchers have explored this task, we want to see how the proposed model works in this scenario. From Table II, it can be observed that the proposed model outperforms all the baselines significantly, and the only model that performs close to ours is mTAN [21]. Since the source codes are not available for multiple baseline models, we did not run the experiments for the published baseline models, but took the results from [21]. Also, we could not compare the model in terms of run time because, the information is not available in the published works [21].

One important observation is that the Activity dataset does not heavily depend on the previous observations while predicting the class label at the current time point. CNN-based models under perform when the kernel lengths are larger than $1$. Hence, we set the kernel length of CNNs to $1$ for the CNN-based models, namely RESNET, RESNET-forward, and our DCSF.

Since the medical datasets were used by all the baseline models in their works, it is possible to have a model bias towards those datasets. Hence, we additionally use two RMTS datasets with synthetic AsTS setup for the comparison.

We consider two RMTS datasets, namely LSST and Phoneme Spectra [19], and induce artificial sparsity to them. These are among the $4$ largest datasets used in [19]. The largest dataset Pen Digits, has very few time stamps (8) and channels (2) in a series. In the second largest dataset Face Detection, the accuracy of the best classifier in asynchronous setup is close to the default rate ($50\%$) of the dataset. Hence, we choose third and fourth largest datasets.

We use the following setup to generate asynchronous datasets from RMTS. Because in an asynchronous dataset variables are observed independently, we assume that at every time stamp only one variable is observed. Hence, in the synthetic setup we choose one variable uniformly at random for a given time point. The total number of observations is equal to the length of the RMTS. For any two different samples, the observation times of a channel may not be consistent.

The experimental results on the datasets with synthetic asynchronous setup are presented in Table III. Accuracy is the evaluation metric. For Phoneme Spectra, proposed model outperforms all the baseline models by a significant margin; the accuracy improvement is around $15\%$ compared to the next best model, Interpolation Networks. For the LSST data, imputation based models, the ResNET-forward, and the RESNET perform slightly better than the DCSF. SEFT-ATTN has an accuracy gain over DCSF, but the difference is not statistically significant.

Since we separate the channels before inputting to the encoder, one might assume that the model is not learning from channel interactions, but just yields the accuracy of a channel that contributes maximum to the output. Hence, we perform an experiment by providing only a single channel to the model, and compared it with the proposed model. We ran the experiments with all the channels, and present the results for the best one in Table IV. From the experimental results, we can conclude that the proposed model does learn from the cross channel interactions as it heavily outperforms the model that learns from only a single channel. One important observation is: for the mini-Physionet dataset, learning a single channel provides directly better results compared to many state-of-the-art models.

In order to demonstrate that the aggregation after computing the latent embeddings is useful in learning the model, we compare the proposed model with an ensemble model. For this, we learn $D$ models for $D$ many channels present in the AsTS data. Each model takes the series from a single variable, and is trained for the classification. While testing, we take the average of encodings of the penultimate layer (while Softmax is the last layer), and compute the class label.

We present the results in Table IV. We observe that the ensemble model provides better results than the model that learns with a single channel, except for the Physionet datasets. The reason for this phenomena is, the channels other than the best one do not provide useful result on their own. Hence, the linear combination used for ensemble, suppress the result of the best channel. Moreover, the proposed model outperforms the ensemble model by a significant margin, showing the advantage of aggregating the embeddings after the encoder.

The purpose of having time information as a dimension in the encoder input is to capture the time dependent channel interactions. Because we are encoding the channels separately, one might doubt that the proposed model is not capturing the channel interactions. In order to show that the time information provided in the encoder input serves the purpose, we performed an ablation study with a toy dataset, where the time information, or channel interactions over time is absolutely necessary for classifying it.

In the toy dataset, each time series has two channels of length $T$, and they are initialized with $0$ for all the time steps. For positive samples, we sample a time location $t\in \{0,1,...,T\}$, and substitute $1$ for both the channels at time point $t$. Where as, for negative samples, we sample two locations $t_1,t_2\in \{0,1,...,T\}(t_1\ne t_2)$ and replaced $0$ with $1$ at $t_1$ for channel $1$, and $t_2$ for channel $2$ as shown in Figure 5. For this experiment, we set $T=20$, and induced sparsity of $10\%$, $50\%$, and $90\%$ without altering the signals.

We conducted experiments on the proposed model with and without time information, and the results are presented in Table V. We can observe that the proposed model performs better with time information (DCSF) compared to the one without (DCSF w$\setminus$o time). It shows that the proposed model can learn the channel correlation towards the output when explicit time information is provided to it. An interesting observation is, the proposed model without time can learn from the relative position of the signal when the data is less sparse. With an increase in sparsity, the performance reduces. We can see that for $10\%$ and $50\%$ sparsity, the model achieves $91\%$ and $76\%$ accuracy, respectively.

Here, we study multiple time embeddings in order to embed time information into the model.

Absolute time information (TE-1) Here, we provide the normalized absolute time values directly to the model, and the embeddings are learned in the latent layers, which is the proposed modeling.

Time differences (TE-2) We consider the distance between the current observation and the previous one in time instead of absolute time information.

Sinusoidal positional embedding (TE-3) Here, we implement sinusoidal positional embedding provided in [9], which is a variant of the one presented in [23].

No Time embedding (TE-4) It is interesting to see how the models work when there is no information about time is given to it. Hence, we did not provide any explicit time information to the model.

Results on the validation dataset are provided in Table VI, shows that the absolute time values are more helpful compared to all the other embeddings. Another observation is that, model provides good accuracy even with no explicit information. This makes us think that the datasets we are using may not have significant time dependency. However, when there is a time dependency, our model excels as shown in the previous section.

Here, we study multiple model configurations and the results for the MIMIC and mini-Physionet are presented in Table VI. It can be seen that the batch normalization (DCSF+BN) is not helpful in the proposed model. We compare DCSF with a model where each channel is trained with an independent encoder (DCSF+DE). When we use $D$ many encoders for $D$ many channels, the channel interactions entirely depend on the aggregated embeddings. In the DCSF, because of the shared parameters, the encoder also receive the channel interactions, and with very few parameters.

As mentioned earlier, both time series with missing values and asynchronous time series can be modelled in a unified manner. Hence, it would be interesting to see the performance of the competing models in the missing value setup. For this, we remove $p\%$ of the observations from the RMTS. As an example, if an RMTS has $5$ variables observed for a length of $50$, there will be $250$ observations. For every time series, we randomly remove $p\%$ of those $250$ observations in order to create TSMV.

We present the results on the LSST, and Phoneme Spectra with $p\in \{10,50,90\}$ in Table VII. With accuracy being the evaluation metric, the best results are presented in bold and the second-best in italics. We observed that the proposed model performs better among all the competing models. The next best model is ResNET-forward which is an imputation model. For Phoneme Spectra, with $p=50\%$, DCSF provides $29\%$ better accuracy compared to baseline models. Results show that, compared to the medical datasets, the lifts in the synthetic setup are significant, because for the medical datasets, the required information for classification is provided, but the observations are not synchronized. Whereas for the synthetic setup, we randomly remove the values, making it difficult for the models to learn. The results indicate that the proposed DCSF can be used for both AsTS and TSMV datasets.

Again, for the setting of $90\%$ sparsity for LSST dataset, DCSF performs worse than ResNET, ResNET-forward and SEFT-ATTN as seen in the synthetic asynchronous setup for the LSST dataset. We observe that with extreme sparsity, in the LSST dataset, the sequence information is destroyed and applying convolutions on the observations with large time gap is not useful.