A data preprocessing workflow is proposed to address key challenges in rockburst data analysis,including dimensionality differences among various sample features,variations in data values within the same feature,missi...A data preprocessing workflow is proposed to address key challenges in rockburst data analysis,including dimensionality differences among various sample features,variations in data values within the same feature,missing data,poor data consistency,and sample class imbalance.The workflow is divided into four steps.Each step introduces multiple algorithms,which are sequentially combined according to the order of the four steps.Then,these coupled algorithms are utilized to preprocess the rockburst data set.The rockburst data set contains 459 samples,and the maximum tangential stress(MTS),the uniaxial compressive strength(UCS),the uniaxial tensile strength(UTS),the elastic strain energy index(WET),the rock stress factor(SCF),and the rock brittleness coefficient(B)are selected as the feature parameters.Subsequently,three architectures,Deep Neural Network(DNN),Convolutional Neural Network(CNN),and Recurrent Neural Network(RNN),are used to evaluate the data sets processed by different coupled algorithms.The hyperband algorithm is introduced to optimize the hyperparameters of the RNN model,and the prediction accuracy of different architectures is compared between the RNN model with dense layers and without dense layers.Finally,a rockburst prediction model based on data preprocessing and the Hyperband-DNN model is developed.The prediction results show that data preprocessing can significantly improve the model prediction accuracy;the model architecture with the highest prediction accuracy can be found quickly using the hyperband algorithm;and adding the dense layer can improve the stability and prediction accuracy of the model.展开更多
The big data generated by tunnel boring machines(TBMs)are widely used to reveal complex rock-machine interactions by machine learning(ML)algorithms.Data preprocessing plays a crucial role in improving ML accuracy.For ...The big data generated by tunnel boring machines(TBMs)are widely used to reveal complex rock-machine interactions by machine learning(ML)algorithms.Data preprocessing plays a crucial role in improving ML accuracy.For this,a TBM big data preprocessing method in ML was proposed in the present study.It emphasized the accurate division of TBM tunneling cycle and the optimization method of feature extraction.Based on the data collected from a TBM water conveyance tunnel in China,its effectiveness was demonstrated by application in predicting TBM performance.Firstly,the Score-Kneedle(S-K)method was proposed to divide a TBM tunneling cycle into five phases.Conducted on 500 TBM tunneling cycles,the S-K method accurately divided all five phases in 458 cycles(accuracy of 91.6%),which is superior to the conventional duration division method(accuracy of 74.2%).Additionally,the S-K method accurately divided the stable phase in 493 cycles(accuracy of 98.6%),which is superior to two state-of-the-art division methods,namely the histogram discriminant method(accuracy of 94.6%)and the cumulative sum change point detection method(accuracy of 92.8%).Secondly,features were extracted from the divided phases.Specifically,TBM tunneling resistances were extracted from the free rotating phase and free advancing phase.The resistances were subtracted from the total forces to represent the true rock-fragmentation forces.The secant slope and the mean value were extracted as features of the increasing phase and stable phase,respectively.Finally,an ML model integrating a deep neural network and genetic algorithm(GA-DNN)was established to learn the preprocessed data.The GA-DNN used 6 secant slope features extracted from the increasing phase to predict the mean field penetration index(FPI)and torque penetration index(TPI)in the stable phase,guiding TBM drivers to make better decisions in advance.The results indicate that the proposed TBM big data preprocessing method can improve prediction accuracy significantly(improving R2s of TPI and FPI on the test dataset from 0.7716 to 0.9178 and from 0.7479 to 0.8842,respectively).展开更多
The Moon-based Ultraviolet Telescope (MUVT) is one of the payloads on the Chang'e-3 (CE-3) lunar lander. Because of the advantages of having no at- mospheric disturbances and the slow rotation of the Moon, we can...The Moon-based Ultraviolet Telescope (MUVT) is one of the payloads on the Chang'e-3 (CE-3) lunar lander. Because of the advantages of having no at- mospheric disturbances and the slow rotation of the Moon, we can make long-term continuous observations of a series of important celestial objects in the near ultra- violet band (245-340 nm), and perform a sky survey of selected areas, which can- not be completed on Earth. We can find characteristic changes in celestial brightness with time by analyzing image data from the MUVT, and deduce the radiation mech- anism and physical properties of these celestial objects after comparing with a phys- ical model. In order to explain the scientific purposes of MUVT, this article analyzes the preprocessing of MUVT image data and makes a preliminary evaluation of data quality. The results demonstrate that the methods used for data collection and prepro- cessing are effective, and the Level 2A and 2B image data satisfy the requirements of follow-up scientific researches.展开更多
Due to the frequent changes of wind speed and wind direction,the accuracy of wind turbine(WT)power prediction using traditional data preprocessing method is low.This paper proposes a data preprocessing method which co...Due to the frequent changes of wind speed and wind direction,the accuracy of wind turbine(WT)power prediction using traditional data preprocessing method is low.This paper proposes a data preprocessing method which combines POT with DBSCAN(POT-DBSCAN)to improve the prediction efficiency of wind power prediction model.Firstly,according to the data of WT in the normal operation condition,the power prediction model ofWT is established based on the Particle Swarm Optimization(PSO)Arithmetic which is combined with the BP Neural Network(PSO-BP).Secondly,the wind-power data obtained from the supervisory control and data acquisition(SCADA)system is preprocessed by the POT-DBSCAN method.Then,the power prediction of the preprocessed data is carried out by PSO-BP model.Finally,the necessity of preprocessing is verified by the indexes.This case analysis shows that the prediction result of POT-DBSCAN preprocessing is better than that of the Quartile method.Therefore,the accuracy of data and prediction model can be improved by using this method.展开更多
Quantum Machine Learning(QML)techniques have been recently attracting massive interest.However reported applications usually employ synthetic or well-known datasets.One of these techniques based on using a hybrid appr...Quantum Machine Learning(QML)techniques have been recently attracting massive interest.However reported applications usually employ synthetic or well-known datasets.One of these techniques based on using a hybrid approach combining quantum and classic devices is the Variational Quantum Classifier(VQC),which development seems promising.Albeit being largely studied,VQC implementations for“real-world”datasets are still challenging on Noisy Intermediate Scale Quantum devices(NISQ).In this paper we propose a preprocessing pipeline based on Stokes parameters for data mapping.This pipeline enhances the prediction rates when applying VQC techniques,improving the feasibility of solving classification problems using NISQ devices.By including feature selection techniques and geometrical transformations,enhanced quantum state preparation is achieved.Also,a representation based on the Stokes parameters in the PoincaréSphere is possible for visualizing the data.Our results show that by using the proposed techniques we improve the classification score for the incidence of acute comorbid diseases in Type 2 Diabetes Mellitus patients.We used the implemented version of VQC available on IBM’s framework Qiskit,and obtained with two and three qubits an accuracy of 70%and 72%respectively.展开更多
Many classifiers and methods are proposed to deal with letter recognition problem. Among them, clustering is a widely used method. But only one time for clustering is not adequately. Here, we adopt data preprocessing ...Many classifiers and methods are proposed to deal with letter recognition problem. Among them, clustering is a widely used method. But only one time for clustering is not adequately. Here, we adopt data preprocessing and a re kernel clustering method to tackle the letter recognition problem. In order to validate effectiveness and efficiency of proposed method, we introduce re kernel clustering into Kernel Nearest Neighbor classification(KNN), Radial Basis Function Neural Network(RBFNN), and Support Vector Machine(SVM). Furthermore, we compare the difference between re kernel clustering and one time kernel clustering which is denoted as kernel clustering for short. Experimental results validate that re kernel clustering forms fewer and more feasible kernels and attain higher classification accuracy.展开更多
In order to reduce the risk of non-performing loans, losses, and improve the loan approval efficiency, it is necessary to establish an intelligent loan risk and approval prediction system. A hybrid deep learning model...In order to reduce the risk of non-performing loans, losses, and improve the loan approval efficiency, it is necessary to establish an intelligent loan risk and approval prediction system. A hybrid deep learning model with 1DCNN-attention network and the enhanced preprocessing techniques is proposed for loan approval prediction. Our proposed model consists of the enhanced data preprocessing and stacking of multiple hybrid modules. Initially, the enhanced data preprocessing techniques using a combination of methods such as standardization, SMOTE oversampling, feature construction, recursive feature elimination (RFE), information value (IV) and principal component analysis (PCA), which not only eliminates the effects of data jitter and non-equilibrium, but also removes redundant features while improving the representation of features. Subsequently, a hybrid module that combines a 1DCNN with an attention mechanism is proposed to extract local and global spatio-temporal features. Finally, the comprehensive experiments conducted validate that the proposed model surpasses state-of-the-art baseline models across various performance metrics, including accuracy, precision, recall, F1 score, and AUC. Our proposed model helps to automate the loan approval process and provides scientific guidance to financial institutions for loan risk control.展开更多
The futures trading market is an important part of the financial markets and soybeans are one of the most strategically important crops in the world.How to predict soybean future price is a challenging topic being stu...The futures trading market is an important part of the financial markets and soybeans are one of the most strategically important crops in the world.How to predict soybean future price is a challenging topic being studied by many researchers.This paper proposes a novel hybrid soybean future price prediction model which includes two stages of data preprocessing and deep learning prediction.In the data preprocessing stage,futures price series are decomposed into subsequences using the ICEEMDAN(improved complete ensemble empirical mode decomposition with adaptive noise)method.The Lempel-Ziv complexity determination method was then used to identify and reconstruct high-frequency subsequences.Finally,the high frequency component is decomposed secondarily using variational mode decomposition optimized by beluga whale optimization algorithm.In the deep learning prediction stage,a deep extreme learning machine optimized by the sparrow search algorithm was used to obtain the prediction results of all subseries and reconstructs them to obtain the final soybean future price prediction results.Based on the experimental results of soybean future price markets in China,Italy,and the United States,it was found that the hybrid method proposed provides superior performance in terms of prediction accuracy and robustness.展开更多
With the continuous expansion and increasing complexity of engineering projects, cost forecasting has become particularly critical in project management. To meet the demands of the big data era, this study utilizes ex...With the continuous expansion and increasing complexity of engineering projects, cost forecasting has become particularly critical in project management. To meet the demands of the big data era, this study utilizes extensive historical engineering data to examine the intrinsic relationships and fluctuation patterns of project costs, covering data collection, preprocessing, and key factor analysis. By employing cutting-edge algorithms for comprehensive comparison and intelligent identification of cost-influencing factors, an efficient cost forecasting methodology was developed, achieving accurate predictions of cost trends. Results demonstrate that this method exhibits high forecasting accuracy across various project scenarios, effectively capturing key cost variation characteristics and providing robust insights for budget preparation and risk control. Practical validation confirms that with sufficient data resources and rigorous analysis of critical factors, cost management precision can be significantly enhanced. Overall, this research not only establishes innovative technical approaches for cost forecasting but also advances the digital transformation of project management, offering substantial practical implications and broad application prospects for improving resource allocation efficiency and optimizing project decision-making.展开更多
This study examines the Big Data Collection and Preprocessing course at Anhui Institute of Information Engineering,implementing a hybrid teaching reform using the Bosi Smart Learning Platform.The proposed hybrid model...This study examines the Big Data Collection and Preprocessing course at Anhui Institute of Information Engineering,implementing a hybrid teaching reform using the Bosi Smart Learning Platform.The proposed hybrid model follows a“three-stage”and“two-subject”framework,incorporating a structured design for teaching content and assessment methods before,during,and after class.Practical results indicate that this approach significantly enhances teaching effectiveness and improves students’learning autonomy.展开更多
Liquid chromatography–mass spectrometry(LC–MS)has enabled the detection of thousands of metabolite features from a single biological sample that produces large and complex datasets.One of the key issues in LC–MS-ba...Liquid chromatography–mass spectrometry(LC–MS)has enabled the detection of thousands of metabolite features from a single biological sample that produces large and complex datasets.One of the key issues in LC–MS-based metabolomics is comprehensive and accurate analysis of enormous amount of data.Many free data preprocessing tools,such as XCMS,MZmine,MAVEN,and MetaboAnalyst,as well as commercial software,have been developed to facilitate data processing.However,researchers are challenged by the inevitable and unconquerable yields of numerous false-positive peaks,and human errors while manually removing such false peaks.Even with continuous improvements of data processing tools,there can still be many mistakes generated during data preprocessing.In addition,many data preprocessing software exist,and every tool has its own advantages and disadvantages.Thereby,a researcher needs to judge what kind of software or tools to choose that most suit their vendor proprietary formats and goal of downstream analysis.Here,we provided a brief introduction of the general steps of raw MS data processing,and properties of automated data processing tools.Then,characteristics of mainly free data preprocessing software were summarized for researchers’consideration in conducting metabolomics study.展开更多
Substantial advancements have been achieved in Tunnel Boring Machine(TBM)technology and monitoring systems,yet the presence of missing data impedes accurate analysis and interpretation of TBM monitoring results.This s...Substantial advancements have been achieved in Tunnel Boring Machine(TBM)technology and monitoring systems,yet the presence of missing data impedes accurate analysis and interpretation of TBM monitoring results.This study aims to investigate the issue of missing data in extensive TBM datasets.Through a comprehensive literature review,we analyze the mechanism of missing TBM data and compare different imputation methods,including statistical analysis and machine learning algorithms.We also examine the impact of various missing patterns and rates on the efficacy of these methods.Finally,we propose a dynamic interpolation strategy tailored for TBM engineering sites.The research results show that K-Nearest Neighbors(KNN)and Random Forest(RF)algorithms can achieve good interpolation results;As the missing rate increases,the interpolation effect of different methods will decrease;The interpolation effect of block missing is poor,followed by mixed missing,and the interpolation effect of sporadic missing is the best.On-site application results validate the proposed interpolation strategy's capability to achieve robust missing value interpolation effects,applicable in ML scenarios such as parameter optimization,attitude warning,and pressure prediction.These findings contribute to enhancing the efficiency of TBM missing data processing,offering more effective support for large-scale TBM monitoring datasets.展开更多
In Global Navigation Satellite Systems(GNSS),accurate and stable atomic clocks need to be equipped on satellites to ensure reliable,high-accuracy positioning,navigation,and timing services.It is essential to continuou...In Global Navigation Satellite Systems(GNSS),accurate and stable atomic clocks need to be equipped on satellites to ensure reliable,high-accuracy positioning,navigation,and timing services.It is essential to continuously monitor the behavior of satellite clocks in space and predict satellite clock corrections for real-time GNSS applications,especially for precise point positioning.Some commercial software is available for clock characterization,but special attention has to be paid when referring to satellite clocks,the analysis and prediction of which may be complicated by outliers,data gaps,and periodic fluctuations in onboard clock data,not often encountered by clock data from a timekeeping laboratory.The typical approaches for clock characterization and prediction currently employed in a timekeeping laboratory are therefore unsuitable for clock applications in space.We present a software package developed in MATLAB at the National Time Service Center,Chinese Academy of Sciences,intended for satellite clock characterization and prediction.The software package includes many subroutines and functionalities of particular interest in characterizing and predicting clock behavior in space,such as dynamic frequency stability evaluation,periodic fluctuation analysis,and multi-step prediction of clock signals.The software package allows handling of satellite clock data directly from Receiver Independent Exchange Format clock files widely used in GNSS,facilitating quick characterization analysis and prediction of satellite clocks,with graphically visualized output.展开更多
It is difficult to detect the anomalies whose matching relationship among some data attributes is very different from others’ in a dataset. Aiming at this problem, an approach based on wavelet analysis for detecting ...It is difficult to detect the anomalies whose matching relationship among some data attributes is very different from others’ in a dataset. Aiming at this problem, an approach based on wavelet analysis for detecting and amending anomalous samples was proposed. Taking full advantage of wavelet analysis’ properties of multi-resolution and local analysis, this approach is able to detect and amend anomalous samples effectively. To realize the rapid numeric computation of wavelet translation for a discrete sequence, a modified algorithm based on Newton-Cores formula was also proposed. The experimental result shows that the approach is feasible with good result and good practicality.展开更多
Artificial intelligence(AI)relies on data and algorithms.State-of-the-art(SOTA)AI smart algorithms have been developed to improve the performance of AI-oriented structures.However,model-centric approaches are limited ...Artificial intelligence(AI)relies on data and algorithms.State-of-the-art(SOTA)AI smart algorithms have been developed to improve the performance of AI-oriented structures.However,model-centric approaches are limited by the absence of high-quality data.Data-centric AI is an emerging approach for solving machine learning(ML)problems.It is a collection of various data manipulation techniques that allow ML practitioners to systematically improve the quality of the data used in an ML pipeline.However,data-centric AI approaches are not well documented.Researchers have conducted various experiments without a clear set of guidelines.This survey highlights six major data-centric AI aspects that researchers are already using to intentionally or unintentionally improve the quality of AI systems.These include big data quality assessment,data preprocessing,transfer learning,semi-supervised learning,machine learning operations(MLOps),and the effect of adding more data.In addition,it highlights recent data-centric techniques adopted by ML practitioners.We addressed how adding data might harm datasets and how HoloClean can be used to restore and clean them.Finally,we discuss the causes of technical debt in AI.Technical debt builds up when software design and implementation decisions run into“or outright collide with”business goals and timelines.This survey lays the groundwork for future data-centric AI discussions by summarizing various data-centric approaches.展开更多
The tendency toward achieving more sustainable and green buildings turned several passive buildings into more dynamic ones.Mosques are the type of buildings that have a unique energy usage pattern.Nevertheless,these t...The tendency toward achieving more sustainable and green buildings turned several passive buildings into more dynamic ones.Mosques are the type of buildings that have a unique energy usage pattern.Nevertheless,these types of buildings have minimal consideration in the ongoing energy efficiency applications.This is due to the unpredictability in the electrical consumption of the mosques affecting the stability of the distribution networks.Therefore,this study addresses this issue by developing a framework for a short-term electricity load forecast for a mosque load located in Riyadh,Saudi Arabia.In this study,and by harvesting the load consumption of the mosque and meteorological datasets,the performance of four forecasting algorithms is investigated,namely Artificial Neural Network and Support Vector Regression(SVR)based on three kernel functions:Radial Basis(RB),Polynomial,and Linear.In addition,this research work examines the impact of 13 different combinations of input attributes since selecting the optimal features has a major influence on yielding precise forecasting outcomes.For the mosque load,the(SVR-RB)with eleven features appeared to be the best forecasting model with the lowest forecasting errors metrics giving RMSE,nRMSE,MAE,and nMAE values of 4.207 kW,2.522%,2.938 kW,and 1.761%,respectively.展开更多
In general,the material properties,loads,resistance of the prestressed concrete continuous rigid frame bridge in different construction stages are time-varying.So,it is essential to monitor the internal force state wh...In general,the material properties,loads,resistance of the prestressed concrete continuous rigid frame bridge in different construction stages are time-varying.So,it is essential to monitor the internal force state when the bridge is in construction.Among them,how to assess the safety is one of the challenges.As the continuous monitoring over a long-term period can increase the reliability of the assessment,so,based on a large number of monitored strain data collected from the structural health monitoring system(SHMS)during construction,a calculation method of the punctiform time-varying reliability is proposed in this paper to evaluate the stress state of this type bridge in cantilever construction stage by using the basic reliability theory.At the same time,the optimal stress distribution function in the bridge mid-span base plate is determined when the bridge is closed.This method can provide basis and direction for the internal force control of this type bridge in construction process.So,it can reduce the bridge safety and quality accidents in construction stages.展开更多
Predicting NOxin the sintering process of iron ore powder in advance was helpful to adjust the denitrification process in time.Taking NOxin the sintering process of iron ore powder as the object,the boxplot,empi...Predicting NOxin the sintering process of iron ore powder in advance was helpful to adjust the denitrification process in time.Taking NOxin the sintering process of iron ore powder as the object,the boxplot,empirical mode decomposition algorithm,Pearson correlation coefficient,maximum information coefficient and other methods were used to preprocess the sintering data and naive Bayes classification algorithm was used to identify the sintering conditions.The regression prediction model with high accuracy and good stability was selected as the sub-model for different sintering conditions,and the sub-models were combined into an integrated prediction model.Based on actual operational data,the approach proved the superiority and effectiveness of the developed model in predicting NOx,yielding an accuracy of 96.17%and an absolute error of 5.56,and thereby providing valuable foresight for on-site sintering operations.展开更多
Accurate prediction of inorganic nitrogen concentrations is critical for assessing water quality and supporting sustainable aquatic ecosystem management.Data mining algorithms have demonstrated significant potential i...Accurate prediction of inorganic nitrogen concentrations is critical for assessing water quality and supporting sustainable aquatic ecosystem management.Data mining algorithms have demonstrated significant potential in inorganic nitrogen prediction.This review provides a systematic examination of data mining algorithms applied to inorganic nitrogen prediction for aquatic environments,focusing on the complete modeling pipeline from data preparation to predictive algorithm deployment.Essential data preparation techniques involving data acquisition,data cleaning,dimensionality reduction,and dataset partitioning are described in detail.Then,the mechanisms,techniques,and applications are critically evaluated according to the following five categories of data mining algorithms:machine learning models,deep learning models,tree-based models,fuzzy neural networks,and hybrid models.FNN and deep learning models yield top-tier accuracy(mean R2>0.90).In addition,hybrid models integrating LSTM have emerged as the most promising and prevalent framework,driving advancements in prediction precision.Despite significant progress,the field continues to face challenges such as insufficient data quality,limited model generalizability,and a lack of interpretability.Integrating multimodal data,developing lightweight real-time prediction systems,combining explainable artificial intelligence with physics-informed constraints,and strengthening uncertainty quantification will strongly advance inorganic nitrogen prediction models toward more accurate,reliable,and practical decision-support tools.This review provides a systematic reference for researchers and practitioners.展开更多
Enhancing the accuracy of real-time ship roll prediction is crucial for maritime safety and operational efficiency.To address the challenge of accurately predicting the ship roll status with nonlinear time-varying dyn...Enhancing the accuracy of real-time ship roll prediction is crucial for maritime safety and operational efficiency.To address the challenge of accurately predicting the ship roll status with nonlinear time-varying dynamic characteristics,a real-time ship roll prediction scheme is proposed on the basis of a data preprocessing strategy and a novel stochastic trainer-based feedforward neural network.The sliding data window serves as a ship time-varying dynamic observer to enhance model prediction stability.The variational mode decomposition method extracts effective information on ship roll motion and reduces the non-stationary characteristics of the series.The energy entropy method reconstructs the mode components into high-frequency,medium-frequency,and low-frequency series to reduce model complexity.An improved black widow optimization algorithm trainer-based feedforward neural network with enhanced local optimal avoidance predicts the high-frequency component,enabling accurate tracking of abrupt signals.Additionally,the deterministic algorithm trainer-based neural network,characterized by rapid processing speed,predicts the remaining two mode components.Thus,real-time ship roll forecasting can be achieved through the reconstruction of mode component prediction results.The feasibility and effectiveness of the proposed hybrid prediction scheme for ship roll motion are demonstrated through the measured data of a full-scale ship trial.The proposed prediction scheme achieves real-time ship roll prediction with superior prediction accuracy.展开更多
基金Educational Commission of Hubei Province of China,Grant/Award Number:T2020005Hubei Provincial Key Laboratory of Construction and Management in Hydropower Engineering,Grant/Award Number:2023KSD04+2 种基金Natural Science Research of Jiangsu Higher Education Institutions of China,Grant/Award Number:21KJB580001National Natural Science Foundation of China,Grant/Award Numbers:52209162,52379128Natural Science Foundation of Hubei Province of China,Grant/Award Numbers:2023AFA048,2023AFB657。
摘要A data preprocessing workflow is proposed to address key challenges in rockburst data analysis,including dimensionality differences among various sample features,variations in data values within the same feature,missing data,poor data consistency,and sample class imbalance.The workflow is divided into four steps.Each step introduces multiple algorithms,which are sequentially combined according to the order of the four steps.Then,these coupled algorithms are utilized to preprocess the rockburst data set.The rockburst data set contains 459 samples,and the maximum tangential stress(MTS),the uniaxial compressive strength(UCS),the uniaxial tensile strength(UTS),the elastic strain energy index(WET),the rock stress factor(SCF),and the rock brittleness coefficient(B)are selected as the feature parameters.Subsequently,three architectures,Deep Neural Network(DNN),Convolutional Neural Network(CNN),and Recurrent Neural Network(RNN),are used to evaluate the data sets processed by different coupled algorithms.The hyperband algorithm is introduced to optimize the hyperparameters of the RNN model,and the prediction accuracy of different architectures is compared between the RNN model with dense layers and without dense layers.Finally,a rockburst prediction model based on data preprocessing and the Hyperband-DNN model is developed.The prediction results show that data preprocessing can significantly improve the model prediction accuracy;the model architecture with the highest prediction accuracy can be found quickly using the hyperband algorithm;and adding the dense layer can improve the stability and prediction accuracy of the model.
基金The support provided by the Natural Science Foundation of Hubei Province(Grant No.2021CFA081)the National Natural Science Foundation of China(Grant No.42277160)the fellowship of China Postdoctoral Science Foundation(Grant No.2022TQ0241)is gratefully acknowledged.
摘要The big data generated by tunnel boring machines(TBMs)are widely used to reveal complex rock-machine interactions by machine learning(ML)algorithms.Data preprocessing plays a crucial role in improving ML accuracy.For this,a TBM big data preprocessing method in ML was proposed in the present study.It emphasized the accurate division of TBM tunneling cycle and the optimization method of feature extraction.Based on the data collected from a TBM water conveyance tunnel in China,its effectiveness was demonstrated by application in predicting TBM performance.Firstly,the Score-Kneedle(S-K)method was proposed to divide a TBM tunneling cycle into five phases.Conducted on 500 TBM tunneling cycles,the S-K method accurately divided all five phases in 458 cycles(accuracy of 91.6%),which is superior to the conventional duration division method(accuracy of 74.2%).Additionally,the S-K method accurately divided the stable phase in 493 cycles(accuracy of 98.6%),which is superior to two state-of-the-art division methods,namely the histogram discriminant method(accuracy of 94.6%)and the cumulative sum change point detection method(accuracy of 92.8%).Secondly,features were extracted from the divided phases.Specifically,TBM tunneling resistances were extracted from the free rotating phase and free advancing phase.The resistances were subtracted from the total forces to represent the true rock-fragmentation forces.The secant slope and the mean value were extracted as features of the increasing phase and stable phase,respectively.Finally,an ML model integrating a deep neural network and genetic algorithm(GA-DNN)was established to learn the preprocessed data.The GA-DNN used 6 secant slope features extracted from the increasing phase to predict the mean field penetration index(FPI)and torque penetration index(TPI)in the stable phase,guiding TBM drivers to make better decisions in advance.The results indicate that the proposed TBM big data preprocessing method can improve prediction accuracy significantly(improving R2s of TPI and FPI on the test dataset from 0.7716 to 0.9178 and from 0.7479 to 0.8842,respectively).
摘要The Moon-based Ultraviolet Telescope (MUVT) is one of the payloads on the Chang'e-3 (CE-3) lunar lander. Because of the advantages of having no at- mospheric disturbances and the slow rotation of the Moon, we can make long-term continuous observations of a series of important celestial objects in the near ultra- violet band (245-340 nm), and perform a sky survey of selected areas, which can- not be completed on Earth. We can find characteristic changes in celestial brightness with time by analyzing image data from the MUVT, and deduce the radiation mech- anism and physical properties of these celestial objects after comparing with a phys- ical model. In order to explain the scientific purposes of MUVT, this article analyzes the preprocessing of MUVT image data and makes a preliminary evaluation of data quality. The results demonstrate that the methods used for data collection and prepro- cessing are effective, and the Level 2A and 2B image data satisfy the requirements of follow-up scientific researches.
基金National Natural Science Foundation of China(Nos.51875199 and 51905165)Hunan Natural Science Fund Project(2019JJ50186)the Ke7y Research and Development Program of Hunan Province(No.2018GK2073).
摘要Due to the frequent changes of wind speed and wind direction,the accuracy of wind turbine(WT)power prediction using traditional data preprocessing method is low.This paper proposes a data preprocessing method which combines POT with DBSCAN(POT-DBSCAN)to improve the prediction efficiency of wind power prediction model.Firstly,according to the data of WT in the normal operation condition,the power prediction model ofWT is established based on the Particle Swarm Optimization(PSO)Arithmetic which is combined with the BP Neural Network(PSO-BP).Secondly,the wind-power data obtained from the supervisory control and data acquisition(SCADA)system is preprocessed by the POT-DBSCAN method.Then,the power prediction of the preprocessed data is carried out by PSO-BP model.Finally,the necessity of preprocessing is verified by the indexes.This case analysis shows that the prediction result of POT-DBSCAN preprocessing is better than that of the Quartile method.Therefore,the accuracy of data and prediction model can be improved by using this method.
基金funded by eVIDA Research group IT-905-16 from Basque Government.
摘要Quantum Machine Learning(QML)techniques have been recently attracting massive interest.However reported applications usually employ synthetic or well-known datasets.One of these techniques based on using a hybrid approach combining quantum and classic devices is the Variational Quantum Classifier(VQC),which development seems promising.Albeit being largely studied,VQC implementations for“real-world”datasets are still challenging on Noisy Intermediate Scale Quantum devices(NISQ).In this paper we propose a preprocessing pipeline based on Stokes parameters for data mapping.This pipeline enhances the prediction rates when applying VQC techniques,improving the feasibility of solving classification problems using NISQ devices.By including feature selection techniques and geometrical transformations,enhanced quantum state preparation is achieved.Also,a representation based on the Stokes parameters in the PoincaréSphere is possible for visualizing the data.Our results show that by using the proposed techniques we improve the classification score for the incidence of acute comorbid diseases in Type 2 Diabetes Mellitus patients.We used the implemented version of VQC available on IBM’s framework Qiskit,and obtained with two and three qubits an accuracy of 70%and 72%respectively.
基金Supported by the National Science Foundation(No.IIS-9988642)the Multidisciplinary Research Program
摘要Many classifiers and methods are proposed to deal with letter recognition problem. Among them, clustering is a widely used method. But only one time for clustering is not adequately. Here, we adopt data preprocessing and a re kernel clustering method to tackle the letter recognition problem. In order to validate effectiveness and efficiency of proposed method, we introduce re kernel clustering into Kernel Nearest Neighbor classification(KNN), Radial Basis Function Neural Network(RBFNN), and Support Vector Machine(SVM). Furthermore, we compare the difference between re kernel clustering and one time kernel clustering which is denoted as kernel clustering for short. Experimental results validate that re kernel clustering forms fewer and more feasible kernels and attain higher classification accuracy.
摘要In order to reduce the risk of non-performing loans, losses, and improve the loan approval efficiency, it is necessary to establish an intelligent loan risk and approval prediction system. A hybrid deep learning model with 1DCNN-attention network and the enhanced preprocessing techniques is proposed for loan approval prediction. Our proposed model consists of the enhanced data preprocessing and stacking of multiple hybrid modules. Initially, the enhanced data preprocessing techniques using a combination of methods such as standardization, SMOTE oversampling, feature construction, recursive feature elimination (RFE), information value (IV) and principal component analysis (PCA), which not only eliminates the effects of data jitter and non-equilibrium, but also removes redundant features while improving the representation of features. Subsequently, a hybrid module that combines a 1DCNN with an attention mechanism is proposed to extract local and global spatio-temporal features. Finally, the comprehensive experiments conducted validate that the proposed model surpasses state-of-the-art baseline models across various performance metrics, including accuracy, precision, recall, F1 score, and AUC. Our proposed model helps to automate the loan approval process and provides scientific guidance to financial institutions for loan risk control.
基金fully supported by the National Natural Science Foundation of China(52072412)。
摘要The futures trading market is an important part of the financial markets and soybeans are one of the most strategically important crops in the world.How to predict soybean future price is a challenging topic being studied by many researchers.This paper proposes a novel hybrid soybean future price prediction model which includes two stages of data preprocessing and deep learning prediction.In the data preprocessing stage,futures price series are decomposed into subsequences using the ICEEMDAN(improved complete ensemble empirical mode decomposition with adaptive noise)method.The Lempel-Ziv complexity determination method was then used to identify and reconstruct high-frequency subsequences.Finally,the high frequency component is decomposed secondarily using variational mode decomposition optimized by beluga whale optimization algorithm.In the deep learning prediction stage,a deep extreme learning machine optimized by the sparrow search algorithm was used to obtain the prediction results of all subseries and reconstructs them to obtain the final soybean future price prediction results.Based on the experimental results of soybean future price markets in China,Italy,and the United States,it was found that the hybrid method proposed provides superior performance in terms of prediction accuracy and robustness.
摘要With the continuous expansion and increasing complexity of engineering projects, cost forecasting has become particularly critical in project management. To meet the demands of the big data era, this study utilizes extensive historical engineering data to examine the intrinsic relationships and fluctuation patterns of project costs, covering data collection, preprocessing, and key factor analysis. By employing cutting-edge algorithms for comprehensive comparison and intelligent identification of cost-influencing factors, an efficient cost forecasting methodology was developed, achieving accurate predictions of cost trends. Results demonstrate that this method exhibits high forecasting accuracy across various project scenarios, effectively capturing key cost variation characteristics and providing robust insights for budget preparation and risk control. Practical validation confirms that with sufficient data resources and rigorous analysis of critical factors, cost management precision can be significantly enhanced. Overall, this research not only establishes innovative technical approaches for cost forecasting but also advances the digital transformation of project management, offering substantial practical implications and broad application prospects for improving resource allocation efficiency and optimizing project decision-making.
基金2024 Anqing Normal University University-Level Key Project(ZK2024062D)。
摘要This study examines the Big Data Collection and Preprocessing course at Anhui Institute of Information Engineering,implementing a hybrid teaching reform using the Bosi Smart Learning Platform.The proposed hybrid model follows a“three-stage”and“two-subject”framework,incorporating a structured design for teaching content and assessment methods before,during,and after class.Practical results indicate that this approach significantly enhances teaching effectiveness and improves students’learning autonomy.
基金National Natural Science Foundation of China(31371515,31671226)。
摘要Liquid chromatography–mass spectrometry(LC–MS)has enabled the detection of thousands of metabolite features from a single biological sample that produces large and complex datasets.One of the key issues in LC–MS-based metabolomics is comprehensive and accurate analysis of enormous amount of data.Many free data preprocessing tools,such as XCMS,MZmine,MAVEN,and MetaboAnalyst,as well as commercial software,have been developed to facilitate data processing.However,researchers are challenged by the inevitable and unconquerable yields of numerous false-positive peaks,and human errors while manually removing such false peaks.Even with continuous improvements of data processing tools,there can still be many mistakes generated during data preprocessing.In addition,many data preprocessing software exist,and every tool has its own advantages and disadvantages.Thereby,a researcher needs to judge what kind of software or tools to choose that most suit their vendor proprietary formats and goal of downstream analysis.Here,we provided a brief introduction of the general steps of raw MS data processing,and properties of automated data processing tools.Then,characteristics of mainly free data preprocessing software were summarized for researchers’consideration in conducting metabolomics study.
基金supported by the National Natural Science Foundation of China(Grant No.52409151)the Programme of Shenzhen Key Laboratory of Green,Efficient and Intelligent Construction of Underground Metro Station(Programme No.ZDSYS20200923105200001)the Science and Technology Major Project of Xizang Autonomous Region of China(XZ202201ZD0003G).
摘要Substantial advancements have been achieved in Tunnel Boring Machine(TBM)technology and monitoring systems,yet the presence of missing data impedes accurate analysis and interpretation of TBM monitoring results.This study aims to investigate the issue of missing data in extensive TBM datasets.Through a comprehensive literature review,we analyze the mechanism of missing TBM data and compare different imputation methods,including statistical analysis and machine learning algorithms.We also examine the impact of various missing patterns and rates on the efficacy of these methods.Finally,we propose a dynamic interpolation strategy tailored for TBM engineering sites.The research results show that K-Nearest Neighbors(KNN)and Random Forest(RF)algorithms can achieve good interpolation results;As the missing rate increases,the interpolation effect of different methods will decrease;The interpolation effect of block missing is poor,followed by mixed missing,and the interpolation effect of sporadic missing is the best.On-site application results validate the proposed interpolation strategy's capability to achieve robust missing value interpolation effects,applicable in ML scenarios such as parameter optimization,attitude warning,and pressure prediction.These findings contribute to enhancing the efficiency of TBM missing data processing,offering more effective support for large-scale TBM monitoring datasets.
基金supported by the National Natural Science Foundation of China(11503031)the Basic Science Research Program of Shaanxi Province(2025JCYBMS-049).
摘要In Global Navigation Satellite Systems(GNSS),accurate and stable atomic clocks need to be equipped on satellites to ensure reliable,high-accuracy positioning,navigation,and timing services.It is essential to continuously monitor the behavior of satellite clocks in space and predict satellite clock corrections for real-time GNSS applications,especially for precise point positioning.Some commercial software is available for clock characterization,but special attention has to be paid when referring to satellite clocks,the analysis and prediction of which may be complicated by outliers,data gaps,and periodic fluctuations in onboard clock data,not often encountered by clock data from a timekeeping laboratory.The typical approaches for clock characterization and prediction currently employed in a timekeeping laboratory are therefore unsuitable for clock applications in space.We present a software package developed in MATLAB at the National Time Service Center,Chinese Academy of Sciences,intended for satellite clock characterization and prediction.The software package includes many subroutines and functionalities of particular interest in characterizing and predicting clock behavior in space,such as dynamic frequency stability evaluation,periodic fluctuation analysis,and multi-step prediction of clock signals.The software package allows handling of satellite clock data directly from Receiver Independent Exchange Format clock files widely used in GNSS,facilitating quick characterization analysis and prediction of satellite clocks,with graphically visualized output.
基金Project(50374079) supported by the National Natural Science Foundation of China
摘要It is difficult to detect the anomalies whose matching relationship among some data attributes is very different from others’ in a dataset. Aiming at this problem, an approach based on wavelet analysis for detecting and amending anomalous samples was proposed. Taking full advantage of wavelet analysis’ properties of multi-resolution and local analysis, this approach is able to detect and amend anomalous samples effectively. To realize the rapid numeric computation of wavelet translation for a discrete sequence, a modified algorithm based on Newton-Cores formula was also proposed. The experimental result shows that the approach is feasible with good result and good practicality.
摘要Artificial intelligence(AI)relies on data and algorithms.State-of-the-art(SOTA)AI smart algorithms have been developed to improve the performance of AI-oriented structures.However,model-centric approaches are limited by the absence of high-quality data.Data-centric AI is an emerging approach for solving machine learning(ML)problems.It is a collection of various data manipulation techniques that allow ML practitioners to systematically improve the quality of the data used in an ML pipeline.However,data-centric AI approaches are not well documented.Researchers have conducted various experiments without a clear set of guidelines.This survey highlights six major data-centric AI aspects that researchers are already using to intentionally or unintentionally improve the quality of AI systems.These include big data quality assessment,data preprocessing,transfer learning,semi-supervised learning,machine learning operations(MLOps),and the effect of adding more data.In addition,it highlights recent data-centric techniques adopted by ML practitioners.We addressed how adding data might harm datasets and how HoloClean can be used to restore and clean them.Finally,we discuss the causes of technical debt in AI.Technical debt builds up when software design and implementation decisions run into“or outright collide with”business goals and timelines.This survey lays the groundwork for future data-centric AI discussions by summarizing various data-centric approaches.
基金The author extends his appreciation to the Deputyship for Research&Innovation,Ministry of Education and Qassim University,Saudi Arabia for funding this research work through the Project Number(QU-IF-4-3-3-30013).
摘要The tendency toward achieving more sustainable and green buildings turned several passive buildings into more dynamic ones.Mosques are the type of buildings that have a unique energy usage pattern.Nevertheless,these types of buildings have minimal consideration in the ongoing energy efficiency applications.This is due to the unpredictability in the electrical consumption of the mosques affecting the stability of the distribution networks.Therefore,this study addresses this issue by developing a framework for a short-term electricity load forecast for a mosque load located in Riyadh,Saudi Arabia.In this study,and by harvesting the load consumption of the mosque and meteorological datasets,the performance of four forecasting algorithms is investigated,namely Artificial Neural Network and Support Vector Regression(SVR)based on three kernel functions:Radial Basis(RB),Polynomial,and Linear.In addition,this research work examines the impact of 13 different combinations of input attributes since selecting the optimal features has a major influence on yielding precise forecasting outcomes.For the mosque load,the(SVR-RB)with eleven features appeared to be the best forecasting model with the lowest forecasting errors metrics giving RMSE,nRMSE,MAE,and nMAE values of 4.207 kW,2.522%,2.938 kW,and 1.761%,respectively.
摘要In general,the material properties,loads,resistance of the prestressed concrete continuous rigid frame bridge in different construction stages are time-varying.So,it is essential to monitor the internal force state when the bridge is in construction.Among them,how to assess the safety is one of the challenges.As the continuous monitoring over a long-term period can increase the reliability of the assessment,so,based on a large number of monitored strain data collected from the structural health monitoring system(SHMS)during construction,a calculation method of the punctiform time-varying reliability is proposed in this paper to evaluate the stress state of this type bridge in cantilever construction stage by using the basic reliability theory.At the same time,the optimal stress distribution function in the bridge mid-span base plate is determined when the bridge is closed.This method can provide basis and direction for the internal force control of this type bridge in construction process.So,it can reduce the bridge safety and quality accidents in construction stages.
基金financially supported by the Natural Science Basic foundation of China(Program No.52174325)the Key Research and Development Program of Shaanxi(Grant No.2020GY-166 and Program No.2020GY-247)the Shaanxi Provincial Innovation Capacity Support Plan(Grant No.2023-CX-TD-53).
摘要Predicting NOxin the sintering process of iron ore powder in advance was helpful to adjust the denitrification process in time.Taking NOxin the sintering process of iron ore powder as the object,the boxplot,empirical mode decomposition algorithm,Pearson correlation coefficient,maximum information coefficient and other methods were used to preprocess the sintering data and naive Bayes classification algorithm was used to identify the sintering conditions.The regression prediction model with high accuracy and good stability was selected as the sub-model for different sintering conditions,and the sub-models were combined into an integrated prediction model.Based on actual operational data,the approach proved the superiority and effectiveness of the developed model in predicting NOx,yielding an accuracy of 96.17%and an absolute error of 5.56,and thereby providing valuable foresight for on-site sintering operations.
基金supported by the National Natural Science Foundation of China(No.32502350).
摘要Accurate prediction of inorganic nitrogen concentrations is critical for assessing water quality and supporting sustainable aquatic ecosystem management.Data mining algorithms have demonstrated significant potential in inorganic nitrogen prediction.This review provides a systematic examination of data mining algorithms applied to inorganic nitrogen prediction for aquatic environments,focusing on the complete modeling pipeline from data preparation to predictive algorithm deployment.Essential data preparation techniques involving data acquisition,data cleaning,dimensionality reduction,and dataset partitioning are described in detail.Then,the mechanisms,techniques,and applications are critically evaluated according to the following five categories of data mining algorithms:machine learning models,deep learning models,tree-based models,fuzzy neural networks,and hybrid models.FNN and deep learning models yield top-tier accuracy(mean R2>0.90).In addition,hybrid models integrating LSTM have emerged as the most promising and prevalent framework,driving advancements in prediction precision.Despite significant progress,the field continues to face challenges such as insufficient data quality,limited model generalizability,and a lack of interpretability.Integrating multimodal data,developing lightweight real-time prediction systems,combining explainable artificial intelligence with physics-informed constraints,and strengthening uncertainty quantification will strongly advance inorganic nitrogen prediction models toward more accurate,reliable,and practical decision-support tools.This review provides a systematic reference for researchers and practitioners.
基金supported by the National Natural Science Foundation of China(Grant Nos.52231014 and 52271361)the Natural Science Foundation of Guangdong Province of China(Grant No.2023A1515010684).
摘要Enhancing the accuracy of real-time ship roll prediction is crucial for maritime safety and operational efficiency.To address the challenge of accurately predicting the ship roll status with nonlinear time-varying dynamic characteristics,a real-time ship roll prediction scheme is proposed on the basis of a data preprocessing strategy and a novel stochastic trainer-based feedforward neural network.The sliding data window serves as a ship time-varying dynamic observer to enhance model prediction stability.The variational mode decomposition method extracts effective information on ship roll motion and reduces the non-stationary characteristics of the series.The energy entropy method reconstructs the mode components into high-frequency,medium-frequency,and low-frequency series to reduce model complexity.An improved black widow optimization algorithm trainer-based feedforward neural network with enhanced local optimal avoidance predicts the high-frequency component,enabling accurate tracking of abrupt signals.Additionally,the deterministic algorithm trainer-based neural network,characterized by rapid processing speed,predicts the remaining two mode components.Thus,real-time ship roll forecasting can be achieved through the reconstruction of mode component prediction results.The feasibility and effectiveness of the proposed hybrid prediction scheme for ship roll motion are demonstrated through the measured data of a full-scale ship trial.The proposed prediction scheme achieves real-time ship roll prediction with superior prediction accuracy.