Introduction
In the realm of construction, the precision of cost management and economic efficiency directly influences the success of a project. At the heart of achieving this precision lies the expertise of the quantity surveyor, professionals dedicated to ensuring that projects are completed within budget and according to financial forecasts. A cornerstone of their work involves generating detailed cost plans that adhere to industry standards, one example would be the Australian Institute of Quantity Surveyors (AIQS). Their guideline splits costs into elemental classifications, providing a structured approach to estimating, controlling project expenses, and sets up a base line for cost plans A to C.
This article explores and reiterates the initial extraction of data from my cost plans to employing advanced artificial intelligence (AI) models, culminating in achieving an impressive accuracy rate when it comes to elemental coding of cost plans under the research projects that i have been tasked on. It aims to unfold the layers of AIQS elemental classification and its practical application in the day-to-day profession of a quantity surveyor, enhancing the accuracy and efficiency of cost plans. So that in future we don’t have to tag descriptions with elemental coding’s.
# Simple Flat Dictionary for Tagging
CODING_ELEMENTAL_AIQS = {
"PR": "Preliminaries",
"SB": "Substructure",
"CL": "Columns",
"UF": "Upper Floors",
"SC": "Staircases",
"RF": "Roof",
"EW": "External Walls",
"WW": "Windows",
"ED": "External Doors",
"NW": "Internal Walls",
"NS": "Internal Screens & Borrowed Lights",
"ND": "Internal Doors",
"WF": "Wall Finishes",
"FF": "Floor Finishes",
"CF": "Ceiling Finishes",
"FT": "Fitments",
"SE": "Special Equipment",
"SF": "Sanitary Fixtures",
"PD": "Sanitary Plumbing",
"ME": "Mechanical Services",
"WS": "Water Supply",
"GS": "Gas Services",
"SH": "Space Heating",
"VE": "Ventilation",
"EC": "Evaporative Cooling",
"AC": "Air Conditioning",
"CE": "Centralised Energy Systems",
"FP": "Fire Protection",
"LP": "Electric Light & Power",
"CM": "Communications",
"TS": "Transportation Systems",
"SS": "Special Services",
"AR": "Alterations & Renovations",
"XP": "Site Preparation",
"XR": "Roads, Footpaths & Paved Areas",
"XN": "Boundary Walls, Fencing & Gates",
"XB": "Outbuildings & Covered Ways",
"XL": "Landscaping & Improvements",
"XK": "External Stormwater Drainage",
"XD": "External Sewer Drainage",
"XW": "External Water Supply",
"XG": "External Gas",
"XF": "External Fire Protection",
"XE": "External Electric Light & Power",
"XC": "External Communications",
"XS": "External Special Services",
"XX": "External Alterations & Renovations",
"YY": "Special Provisions",
"ET": "Escalation to Tender",
# Add any additional codes as needed
}Data Extraction and Preparation
The journey commenced with the extraction of meaningful data from previous cost plans that i worked on CostX software, renowned for its elemental and trade coding capabilities. Following the extraction, the data were exported to Excel for preliminary analysis, setting the stage for a crucial clean-up phase where I had to go in and clean up all unnecessary parts of the dataset.
A custom Python script was developed to refine the exported data. The clean-up process involved removing empty rows, discarding non-numerical quantity and rate values, and ensuring the presence of a description and elemental code for each entry. This meticulous data preparation was foundational, ensuring the integrity and usefulness of the data for training AI models. The script served not just as a filter but as a bridge between raw data and a structured dataset ready for advanced analysis.
import subprocess
import os
import pandas as pd
def convert_xlsx_to_csv_with_xlsx2csv(directory):
for filename in os.listdir(directory):
if filename.endswith('.xlsX'):
file_path = os.path.join(directory, filename)
csv_path = file_path.rsplit('.', 1)[0] + '.csv'
try:
subprocess.run(['xlsx2csv', file_path, csv_path], check=True)
print(f"Converted {file_path} to {csv_path}")
except subprocess.CalledProcessError as e:
print(f"Failed to convert {file_path}: {e}")
def clean_csv_files(directory):
"""Clean all CSV files in the directory."""
print(f"Cleaning CSV files in directory: {directory}.")
for filename in os.listdir(directory):
if filename.endswith('.csv'):
csv_path = os.path.join(directory, filename)
print(f"Cleaning data in {csv_path}")
******************************************
column_names = ['Code', 'Description', 'Quantity', 'UOM', 'Rate', 'SubTotal', 'Factor', 'Total', 'Total', 'Building Name', 'Workbook Name']
df.columns = column_names # Assign column names
# Convert 'SubTotal' column to numeric
df['SubTotal'] = pd.to_numeric(df['SubTotal'], errors='coerce')
# Remove rows where SubTotal column value is not bigger than 1
df = df[df['SubTotal'] > 1]
df = df.dropna(how='all') # Delete rows that do not contain any information
# Save the cleaned data back to CSV
df.to_csv(csv_path, index=False)
print(f"Cleaned data saved to CSV: {csv_path}")
def main(directory):
convert_xlsx_to_csv_with_xlsx2csv(directory)
clean_csv_files(directory)
if __name__ == "__main__":
directory = os.path.dirname(os.path.realpath(__file__))
main(directory)From Simple Models to NLP
The initial foray into AI involved simple models that laid the groundwork for understanding how AI could be leveraged in quantity surveying. However, the quest for higher accuracy and more nuanced data interpretation led to the exploration of Natural Language Processing (NLP) and BERT models. As a non-profesional on data science, it is quite impressive to see these potentials even with a small sample data. This transition marked a significant evolution in the approach to analysing cost plan data, moving from basic predictive models to sophisticated algorithms capable of understanding and processing language-based data.
Employing NLP and BERT models allowed for a deeper dive into the textual data within cost plans, enabling the extraction of more detailed insights and improving the accuracy of cost estimations. This shift underscored the importance of adopting advanced AI technologies to stay at the forefront of the quantity surveying field, ensuring that cost plans are not only accurate but also rich in detail and analysis.
# Simple learning using sklearn libraries
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import classification_report
# Function to train a more complex AI model for text classification
def train_ai_model(csv_file_name):
# Load the dataset
data = pd.read_csv(csv_file_name)
# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
data['Description'],
data['Code'],
test_size=0.2,
random_state=42
)
# Create a pipeline with TfidfVectorizer and LinearSVC
model = make_pipeline(TfidfVectorizer(max_features=5000), LinearSVC(class_weight='balanced'))
# Define a parameter grid to search for the best hyperparameters
param_grid = {
'tfidfvectorizer__ngram_range': [(1, 1), (1, 2)],
'linearsvc__C': [0.1, 1, 10]
}
# Perform grid search
grid_search = GridSearchCV(model, param_grid, cv=5, scoring='accuracy')
grid_search.fit(X_train, y_train)
# Best model after grid search
best_model = grid_search.best_estimator_
# Predict on the testing set
y_pred = best_model.predict(X_test)
# Evaluate the model
print(classification_report(y_test, y_pred))
# Return the best model
return best_model
# Function call
model = train_ai_model('00combined_cleaned_data_trainingset.csv')
Optimization with Optuna
After playing around with a few model trainers, I realised that there are heaps of parameters that can affect the quality of the prediction performance. That’s why I ended up with automating the process of fine tuning with Optuma, so that it would find the sweetspot by recursive iterations. The optimization of the learning algorithms was facilitated by Optuna, a hyperparameter optimization framework. Optuna was instrumental in fine-tuning the model parameters, such as the learning rate and batch sizes, to achieve optimal performance. Through an iterative process as mentioned earlier, Optuna evaluated various combinations of parameters, guiding the adjustment of the models for better accuracy and efficiency.
This optimization process was critical in enhancing the model’s performance, pushing the accuracy rate to new heights. The use of Optuna highlighted the importance of not just selecting the right models but also fine-tuning them to fit the specific nuances of the cost plan data, ensuring that the AI models were fully optimized for the task at hand.
Achieving High Accuracy
The pinnacle of this journey was reaching an accuracy rate of 94% with the AI models, a significant achievement that underscored the success of the meticulous data preparation and optimization efforts. Also its worth mentining that upon reviewing the dataset, I realised as a human even my ability to put elemental codes on certain items are situation, and cant tag certain descriptions purely by looking into the description itself since a masonry wall can be both external or internal walls. Therefore It was quite satisfactory for an ai model to reach 94% accuracy on this task even with using an 80% to 20% conservative ratio for training and validation, respectively. This approach ensured that the model was trained on a comprehensive dataset while still having a robust set for validation and testing.
The training process was rigorous, involving night-long training sessions to fine-tune the models and optimize their learning rates and batch sizes. It’s surprising that this workflow is actually very similar to those Vray Nightlong renders when I was doing in my architecture career back in the graduate years of my life. So my dedication to refining the AI models paid off, as evidenced by the high accuracy rate, showcasing the potential of AI in transforming the field of quantity surveying and setting a new standard for cost plan accuracy.
precision recall f1-score support
accuracy 0.93 3116
macro avg 0.89 0.90 0.89 3116
weighted avg 0.93 0.93 0.93 3116
Case Study: Implementing AIQS Classification
The practical application of the refined AI models to real-world cost plans can bring the theoretical aspects of this journey into reality. Implementing the AIQS elemental classification using the AI models can streamline the process of generating cost plans, significantly improving the workflow and accuracy of cost estimations in future. Hope CostX can implement such feature in future and provide some extension or addin library for professionals working in the field to make their life easier. In my research, we demonstrated the tangible benefits of integrating AI into quantity surveying, including increased efficiency, reduced errors, and a deeper analytical insight into cost elements.
The successful integration of AIQS elemental classification through AI models signifies a leap forward for the profession, offering a glimpse into the future of quantity surveying where AI plays a central role in enhancing the accuracy and efficiency of cost planning.
