In this part, we will prepare the base model and train the model.
The blog post is a continuation of the post on End to End Deep Learning project to classify normal kidney and tumor kidney images. In this part, we will focus on preparing the base model and training the model using the procedure of modular coding.
Preparing the base model
We already created the project structure in the previous blog, and we have folders of components, pipeline, config, entity, utils, and constants. We have a procedure in the README.md file for updating each file inside each folder. The procedure is as follows.
## Workflows1. Update config.yaml
2. Update secrets.yaml [Optional]
3. Update params.yaml
4. Update the entity
5. Update the configuration manager in src config
6. Update the components
7. Update the pipeline
8. Update the main.py
9. Update the dvc.yaml
10. app.py
For reference, we will not be using app.py and dvc.yaml for now since these will come in the next part.
In the research folder, we created a notebook named 02_prepare_base_model.ipynb. Then we keep updating each block according to the README.md file and then update each file in the folders too.
First, we update the config.yaml file and add this part after the data ingestion block.
prepare_base_model:
root_dir: artifacts/prepare_base_model
base_model_path: artifacts/prepare_base_model/base_model.h5
updated_base_model_path: artifacts/prepare_base_model/base_model_updated.h5
The root directory in the artifacts folder is created which is prepare_base_model, then the base model is created named base_model.h5, and the updated base model is created named base_model_updated.h5.
For updating params.yaml file we will be using the parameters of image_size, learning_rate, include_top, weights and classes.
Next, we update the config_entity.py inside the entity folder and add the following after the data ingestion config part.
@dataclass(frozen=True)
class PrepareBaseModelConfig:
root_dir: Path
base_model_path: Path
updated_base_model_path: Path
params_image_size: list
params_learning_rate: float
params_include_top: bool
params_weights: str
params_classes: int
Next, in the configuration.py file inside the config folder (inside the src folder), we add the block of prepare base model after the get_data_ingestion_config function.
def get_prepare_base_model_config(self) -> PrepareBaseModelConfig:
config = self.config.prepare_base_modelcreate_directories([config.root_dir])
prepare_base_model_config = PrepareBaseModelConfig(
root_dir=Path(config.root_dir),
base_model_path=Path(config.base_model_path),
updated_base_model_path=Path(config.updated_base_model_path),
params_image_size=self.params.IMAGE_SIZE,
params_learning_rate=self.params.LEARNING_RATE,
params_include_top=self.params.INCLUDE_TOP,
params_weights=self.params.WEIGHTS,
params_classes=self.params.CLASSES
)
return prepare_base_model_config
In the above function get_prepare_base_model_config() we prepare the configuration containing the inputs from the config.yaml file (root_dir, base_model_path, updated_base_model_path), the image_size, learning_rate, include_top, weights and classes from the params.yaml file.
So, I think everything makes sense now as to why we updated the config.yaml, params.yaml, and config_entity.py files before we add to configuration.py.
After that, we create the prepare_base_model.py inside the components folder.
import os
import urllib.request as request
from zipfile import ZipFile
import tensorflow as tf
from pathlib import Path
from cnnClassifier.entity.config_entity import PrepareBaseModelConfigclass PrepareBaseModel:
def __init__(self, config: PrepareBaseModelConfig):
self.config = config
def get_base_model(self):
self.model = tf.keras.applications.vgg16.VGG16(
input_shape=self.config.params_image_size,
weights=self.config.params_weights,
include_top=self.config.params_include_top
)
self.save_model(path=self.config.base_model_path, model=self.model)
@staticmethod
def _prepare_full_model(model, classes, freeze_all, freeze_till, learning_rate):
if freeze_all:
for layer in model.layers:
model.trainable = False
elif (freeze_till is not None) and (freeze_till > 0):
for layer in model.layers[:-freeze_till]:
model.trainable = False
flatten_in = tf.keras.layers.Flatten()(model.output)
prediction = tf.keras.layers.Dense(
units=classes,
activation="softmax"
)(flatten_in)
full_model = tf.keras.models.Model(
inputs=model.input,
outputs=prediction
)
full_model.compile(
optimizer=tf.keras.optimizers.SGD(learning_rate=learning_rate),
loss=tf.keras.losses.CategoricalCrossentropy(),
metrics=["accuracy"]
)
full_model.summary()
return full_model
def update_base_model(self):
self.full_model = self._prepare_full_model(
model=self.model,
classes=self.config.params_classes,
freeze_all=True,
freeze_till=None,
learning_rate=self.config.params_learning_rate
)
self.save_model(path=self.config.updated_base_model_path, model=self.full_model)
@staticmethod
def save_model(path: Path, model: tf.keras.Model):
model.save(path)
Inside the get_base_model() function, we have used the Keras VGG16 model. We are using the softmax activation function. The main body of the code is referred from the main Keras codebase.
After this in the pipeline folder, we create the stage_02_prepare_base_model.py file.
from cnnClassifier.config.configuration import ConfigurationManager
from cnnClassifier.components.prepare_base_model import PrepareBaseModel
from cnnClassifier import loggerSTAGE_NAME = "Prepare base model"
class PrepareBaseModelTrainingPipeline:
def __init__(self):
pass
def main(self):
config = ConfigurationManager()
prepare_base_model_config = config.get_prepare_base_model_config()
prepare_base_model = PrepareBaseModel(config=prepare_base_model_config)
prepare_base_model.get_base_model()
prepare_base_model.update_base_model()
if __name__ == '__main__':
try:
logger.info(f"*******************")
logger.info(f">>>>>> stage {STAGE_NAME} started <<<<<<")
obj = PrepareBaseModelTrainingPipeline()
obj.main()
logger.info(f">>>>>> stage {STAGE_NAME} completed <<<<<<\n\nx==========x")
except Exception as e:
logger.exception(e)
raise e
The pipeline is created using the get_prepare_base_model_config() from the ConfigurationManager, the PrepareBaseModel is used to create the base model from which we get the prepare_base_model.get_base_model() and prepare_base_model.update_base_model().
We update the same in the main.py file after the data ingestion stage.
STAGE_NAME = "Prepare base model"
try:
logger.info(f"*******************")
logger.info(f">>>>>> stage {STAGE_NAME} started <<<<<<")
prepare_base_model = PrepareBaseModelTrainingPipeline()
prepare_base_model.main()
logger.info(f">>>>>> stage {STAGE_NAME} completed <<<<<<\n\nx==========x")
except Exception as e:
logger.exception(e)
raise e
When we run main.py in the terminal, these steps will follow.
Model training
In the research folder, we created a notebook file 03_model_training.ipynb and we keep updating it according to README.md and also update the files in the folders.
We follow the same steps for updating the config.yaml file first.
training:
root_dir: artifacts/training
trained_model_path: artifacts/training/model.h5
The training folder will be created inside the artifacts folder and model.h5 will be created after the training is done.
Next, we update the config_entity.py inside the entity folder and add the following after the prepare base model config part.
@dataclass(frozen=True)
class TrainingConfig:
root_dir: Path
trained_model_path: Path
updated_base_model_path: Path
training_data: Path
params_epochs: int
params_batch_size: int
params_is_augmentation: bool
params_image_size: list
The parameters of epochs, batch_size, is_augmentation, image_size, training_data, trained_model_path and updated_base_model_path are used for training.
Next, in the configuration.py file inside the config folder (inside the src folder), we add the block of train config model after the get_prepare_base_model_config function.
def get_training_config(self) -> TrainingConfig:
training = self.config.training
prepare_base_model = self.config.prepare_base_model
params = self.params
training_data = os.path.join(self.config.data_ingestion.unzip_dir, "kidney-ct-scan-image")
create_directories([
Path(training.root_dir)
])training_config = TrainingConfig(
root_dir=Path(training.root_dir),
trained_model_path=Path(training.trained_model_path),
updated_base_model_path=Path(prepare_base_model.updated_base_model_path),
training_data=Path(training_data),
params_epochs=params.EPOCHS,
params_batch_size=params.BATCH_SIZE,
params_is_augmentation=params.AUGMENTATION,
params_image_size=params.IMAGE_SIZE
)
return training_config
The steps that are followed are utilizing the training data that was created by unzipping the downloaded data in the data ingestion stage, the prepare base model, and the parameters like epoch, batch size, augmentation, and image size.
After that, we create the model_training.py inside the components folder.
import os
import urllib.request as request
from zipfile import ZipFile
import tensorflow as tf
import time
from pathlib import Path
from cnnClassifier.entity.config_entity import TrainingConfigclass Training:
def __init__(self, config: TrainingConfig):
self.config = config
def get_base_model(self):
self.model = tf.keras.models.load_model(
self.config.updated_base_model_path
)
def train_valid_generator(self):
datagenerator_kwargs = dict(
rescale = 1./255,
validation_split=0.20
)
dataflow_kwargs = dict(
target_size=self.config.params_image_size[:-1],
batch_size=self.config.params_batch_size,
interpolation="bilinear"
)
valid_datagenerator = tf.keras.preprocessing.image.ImageDataGenerator(
**datagenerator_kwargs
)
self.valid_generator = valid_datagenerator.flow_from_directory(
directory=self.config.training_data,
subset="validation",
shuffle=False,
**dataflow_kwargs
)
if self.config.params_is_augmentation:
train_datagenerator = tf.keras.preprocessing.image.ImageDataGenerator(
rotation_range=40,
horizontal_flip=True,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
**datagenerator_kwargs
)
else:
train_datagenerator = valid_datagenerator
self.train_generator = train_datagenerator.flow_from_directory(
directory=self.config.training_data,
subset="training",
shuffle=True,
**dataflow_kwargs
)
@staticmethod
def save_model(path: Path, model: tf.keras.Model):
model.save(path)
def train(self):
self.steps_per_epoch = self.train_generator.samples // self.train_generator.batch_size
self.validation_steps = self.valid_generator.samples // self.valid_generator.batch_size
self.model.fit(
self.train_generator,
epochs=self.config.params_epochs,
steps_per_epoch=self.steps_per_epoch,
validation_steps=self.validation_steps,
validation_data=self.valid_generator
)
self.save_model(
path=self.config.trained_model_path,
model=self.model
)
The model_training.py file is created with test data of 20% and training data of 80%.
After this in the pipeline folder, we create the stage_03_model_training.py file.
from cnnClassifier.config.configuration import ConfigurationManager
from cnnClassifier.components.model_training import Training
from cnnClassifier import loggerSTAGE_NAME = "Training"
class ModelTrainingPipeline:
def __init__(self):
pass
def main(self):
config = ConfigurationManager()
training_config = config.get_training_config()
training = Training(config=training_config)
training.get_base_model()
training.train_valid_generator()
training.train()
if __name__ == '__main__':
try:
logger.info(f"*******************")
logger.info(f">>>>>> stage {STAGE_NAME} started <<<<<<")
obj = ModelTrainingPipeline()
obj.main()
logger.info(f">>>>>> stage {STAGE_NAME} completed <<<<<<\n\nx==========x")
except Exception as e:
logger.exception(e)
raise e
The ConfigurationManager is used as before in the prepare base model part, the Training class is used from model_training.py in the components folder.
We update the same in the main.py after the prepare base model stage.
STAGE_NAME = "Training"
try:
logger.info(f"*******************")
logger.info(f">>>>>> stage {STAGE_NAME} started <<<<<<")
model_trainer = ModelTrainingPipeline()
model_trainer.main()
logger.info(f">>>>>> stage {STAGE_NAME} completed <<<<<<\n\nx==========x")
except Exception as e:
logger.exception(e)
raise eThe process of Preparing the base model is shown below.
The process of training the model is shown below.
The above screenshots show the process of base model preparation and training the model. The logs are present in the files in the pipeline folder and main.py. Since we used only 1 epoch to train the model the accuracy is horrible and the loss is off the roof, so you can try with higher epochs and make the model better. Higher epochs do take time to train the model.
The idea here is not to prepare the ideal Kidney tumor image classification algorithm but to create an end-to-end model where we will see the process of data ingestion to deployment in AWS.
Note: It needs to be understood that there can be dependency issues, conflicts, and several other kinds of bugs or errors while executing the project at every stage, so we have to check thoroughly every time we run the project.
Everything is updated on the GitHub repo, so make sure to look into it.
Conclusion
We continued the project from the previous part and prepared the base model and trained the model. The project structure almost remains the same for any end-to-end Machine Learning or Deep Learning project so we must understand it very well and keep this project structure as a future reference to recreate new projects and deploy.
In the next part, we will cover the Model evaluation and deploy it on Dagshub and learn about MLflow too to understand model tracking and experimentation.
Stay tuned for the next part and follow me here and say hi.
Email: abhi2652254@gmail.com
https://aigents.co/learn/roadmaps/data-science-roadmap?source=post_page—–8ca27c4ed4ea——————————–
Reference
Be the first to comment