Repository navigation
Task B.3
This week I continued to further pull the code apart and make it more modular aiding is easier understanding and ease of development. I also added a few more features to the code to give more control over the data presentation.
The predict function was further seperated out into a predict_test function, candlestick and boxplot fucntions and the final prediction left in the predict function. This was done to make the code more modular and easier to understand. As well as allowing the major graphing changes to amde easier.
def predict_test(model, scaler, x_test, test_index, feature_columns=['Open', 'High', 'Low', 'Close', 'Volume']):
# Uses model and data to make prediction
# Params:
# model : The model previously generated to actually be tested
# scaler : The scaler used to process the data so it can be de-normalised
# x_test : The test data to be used for prediction
# test_index (list) : Datetime index of the test data to be added to prediction_df for graphs
# feature_columns (list) : The list of features to use to feed into the model, default is everything grabbed from yahoo
prediction_df = pd.DataFrame(index=test_index)
#------------------------------------------------------------------------------
# Make predictions on test data
#------------------------------------------------------------------------------
for column in feature_columns:
predicted_prices = model.predict(x_test[column])
# Clearly, as we transform our data into the normalized range (0,1),
# we now need to reverse this transformation
predicted_prices = scaler[column].inverse_transform(predicted_prices)
predicted_prices = np.array(predicted_prices)
prediction_df[column] = predicted_prices
return prediction_dfThe predict_test function takes the model, scaler, test data and test index and returns a dataframe of the predicted values. This is done by looping through the feature columns and predicting the values for each column. The predicted values are then de-normalised and added to the dataframe.
def candlestick(test_df, predicted_df, days=1, feature_columns=['Open', 'High', 'Low', 'Close', 'Volume']):
# Uses model and data to make prediction
# Params:
# test_df (df) : The test data downloaded from yahoo
# predicted_df (df) : The predictions based on the test data
# days (int) : The amount of days to show in a candlestick, default is 1
# feature_columns (list) : The list of features graphed from the prediction, default is everything grabbed from yahoo
#------------------------------------------------------------------------------
# Plot the test predictions
## To do:
# 1) Show chart of next few days (predicted)
#------------------------------------------------------------------------------
# Ensure that days is valid
assert days >= 0, 'days must be bigger than or equal to 1'
# Tell how the pd resample should treat the different vales
aggregation = {
'Open':'first',
'High':'max',
'Low':'min',
'Close':'last',
'Volume':'sum'
}
# Resample both the test data and the prediction data based on 'days'
resampled_test_df = test_df.resample(f'{days}d').agg(aggregation)
resampled_predicted_df = predicted_df.resample(f'{days}d').agg(aggregation)
# Derrive the dates of of the graph from the dataset
date_range = f'{test_df.index[0]:%b %Y} - {test_df.index[-1]:%b %Y}'
# Create candlestick graph of test data
test_data_graph = graphs.Candlestick(
x=resampled_test_df.index,
open=resampled_test_df['Open'],
high=resampled_test_df['High'],
low=resampled_test_df['Low'],
close=resampled_test_df['Close'],
showlegend=False
)
# Create scatter graphs of predicted feature
predicted_data_graphs = []
for column in feature_columns:
graph = graphs.Scatter(
x=resampled_predicted_df.index,
y=resampled_predicted_df[column],
name=f'Predicted {column}'
)
predicted_data_graphs.append(graph)
# Put graphs together in figure
fig = graphs.Figure(data=([test_data_graph] + predicted_data_graphs))
# Update figure titles
fig.update_layout(
title=f'{COMPANY} Share Prices {date_range}',
yaxis_title='Price ($)'
)
return figThe candlestick function takes the test data, predicted data, days and feature columns and returns a plotly figure of the candlestick graph of the test data and the predicted data. This is done by resampling the data based on the desired days and then creating a candlestick graph of the test data and scatter graphs of the predicted data. These are then added to a plotly figure and returned.
def boxplot(test_df, predicted_df, days=1, feature_columns=['Open', 'High', 'Low', 'Close']):
# Uses model and data to make prediction
# Params:
# test_df (df) : The test data downloaded from yahoo
# predicted_df (df) : The predictions based on the test data
# days (int) : The amount of days to show in a candlestick, default is 1
# feature_columns (list) : The list of features graphed from the prediction, default is everything grabbed from yahoo
#------------------------------------------------------------------------------
# Plot the test predictions
## To do:
# 1) Show chart of next few days (predicted)
#------------------------------------------------------------------------------
# Ensure that days is valid
assert days >= 0, 'days must be bigger than or equal to 1'
# Tell how the pd resample should treat the different vales
aggregation = {
'Open':'first',
'High':'max',
'Low':'min',
'Close':'last',
'Volume':'sum'
}
# Resample both the test data and the prediction data based on 'days'
resampled_test_df = test_df.resample(f'{days}d').agg(aggregation)
resampled_predicted_df = predicted_df.resample(f'{days}d').agg(aggregation)
# Derrive the dates of of the graph from the dataset
date_range = f'{test_df.index[0]:%b %Y} - {test_df.index[-1]:%b %Y}'
# Create a box plot for each day
boxes = []
for date in resampled_test_df.index:
# Select the data for this day
day_data = resampled_test_df[resampled_test_df.index == date][['Open', 'High', 'Low', 'Close']]
# Create a box plot for this day's data
box = graphs.Box(
y=day_data.values[0],
name=str(date),
showlegend=False
)
boxes.append(box)
# Create scatter graphs of predicted feature
predicted_data_graphs = []
for column in feature_columns:
graph = graphs.Scatter(
x=resampled_predicted_df.index,
y=resampled_predicted_df[column],
name=f'Predicted {column}'
)
predicted_data_graphs.append(graph)
# Put graphs together in figure
fig = graphs.Figure(data=(boxes + predicted_data_graphs))
# Update figure titles
fig.update_layout(
title=f'{COMPANY} Share Prices {date_range}',
yaxis_title='Price ($)'
)
return figThe boxplot function takes the test data, predicted data, days and feature columns and returns a plotly figure of the boxplot graph of the test data and the predicted data. This is done by resampling the data based on the desired days and then creating a boxplot graph of the test data and scatter graphs of the predicted data. These are then added to a plotly figure and returned. The boxplot graph is created by creating a box for each day and adding it to the figure.
def predict(model, scaler, model_inputs, prediction_days=60):
# Uses model and data to make prediction
# Params:
# model : The model previously generated to actually be tested
# scaler : The scaler used to process the data so it can be de-normalised
# actual_prices (list) : The prices downloaded from yahoo to compare against
# prediction_days (int) : How far back the final prediction should look, default is 60 (e.g last 60 days of model inputs)
#------------------------------------------------------------------------------
# Predict next day
#------------------------------------------------------------------------------
real_data = [model_inputs[len(model_inputs) - prediction_days:, 0]]
real_data = np.array(real_data)
real_data = np.reshape(real_data, (real_data.shape[0], real_data.shape[1], 1))
prediction = model.predict(real_data)
prediction = scaler.inverse_transform(prediction)
# A few concluding remarks here:
# 1. The predictor is quite bad, especially if you look at the next day
# prediction, it missed the actual price by about 10%-13%
# Can you find the reason?
# 2. The code base at
# https://github.com/x4nth055/pythoncode-tutorials/tree/master/machine-learning/stock-prediction
# gives a much better prediction. Even though on the surface, it didn't seem
# to be a big difference (both use Stacked LSTM)
# Again, can you explain it?
# A more advanced and quite different technique use CNN to analyse the images
# of the stock price changes to detect some patterns with the trend of
# the stock price:
# https://github.com/jason887/Using-Deep-Learning-Neural-Networks-and-Candlestick-Chart-Representation-to-Predict-Stock-Market
# Can you combine these different techniques for a better prediction??
return predictionThe predict function takes the model, scaler, model inputs and prediction days and returns a prediction. This is done by taking the last 60 days of the model inputs and predicting the next day. The prediction is then de-normalised and returned. This is the predict function in it's most simplest form and allows for easy testing of the model.
- Figuring out how I should plot the predicted data on the same graph as the test data was a big challenge this week. I had to do a lot of research into plotly and how it works to figure out how to do this.
- Creating the boxplot function was the toughest part of this week as I was looking for a solution much like the candlestick plot where plotly would do the heavy lifting for me. However, I could not find a way to do this and had to create the boxes myself. Many suggested that a candlestick was the better solution and weren't very helpful in forums.
- Resampling the data was also a challange as it wasn't imdeiatle evident what the best way to do this was and I had to do a lot of research into it.