I'm trying to train a machine learning model to detect if an image is blurred or not.
I have 11,798 unblurred images, and I have a script to blur them and then use that to train my model.
However when I run the exact same training 5 times the results are wildly inconsistent (as you can see below). It also only gets to 98.67% accuracy max.
I'm pretty new to machine learning, so maybe I'm doing something really wrong. But coming from a software engineering background and just starting to learn machine learning, I have tons of questions. It's a struggle to know why it's so inconsistent between runs. It's a struggle to know how good is good enough (ie. when should I deploy the model). It's a struggle to know how to continue to improve the accuracy and make the model better.
Any advice or insight would be greatly appreciated.
View all the code: https://gist.github.com/fishcharlie/68e808c45537d79b4f4d33c26e2391dd
I'm not sure what the issue is, but I can't see your confusion matrix. There's a video player placeholder, but it doesn't load anything.
Hard to tell whether the larger convolution was a bust or not. It's got the one big spike away from the training loss, but every other epoch is moving downward with training, which looks good. Maybe try a few more runs, and if they commonly have that spike, try to give them longer to see if that loss keeps going downward to meet the training.
For efficiency you can also increase the convolutional stride. If you're doing a 10x10 you can move by more than one pixel each stride. Since you're not really trying to build local structures, going 5 or 10 pixels at a time seems reasonable.
Sorry for the delayed reply. I really appreciate your help so far.
Here is the raw link to the confusion matrix: https://eventfrontier.com/pictrs/image/1a2bc13e-378b-4920-b7f6-e5b337cd8c6f.webm
I changed it to
keras.layers.Conv2D(16, 10, strides=(5, 5), activation='relu'). Dense units still at 64.And in case the confusion matrix still doesn't work, here is a still image from the last run.
EDIT: The wrong image was uploaded originally.
The new runs don't look good. I wouldn't have expected a half-width stride to cause issues though.
Are you sure the confusion matrix for the validation set? It doesn't match a ~90% accuracy (previous solo run) or ~70-80% accuracy for the validation in the new runs.
So someone else suggested to reduce the learning rate. I tried that and at least to me it looks a lot more stable between runs. All the code is my original code (none of the suggestions you made) but I reduced the learning rate to 0.00001 instead of 0.0001.
Not quite sure what that means exactly tho. Or if more adjustments are needed.
As for the confusion matrix. I think the issue is the difference between smoothed values in TensorBoard vs the actual values. But I just ran it again with the previous values to verify. It does look like it matches up if you look at the actual value instead of the smoothed value.
Yeah, that's looking a lot better. Too high learning rate is something I usually expect to see represented in an erratic training curve rather than (or I guess in addition to) an erratic validation curve though.
The learning rate is basically a measure of how big a step you're going to take when trying to update your weights. If it's large you'll approach solutions quickly but are likely to overshoot, if it's small you'll approach slowly and may end up stuck in a local minima as the step might be too small to leave it so every attempt will always be reversed by further training. IIRC, you had a decay to start high and get lower, which tries to get the best of both worlds, but it may have been that it started too high and/or didn't reduce quickly enough. The "steps" parameter there is counting in batches of images, so you're probably not getting much movement in 6 epochs. It looks like changing the initial rate solved your problem though, so there's not much reason to try to tweak that. Something to keep in mind for future efforts though.
And yeah, I wasn't looking at the unsmoothed data. That is quite a lot of variation.
Got it. Thanks so much for your help!! Still a lot to learn here.
Coming from a world of building software where things are very binary (it works or it doesn't), it's also really tough to judge how good is "good enough". There is a point of diminishing returns, and not sure at what point to say that it's good enough vs continuing to learn and improve it.
Really appreciate your help here tho.
No problem, happy to help. In a lot of cases, even direct methods couldn't reach 100%. Sometimes the problem definition, combined with just regular noise in you input, will mean that you can have examples that have basically the same input, but different classes.
In the blur-domain, for example, if one of your original "unblurred" images was already blurred (or just out of focus) it might look pretty indistinguishable from "blurred" image. Then the only way for the net to "learn" to solve that problem is by overfitting to some unique value in that image.
A lot of machine learning is just making sure the nets are actually solving your problem rather than figuring out a way to cheat.