[Q] [D] How do machine learning researchers come up with new neural network architectures?
How do machine learning researchers come up with new neural network architectures? By 'architectures' I do not mean additional layers to a preexisting neural network model but a completely new model or a noticeable improvement to a preexisting one, i.e., Cycle GANs vs GANs, Autoencoders vs Variational Autoencoders, etc. Do they:
Study a pre-existing model, analyze its deficiencies and bring about a remedy.
Come up with a 'wild' application that no model addresses and then engineer a model that would make such application feasible.
Start from total mathematical abstraction then stumble on their way upon something that might be useful in the machine learning field.
As a total beginner in machine learning I found the latter to be an extremely exciting field but as the months go by I found the application of the preexisting models to the existing datasets or any new datasets that require nothing but pre-processing and feeding the latter to a deep learning model even if you understood the ins and outs of the model to be extremely boring. What advice(s) would you give to someone who would wish to go beyond applying preexisting deep learning models to actually building one but he isn't a researcher (meaning he is independent and doesn't have the backing of a department, a team, a sponsor, or a company). Thank you for your time.