I was recently reading Evan Martin's blog post on his takeaways from developing the ninja build system, and one thing Evan mentions is his frustration at engineers that don't look at related work when designing systems. This reminded me of Rado Kirov's posts on incremental computation, where Rado argues that a lot of work in the space of incremental software is underdeveloped because of the lack of cooperation between academic and industry actors. Which isn't a novel realization in general, but recently was a novel realization to me, and has me thinking about something more general. So first a motivating comment on that, and then we'll specialize back to the "divide" between academic and industry work.
A large thing I have been struggling with in my (so far incredibly short) career is a frequent incongruency between what I believe is impactful and what I believe is innovative work in an opportunity. Though "impactful" and "innovative" are broad and personal terms, in general I mean "impactful" work to be that which improves the quality of products broader than the work itselfFor example, reducing store waiting lines by 20% is impactful to a business and its consumers, and "innovative" work to be that which is novel in its domainFor example, the derivation of a theorem.. Now, I certainly don't mean to claim there is no overlap in impactful and innovative opportunities, or even that some overlap is uncommon, but I do argue that opportunities with a large degree of impact and innovation in the steady statebecause certainly, anything can one day be made into an innovation or induce a great impact! are not all that common.
Now back to the note we started this discussion with -- I think the clearest place where this divide comes up is in this distinction between industry work (impact by way of application) and academic work (innovation by way of theory).
Even though theory and application are complementary - theory inspires application, inspires new theory, etc. - I believe that popular culture has separated these two ideas by a thin bridge. This is evident even in the divide we understand between theory and application!
Of course, this divide is very purposeful. You can't do everything at once, and we know that the $\text{scientist}\leftrightarrow\text{engineer}$ model works well; it's been alive for a very long time. The problem comes when people working on applications lack knowledge of theory (and do not seek to learn it!), or vice versa. Then, what is novel to an individual is not necessarily novel in general, and time is wasted approaching a problem from an industrial direction when theory has been developed for a version of the problem. Or vice versa.A related problem is effective knowledge sharing between domains and how to fill "valleys" in the N-dimensional landscape of popular knowledge.
Here's an example: my experience contributing to teams working on industrial programming languages, I have noticed that relatively few people working in this space in have background knowledge in programming language theory (even self-taught!). Generally this is not an issue, but sometimes a problem comes up that the team has not experienced before, and spends a lot of time "figuring out", but the problem in general is well-known and solved.
This is a hard problem -- the biggest challenge is that you don't know what you don't know, and the first step is even learning enough to realize what you don't know. A final example: recently I got into abstract rewriting systems, but for a while I didn't even realize the problems I was trying to solve were well-known in the theory of rewrite systems until about a month in! Obviously, after that, things got a lot easier.
All right, so maybe now you're saying, "this is an ok observation, but so what? What can we do about it?" Well, I suggest the following personal steps:
- Inspire the relationship between formal methods and their applications during education - concentrations like "Applied Mathematics" deserve a fair share of theory, and "Theoretical Algorithms" deserve a fair share of applications, at least enough so that individuals new to a field have enough of an understanding of the larger ecosystem to know what they don't know, what is useful, and what isn't.
- Seek out answers to ideas that are novel to you - while they may be novel to you, they are likely not novel in general. Instead of reinventing the wheel, first research to discover if the problem has already been solved. As mentioned, figuring out how to discover whether the problem has been solved may not be trivial, and so some prerequisite knowledge of the domain is required (perhaps via (1)).
- When developing novel solutions, start from formal models.
It's either that, or get very knowledgeable about everything. I'm not certain which is more realistic, but I have a suspicion. And as always, if you have thoughts, I would love to hear from you!
