I initially sat down to write this for our younger managers to understand what went into our hiring process, why we chose them out of a large applicant pool, and ultimately be able to pinpoint where improvements could be made. Whether it's a college baseball program or corporate America, I feel this process can be used across the board.
Bringing student managers on board is not as simple as it may seem. In my three years since starting our analytics team, we have had over 200 students interested in helping out the program. And while they are unpaid positions, hiring them all is unrealistic. I even learned from Year One to Year Three that the more students on board, is not necessarily the merrier (read about the Mythical Man-Month).
There was so much I learned from the beginning to now and I’m lucky to have been able to go through the process three times to correct my mistakes (and continue to make new ones). Between a class I took last spring at JMU focused on workplace analytics and reading Ray Dalio’s “Principles”, I had some ideas on how to incorporate what I learned into our hiring process.
I had not anticipated on returning to JMU this fall (COVID delayed my post-grad plans) and wanted our upper-class managers (Sam Gjormand and Camden Kay) to have the experience of conducting interviews, so I was more of a guide to them. I led them through the process, they did a great job carrying out the actual interviews.
Step One: Define the Traits of a Quality Manager
We divided this into two groups: Analytics Managers and On-Field Managers. Individually, we put together a list of traits we felt made up a good manager of each type. Then we combined them, identified where some duplicates were, and narrowed our list.
Going forward, we could probably trim this list down even further, but those were the traits we felt made a manager successful.
Step Two: Assign Weights to Each Trait
I then gave our current managers 100 points to “spend.” They divvied up their 100 points to the traits on an individual basis to prevent bias, creating a weight for each trait with the most important getting the most points. Because we had multiple managers and myself included in this process, we averaged out the scores for each trait, also weighting for “believability.” For example, I spent two years primarily on the field and then two years with my primary focus on the analytics side, so my point allotment accounted for just 50% of another manager who was solely focused on one or the other. Here’s what those scores looked like.
Why was it important to assign points to each trait? More on that later.
Step Three: Create a Questionnaire (Google Form to make it easy)
After sending an initial interest email to the student body, defining what exactly a “good manager” is and ranking the most important traits, we needed to filter down the number of interested students because it is unrealistic to interview 100+ students. Our questionnaire captured some additional information about the students, such as what they are studying, what technical skills they have or are working towards and generally why they are interested.
Then, to get more granular, we attached the following spray chart and asked the students where they would position the defense and why. We followed that up by asking, “If this hitter were to hit a ball down the third base line in his first two at-bats, where would you position your defense for his third at-bat and why?” What we felt was more important than their actual answer was their thought process for that answer. Cases can be made for both, we just wanted to see how they thought through it because we felt that same process would be applied once on the job.
Other questions on the app included asking what metrics they would use to evaluate a hitter and pitcher, and lastly asking them for something we left out that makes them a strong candidate.
Step Four: Forming Interview Questions
The way I interviewed as a junior starting our analytics program has changed significantly. What I have learned is that interviewing is an art of making the interviewee feel comfortable and uncomfortable, asking thought-provoking questions to ultimately create a “best-guess” about how they would perform on the job. Since I had two years of doing so and was not initially planning on returning, I wanted to leave the experience of actually conducting interviews to our head managers and they consult me if needed. They did an excellent job and needed very little help from me.
Where I did help was the actual forming of questions. By no means am I an expert, but I learned from asking poor questions, and from my class last spring how to ask better questions.
It’s important to formulate questions around the traits you deemed most important and disseminate between “Can we teach this?” and “Do they need this inherently?”. The question on the application where we asked what metrics they would use to evaluate a player is a good initial filter, but ultimately not a great in-person interview question because we should be confident in teaching how we go about evaluating players (or challenge them to come up with a thoughtful way to evaluate players on their own!).
Rather than asking something like “Are you an initiative taker?” ask something like, “Tell us about something you did or learned about during the quarantine.” Not only does that show some initiative (or lack thereof), but it could also help check the box for willingness to learn and curiosity.
Other good questions include those that require examples, or scenarios based on possible challenges you know they will eventually face. For example, if you know they will be helping put advance scouting reports together, ask them what they would do if they spent hours preparing for a game only to realize they made a huge mistake and everything is wrong—I’ve been there! Their answer should illicit a response in which you can get a feel for their attention to detail, reliability, and problem-solving skills. Or, what about trying to figure out how they will blend with your culture? Rather than asking “What are your strengths in a team setting?” ask something like “describe your ideal supervisor” or, for “a time you worked well in a team setting and what made that team successful?”
Step Five: Grading the Interviews
This is where that point system from Step Two comes back into play. Immediately following every interview, our head managers would separately grade each interviewee on each trait we deemed important. Those grades were then averaged out and multiplied by the points for each trait (AKA their weight). Each interviewee was graded as both an On-Field manager and Analytics manager to see where they ranked in the grand scheme. Maybe they wanted to be an Analytics manager, but we believed their skillset was more conducive to being on the field, grading them as both enabled us to communicate that if needed.
Step Six: Rank Candidates and Extend Offers
Once all of the grades were compiled and the applicants were in an ordered system, it was easy for our head managers to say something like “Yup, this lines up almost exactly the way I had thought.” One of the great things data is useful for is to confirm what we have believed to be true. And in this case, that is just what the data did. There was definitely some subjectivity in how the offers were made, mainly due to students possessing the “it” factor that may not have shown up in the scoring system, and the fact that we were not constrained by a budget.
Final Thoughts
Creating a process for what was important, developing interview questions around that, and creating some objectivity in a process that I had made solely subjectively for two years at least gives us something to work with. We were able to justify our decisions. If one of the hires does not pan out, we are able to look back to our process iteratively, identify where we think that flaw in the system was, and improve it going forward.
Is all of this overkill for hiring student managers? I don’t think so at all. I believe we owe it to the number of students applying to create something evidence-based, especially considering half of the managers are focused on the analytics side of the program.
Comments