Machine Learning (Theory)

1/15/2007

The Machine Learning Department

Tags: Machine Learning,Statistics jl@ 7:40 pm

Carnegie Mellon School of Computer Science has the first academic Machine Learning department. This department already existed as the Center for Automated Learning and Discovery, but recently changed it’s name.

The reason for changing the name is obvious: very few people think of themselves as “Automated Learner and Discoverers”, but there are number of people who think of themselves as “Machine Learners”. Machine learning is both more succinct and recognizable—good properties for a name.

A more interesting question is “Should there be a Machine Learning Department?”. Tom Mitchell has a relevant whitepaper claiming that machine learning is answering a different question than other fields or departments. The fundamental debate here is “Is machine learning different from statistics?”

At a cultural level, there is no real debate: they are different. Machine learning is characterized by several very active large peer reviewed conferences, operating in a computer science mode. Statistics tends to function with a greater emphasis on journals and a lesser emphasis on conferences which often implies a much longer publishing cycle.

In terms of the basic questions driving the field, the answer seems less clear. It is true that the core problems of statistics in the past have typically differed from the core problems of machine learning today. Yet, there has been some substantial overlap, and there are a number of statisticians nowadays that are actively doing machine learning. It’s reasonably plausible that in the long term statistics departments will adopt the core problems of machine learning, removing the reasons for a separate machine learning department.

The parallel question for computer science comes up less often perhaps because computer science is a notoriously broad field.

The practical implication of a new department is the ability to create a more specific curricula, admit more specific students, and hire faculty based upon more specific interests. Compared to a computer science program, classes on programming languages, computer architecture, or graphics might be dropped in favor of classes on learning theory, statistics, etc… Compared to a statistics program, classes on advanced parameter estimation and measure theory might be dropped in favor of algorithms and programming experience.

An alternative solution like “learn everything from computer science and statistics” is personally appealing to me, and I have benefitted from and recommend a broad education. However this is not practical for everyone. In my experience, a machine learning skill set is an effective specialization with which people can do important things in the world. Given this, having a department with a machine learning centered curricula seems like a good idea. At Carnegie Mellon, this is the Machine Learning department. In the future and elsewhere it may have a different name, but the value of the machine learning skill set should grow with research, improving computers, and improving data sources.

12 Comments to “The Machine Learning Department”
  1. Amory Blaine says:

    As you mentioned, CS is a very broad field. Do you think that ML is more deserving of its own department than other subfields? Does a Machine Learning department make more sense than an algorithms and complexity department? A systems department? A graphics and vision department?

  2. Anonymous says:

    Don’t you think machine learning should get involved more with other fields of AI, such as planning, logic and cognitive science?

  3. Anonymous says:

    I am skeptical of the narrow-education approach. To me, one of the greatest appeals of machine learning techniques is their applicability to a large set of problems in very disparate areas, and their potential in areas they haven’t been used yet. Wouldn’t you be slowing down these applications if students don’t quite understand these areas of potential application?

  4. I think that the divisions by which we group academics are somewhat arbitrary. Should computer science be in the Engineering Schools or in Arts and Sciences? There’s no right answer: some computer scientists are closer to electrical engineering, and some are closer to math; some CS is like math and some is like engineering. I think it’s good that some universities do it one way and some do it the other. If you study Greek religion, should be in the history, classics, or religion department, or should you have your own department? Who we choose to be our next-door colleagues is largely a matter of interest, not of an objective best division of research areas.

    Don’t you think machine learning should get involved more with other fields of AI, such as planning, logic and cognitive science?

    You could also argue: shouldn’t ML be more involved with computational neuroscience? Or maybe robotics? Or maybe the medicine school? Operations research? All of these connections make a lot of sense, and it would be a shame if every university forced one sort of connection and neglected the rest.

    Each researcher has a finite bandwith in the number of colleagues they can work with, chat with in the halls, go to lunch with, and so on. If you spend some of that bandwidth on external collaborations, then you lose some bandwidth with other kinds of ML researchers. And so on.

    I personally, would not want to be in a department with such a narrow focus (in some sense), but I can certainly understand why someone would.

  5. jl says:

    When something new comes up, it’s easy to look for disadvantages. Relative to either computer science or statistics, the focus of this department does look narrow. We could isntead look at the advantages.

    Looking at statistics relative to machine learning, statisticians don’t have the algorithms and programming experience. Can a person be broadly educated without having that experience? Maybe statistics is a narrow field. Looking at computer science, which doesn’t include any statistics, can a person be considered broadly educuated and yet still not understand how to carefully describe or think about the unknown? This isn’t idle ‘cup-is-half-fulling’. I regard ‘rogramming as the missing member of reading, ‘riting, and ‘rithmetic, and I’ve found a statistical understanding of the world genuinely valuable.

    Another reason why the department might be regarded as narrow is that an ML education is not as obviously useful as a CS or statistics education. This is partly a question for the future: will an ML education become increasingly useful? I believe the answer is yes.

  6. jl says:

    I’d say “yes”, ML is more deserving than some other subfields.

    “No”, an algorithms department doesn’t make sense, because it is too core to the purpose of computer science. It’s difficult for me to imagine CS department without algorithms, while a CS department without machine learning is plausible.

    “Yes”, a systems departments may make sense. They are often called ECE.

    I don’t know about graphics and vision.

  7. Vicente Malave says:

    I think the question should be: is there a “machine learning” way of thinking that is distinct from the practice of statistics (and perhaps computer science). I feel this is true, and Breiman summarizes it nicely in “Statistical Modeling: The Two Cultures”.

    With regards to the narrow focus of the department, the most attractive part of the PhD program to me is the lack of all the messy computer science things (i.e. Systems programming) needed for a CS degree in favor of a focus on more interesting things (like upper level courses in Statistical Learning Theory).

    I dont think collaboration will suffer if ML becomes its own discipline, after all we are in the business of finding interesting things in other people’s data, and there seem to be no shortage of interesting problems (my own interests have driven me into Cognitive Neuroscience).

  8. ingo says:

    Well, I looked at their homepage and found they offer Masters and PhD level study. This is exactly what I would have expected: Get breadth during your bachelors (probably in either CS or math) and focus during Masters/PhD time. IMHO, very natural and not at all risky.

    Now, ML-only study at the undergrad level, that’s something I would have serious doubts about. At the end of the day, we still need to turn our knowledge into working software systems and I believe the engineering content of a standard CS course to be the very minimum of what is required for that.

  9. Dave Bacon says:

    “Statistics tends to function with a greater emphasis on journals and a lesser emphasis on conferences which often implies a much longer publishing cycle.”

    Interesting…it would be fun to compare publishing cycles for different fields! In physics, and in particular in high energy theory and quantum computing, publication is two-fold, first on the arxiv, and then in a journal. I think the later increases the speed of the science, perhaps even beyond the CS model whose publication deadline is set by “elite” conference dates. I wonder how one would get this data…

  10. Some points:

    1. CMU is one of 5-6 places with *schools* of Computing (others include Utah, UC Irvine, and Georgia Tech). Being a school, CMU can choose to have departments in any sizeable area of CS. They already have units in Robotics, Software Engineering, Language Technologies, etc.

    2. UC Irvine is another such place with a school of Information and Computer Science with three units – Information Science, Computer Science, and Statistics.

    3. At Michigan, we have a department of Electrical Engineering and Computer Science under Engineering and separate units in Statistics (in the Literature, Sciences, and the Arts college) and Information Science (separate school).

    I personally like Irvine’s model best as it brings all related units into one place.

  11. I sympathize with the idea of an ML department because I suspect that computing (and really I mean statistics- and machine learning-style computing) will continue to, as it has aggressively in the past 50 years, move to the center of all scientific disciplines. I’m sure that simulations and numerical models will become increasingly relevant to theorists in testing their ideas and to empiricists in understanding the significance of their data. (On that point, John at CosmicVariance has a great post on the importance of statistical modeling to understanding recent particle physics data.)

    So, in this respect, I think Colleges of Computing are a good thing, since computing seems vaguely equally significant as say, the arts. On the other hand, I’m a little wary about the disciplinary barriers that get erected when things like Colleges or Departments are created, and specialties spun off into their own institutions. Are we sure that ML won’t benefit from having its practitioners also trained (to some degree) in programming languages, systems engineering, and theoretical computer science? I’m willing to believe that important innovations in ML won’t necessarily come from within the core ML way of approaching problems, but may come from connections to other kinds of computing (I guess I’m primarily thinking about theory, but certainly systems and languages has something to say, as evidenced by the recent post on parallel ML algorithms).

    I suppose a part of me is also reluctant to great further specializations because I’m not sure what field I rightly “belong to.” That’s a comment for another day, though, I think.

  12. [...] leo en el blog de John Langford que Carnegie Mellon ha creado el primer departamento académico de “Aprendizaje Automático”. Más allá de las apreciaciones personales (anda que no estaría bien trabajar en un departamento [...]

Leave a Reply


Powered by WordPress