The term "math habits of mind" was first used by Cuoco, Goldenburg and Mark in a great 1996 article (linked here). The habits of mind that they envisioned are process and disposition standards rather than content standards. The general idea was that the content of mathematics will evolve, so we want to give students "a set of tools that they'll need to use, understand and make mathematics that doesn't yet exist" (Cuoco, Goldenburg, & Mark, p. 2). The habits of mind are the tools they proposed.
The Common Core Standards for Mathematical Practice (also know as the 8 Math Practices) were influenced by the habits of mind as well as the NCTM problem solving standards and the National Research Council's 2001 Report "Adding it up" (linked here) Like the habits of mind, the Math Practices address thinking processes, and dispositions that help students develop "deep, flexible, and enduring understanding of mathematics" (Briars, Mills, & Mitchell, 2011, p. 20).